[libc++][CI] Increases the Clang version used.

[Mordante]

Summary

Changes the CI to use the Clang 16 nightly builds instead of Clang 14.
(The libc++15 branch was accidentally build using Clang 14 instead of
Clang 15; hence the skipping of a number.)

Also adds a Clang 15 build to the test matrix.

Based on the private discussion with @ldionne we decided to move
the configuration parameters from the run-buildbot script to the
CI configuration buildkite-pipeline.yml. Other hard-coded values
from the Dockerfile should be move to the CI configuration too. That
will be done in another commit.

C++17 will use Clang-15 since D131479 causes a test to fail.

[Mordante]

Retry CI with new Docker image.

[Mordante]

Fixes CI.

[Mordante]

Attempts to fix the CI.

[Mordante]

Another attempt to fix the CI.

[ldionne]

As discussed during live review, instead I think we should define the compiler in use inside buildkite-pipeline.yml via env (https://buildkite.com/docs/pipelines/command-step#command-step-attributes). This would allow dropping various versioned jobs from the run-buildbot script (e.g. generic-clang-14 would become a call to generic-cxx2b with env set to CXX=clang++-14).

[ldionne]

I think this means we should also somehow hardcode what version of e.g. clang-tidy is used in the .yml file. Basically, the Dockerfile installs a bunch of stuff but makes no decision as to which version of a given tool is used. Then, the .yml file picks the default versions it wants to use, and the run-buildbot script is as version-agnostic as possible.

[Mordante]

Tests the alternative approach as discussed during the live review.
(When this works the other jobs needs to be adjusted too.)

[Mordante]

Test CI with a global version.

[Mordante]

More testing.

[Mordante]

Convert all builders to the new style.

[Mordante]

Test.

[Mordante]

Try to use Clang 16, this might fail if the updated Docker image hasn't propagated to the CI runners.

[Mordante]

Rebased to verify the modular build failures are in ToT too.

[Mordante]

Allow all builds for testing.

[Mordante]

Test modular build with a newer Clang 15.

[Mordante]

Test with new Docker image and rebased.

[ldionne]

There's a few comments, but this LGTM. Thanks a lot, this is a realllllly nice improvement. This will decouple updating the compilers for CI from updating the Docker image on runners, which has been a source of pain in the past.

Just so that we're on the same page, this will also expose us to potential breaks in the nightly Clang build when we update the Docker image. However, I think that's acceptable since we can do basic testing before updating the Docker image.

libcxx/utils/ci/buildkite-pipeline.yml

169

I think these jobs were forgotten.

210

Is this supported? That would be nice.

334

I think this one should be using clang-head too. It just means that we will build Clang from scratch using clang-head, and then we'll build libc++ using the just-built clang-fromscratch.

It's mostly for consistency with other CI jobs, I guess, because in practice we should be able to build Clang from scratch using almost any system compiler.

[Mordante]

Addresses review comments.

[Mordante]

libcxx/utils/ci/buildkite-pipeline.yml

210

Interesting suggestion! It works
https://buildkite.com/llvm-project/libcxx-ci/builds/13149

libcxx/utils/ci/buildkite-pipeline.yml

 # This file should never contain logic -- all the logic must be offloaded
 # into scripts. This is critical to being able to reproduce CI issues outside
 # of the CI environment, which is important for debugging.
 #
 # It is also worth noting that this script is split into several sections, the
 # goal being to reduce the load on testers when a commit is known to fail.
 #
 
+env:
+    LLVM_HEAD_VERSION: "15"
+    GCC_STABLE_VERSION: "12"
 steps:
   #
   # Light pre-commit tests for things like formatting or when people forget
   # to update generated files.
   #
   - label: "Format"
     command: "libcxx/utils/ci/run-buildbot check-format"
     artifact_paths:
steps:
   #
   - wait
 
   - label: "C++2b"
     command: "libcxx/utils/ci/run-buildbot generic-cxx2b"
     artifact_paths:
       - "**/test-results.xml"
       - "**/*.abilist"
+    env:
+        CC: "clang-${LLVM_HEAD_VERSION}"
+        CXX: "clang++-${LLVM_HEAD_VERSION}"
     agents:
       queue: "libcxx-builders"
       os: "linux"
     retry:
       automatic:
         - exit_status: -1  # Agent was lost
           limit: 2
     timeout_in_minutes: 120
 
   - label: "C++11"
     command: "libcxx/utils/ci/run-buildbot generic-cxx11"
     artifact_paths:
       - "**/test-results.xml"
       - "**/*.abilist"
+    env:
+        CC: "clang-${LLVM_HEAD_VERSION}"
+        CXX: "clang++-${LLVM_HEAD_VERSION}"
     agents:
       queue: "libcxx-builders"
       os: "linux"
     retry:
       automatic:
         - exit_status: -1  # Agent was lost
           limit: 2
     timeout_in_minutes: 120
 
   - label: "C++03"
     command: "libcxx/utils/ci/run-buildbot generic-cxx03"
     artifact_paths:
       - "**/test-results.xml"
       - "**/*.abilist"
+    env:
+        CC: "clang-${LLVM_HEAD_VERSION}"
+        CXX: "clang++-${LLVM_HEAD_VERSION}"
     agents:
       queue: "libcxx-builders"
       os: "linux"
     retry:
       automatic:
         - exit_status: -1  # Agent was lost
           limit: 2
     timeout_in_minutes: 120
 
   - label: "Modular build"
     command: "libcxx/utils/ci/run-buildbot generic-modules"
     artifact_paths:
       - "**/test-results.xml"
       - "**/*.abilist"
+    env:
+        CC: "clang-${LLVM_HEAD_VERSION}"
+        CXX: "clang++-${LLVM_HEAD_VERSION}"
     agents:
       queue: "libcxx-builders"
       os: "linux"
     retry:
       automatic:
         - exit_status: -1  # Agent was lost
           limit: 2
     timeout_in_minutes: 120
 
   - label: "GCC 12 / C++latest"
     command: "libcxx/utils/ci/run-buildbot generic-gcc"
     artifact_paths:
       - "**/test-results.xml"
       - "**/*.abilist"
+    env:
+        CC: "gcc-${GCC_STABLE_VERSION}"
+        CXX: "g++-${GCC_STABLE_VERSION}"
     agents:
       queue: "libcxx-builders"
       os: "linux"
     retry:
       automatic:
         - exit_status: -1  # Agent was lost
           limit: 2
     timeout_in_minutes: 120
 
   #
   # All other supported configurations of libc++.
   #
   - wait
 
   - label: "C++20"
     command: "libcxx/utils/ci/run-buildbot generic-cxx20"
     artifact_paths:
       - "**/test-results.xml"
       - "**/*.abilist"
     agents:
       queue: "libcxx-builders"
       os: "linux"
     retry:
       automatic:
         - exit_status: -1  # Agent was lost
agents:
       os: "linux"
     retry:
       automatic:
         - exit_status: -1  # Agent was lost
           limit: 2
     timeout_in_minutes: 120
 
   # Tests with the supported compilers.
   - label: "GCC 12 / C++11"
     command: "libcxx/utils/ci/run-buildbot generic-gcc-cxx11"
     artifact_paths:
       - "**/test-results.xml"
       - "**/*.abilist"
+    env:
+        CC: "gcc-${GCC_STABLE_VERSION}"
+        CXX: "g++-${GCC_STABLE_VERSION}"
     agents:
       queue: "libcxx-builders"
       os: "linux"
     retry:
       automatic:
         - exit_status: -1  # Agent was lost
           limit: 2
     timeout_in_minutes: 120
 
   - label: "Clang 13"
-    command: "libcxx/utils/ci/run-buildbot generic-clang-13"
+    command: "libcxx/utils/ci/run-buildbot generic-cxx2b"
     artifact_paths:
       - "**/test-results.xml"
       - "**/*.abilist"
+    env:
+        CC: "clang-13"
+        CXX: "clang++-13"
     agents:
       queue: "libcxx-builders"
       os: "linux"
     retry:
       automatic:
         - exit_status: -1  # Agent was lost
           limit: 2
     timeout_in_minutes: 120
 
   - label: "Clang 14"
-    command: "libcxx/utils/ci/run-buildbot generic-clang-14"
+    command: "libcxx/utils/ci/run-buildbot generic-cxx2b"
+    env:
+        CC: "clang-14"
+        CXX: "clang++-14"
     artifact_paths:
       - "**/test-results.xml"
       - "**/*.abilist"
     agents:
       queue: "libcxx-builders"
       os: "linux"
     retry:
       automatic:
         - exit_status: -1  # Agent was lost
           limit: 2
     timeout_in_minutes: 120
 
   # Tests with the sanitizers.
   - group: "Sanitizers"
     steps:
     - label: "ASAN"
       command: "libcxx/utils/ci/run-buildbot generic-asan"
       artifact_paths:
         - "**/test-results.xml"
         - "**/*.abilist"
+      env:
+          CC: "clang-${LLVM_HEAD_VERSION}"
+          CXX: "clang++-${LLVM_HEAD_VERSION}"
       agents:
         queue: "libcxx-builders"
         os: "linux"
       retry:
         automatic:
           - exit_status: -1  # Agent was lost
             limit: 2
       timeout_in_minutes: 120
 
     - label: "TSAN"
       command: "libcxx/utils/ci/run-buildbot generic-tsan"
       artifact_paths:
         - "**/test-results.xml"
         - "**/*.abilist"
+      env:
+          CC: "clang-${LLVM_HEAD_VERSION}"
+          CXX: "clang++-${LLVM_HEAD_VERSION}"
       agents:
         queue: "libcxx-builders"
         os: "linux"
       retry:
         automatic:
           - exit_status: -1  # Agent was lost
             limit: 2
       timeout_in_minutes: 120
 
     - label: "UBSAN"
       command: "libcxx/utils/ci/run-buildbot generic-ubsan"
       artifact_paths:
         - "**/test-results.xml"
         - "**/*.abilist"
+      env:
+          CC: "clang-${LLVM_HEAD_VERSION}"
+          CXX: "clang++-${LLVM_HEAD_VERSION}"
       agents:
         queue: "libcxx-builders"
         os: "linux"
       retry:
         automatic:
           - exit_status: -1  # Agent was lost
             limit: 2
       timeout_in_minutes: 120
 
     - label: "MSAN"
       command: "libcxx/utils/ci/run-buildbot generic-msan"
       artifact_paths:
         - "**/test-results.xml"
         - "**/*.abilist"
+      env:
+          CC: "clang-${LLVM_HEAD_VERSION}"
+          CXX: "clang++-${LLVM_HEAD_VERSION}"
       agents:
         queue: "libcxx-builders"
         os: "linux"
       retry:
         automatic:
           - exit_status: -1  # Agent was lost
             limit: 2
       timeout_in_minutes: 120
 
   # Tests with the various supported ways to build libc++.
   - label: "Bootstrapping build"
     command: "libcxx/utils/ci/run-buildbot bootstrapping-build"
     artifact_paths:
       - "**/test-results.xml"
       - "**/*.abilist"
     agents:
       queue: "libcxx-builders"
       os: "linux"
     retry:
       automatic:
         - exit_status: -1  # Agent was lost
           limit: 2
     timeout_in_minutes: 120
 
   - group: "Legacy"
     steps:
     - label: "Legacy Lit configuration"
       command: "libcxx/utils/ci/run-buildbot legacy-test-config"
       artifact_paths:
         - "**/test-results.xml"
         - "**/*.abilist"
+      env:
+          CC: "clang-${LLVM_HEAD_VERSION}"
+          CXX: "clang++-${LLVM_HEAD_VERSION}"
       agents:
         queue: "libcxx-builders"
         os: "linux"
       retry:
         automatic:
           - exit_status: -1  # Agent was lost
             limit: 2
       timeout_in_minutes: 120
 
     - label: "Legacy LLVM_ENABLE_PROJECTS build"
       command: "libcxx/utils/ci/run-buildbot legacy-project-build"
       artifact_paths:
         - "**/test-results.xml"
         - "**/*.abilist"
+      env:
+          CC: "clang-${LLVM_HEAD_VERSION}"
+          CXX: "clang++-${LLVM_HEAD_VERSION}"
       agents:
         queue: "libcxx-builders"
         os: "linux"
       retry:
         automatic:
           - exit_status: -1  # Agent was lost
             limit: 2
       timeout_in_minutes: 120
 
   # Tests with various build configurations.
   - label: "Static libraries"
     command: "libcxx/utils/ci/run-buildbot generic-static"
     artifact_paths:
       - "**/test-results.xml"
       - "**/*.abilist"
+    env:
+        CC: "clang-${LLVM_HEAD_VERSION}"
+        CXX: "clang++-${LLVM_HEAD_VERSION}"
     agents:
       queue: "libcxx-builders"
       os: "linux"
     retry:
       automatic:
         - exit_status: -1  # Agent was lost
           limit: 2
     timeout_in_minutes: 120
 
   - label: "Shared library with merged ABI and unwinder libraries"
     command: "libcxx/utils/ci/run-buildbot generic-merged"
     artifact_paths:
       - "**/test-results.xml"
       - "**/*.abilist"
+    env:
+        CC: "clang-${LLVM_HEAD_VERSION}"
+        CXX: "clang++-${LLVM_HEAD_VERSION}"
     agents:
       queue: "libcxx-builders"
       os: "linux"
     retry:
       automatic:
         - exit_status: -1  # Agent was lost
           limit: 2
     timeout_in_minutes: 120
 
   - label: "Assertions enabled"
     command: "libcxx/utils/ci/run-buildbot generic-assertions"
     artifact_paths:
       - "**/test-results.xml"
       - "**/*.abilist"
+    env:
+        CC: "clang-${LLVM_HEAD_VERSION}"
+        CXX: "clang++-${LLVM_HEAD_VERSION}"
     agents:
       queue: "libcxx-builders"
       os: "linux"
     retry:
       automatic:
         - exit_status: -1  # Agent was lost
           limit: 2
     timeout_in_minutes: 120
 
   - label: "Debug mode"
     command: "libcxx/utils/ci/run-buildbot generic-debug-mode"
     artifact_paths:
       - "**/test-results.xml"
       - "**/*.abilist"
+    env:
+        CC: "clang-${LLVM_HEAD_VERSION}"
+        CXX: "clang++-${LLVM_HEAD_VERSION}"
     agents:
       queue: "libcxx-builders"
       os: "linux"
     retry:
       automatic:
         - exit_status: -1  # Agent was lost
           limit: 2
     timeout_in_minutes: 120
 
   - label: "No transitive includes"
     command: "libcxx/utils/ci/run-buildbot generic-no-transitive-includes"
     artifact_paths:
       - "**/test-results.xml"
       - "**/*.abilist"
+    env:
+        CC: "clang-${LLVM_HEAD_VERSION}"
+        CXX: "clang++-${LLVM_HEAD_VERSION}"
     agents:
       queue: "libcxx-builders"
       os: "linux"
     retry:
       automatic:
         - exit_status: -1  # Agent was lost
           limit: 2
     timeout_in_minutes: 120
 
   - label: "With LLVM's libunwind"
     command: "libcxx/utils/ci/run-buildbot generic-with_llvm_unwinder"
     artifact_paths:
       - "**/test-results.xml"
       - "**/*.abilist"
+    env:
+        CC: "clang-${LLVM_HEAD_VERSION}"
+        CXX: "clang++-${LLVM_HEAD_VERSION}"
     agents:
       queue: "libcxx-builders"
       os: "linux"
     retry:
       automatic:
         - exit_status: -1  # Agent was lost
           limit: 2
     timeout_in_minutes: 120
 
   - group: "Parts disabled"
     steps:
     - label: "No threads"
       command: "libcxx/utils/ci/run-buildbot generic-no-threads"
       artifact_paths:
         - "**/test-results.xml"
         - "**/*.abilist"
+      env:
+          CC: "clang-${LLVM_HEAD_VERSION}"
+          CXX: "clang++-${LLVM_HEAD_VERSION}"
       agents:
         queue: "libcxx-builders"
         os: "linux"
       retry:
         automatic:
           - exit_status: -1  # Agent was lost
             limit: 2
       timeout_in_minutes: 120
 
     - label: "No filesystem"
       command: "libcxx/utils/ci/run-buildbot generic-no-filesystem"
       artifact_paths:
         - "**/test-results.xml"
         - "**/*.abilist"
+      env:
+          CC: "clang-${LLVM_HEAD_VERSION}"
+          CXX: "clang++-${LLVM_HEAD_VERSION}"
       agents:
         queue: "libcxx-builders"
         os: "linux"
       retry:
         automatic:
           - exit_status: -1  # Agent was lost
             limit: 2
       timeout_in_minutes: 120
 
     - label: "No random device"
       command: "libcxx/utils/ci/run-buildbot generic-no-random_device"
       artifact_paths:
         - "**/test-results.xml"
         - "**/*.abilist"
+      env:
+          CC: "clang-${LLVM_HEAD_VERSION}"
+          CXX: "clang++-${LLVM_HEAD_VERSION}"
       agents:
         queue: "libcxx-builders"
         os: "linux"
       retry:
         automatic:
           - exit_status: -1  # Agent was lost
             limit: 2
       timeout_in_minutes: 120
 
     - label: "No locale"
       command: "libcxx/utils/ci/run-buildbot generic-no-localization"
       artifact_paths:
         - "**/test-results.xml"
         - "**/*.abilist"
+      env:
+          CC: "clang-${LLVM_HEAD_VERSION}"
+          CXX: "clang++-${LLVM_HEAD_VERSION}"
       agents:
         queue: "libcxx-builders"
         os: "linux"
       retry:
         automatic:
           - exit_status: -1  # Agent was lost
             limit: 2
       timeout_in_minutes: 120
 
     - label: "No Unicode"
       command: "libcxx/utils/ci/run-buildbot generic-no-unicode"
       artifact_paths:
         - "**/test-results.xml"
         - "**/*.abilist"
+      env:
+          CC: "clang-${LLVM_HEAD_VERSION}"
+          CXX: "clang++-${LLVM_HEAD_VERSION}"
       agents:
         queue: "libcxx-builders"
         os: "linux"
       retry:
         automatic:
           - exit_status: -1  # Agent was lost
             limit: 2
       timeout_in_minutes: 120
 
     - label: "No wide characters"
       command: "libcxx/utils/ci/run-buildbot generic-no-wide-characters"
       artifact_paths:
         - "**/test-results.xml"
         - "**/*.abilist"
+      env:
+          CC: "clang-${LLVM_HEAD_VERSION}"
+          CXX: "clang++-${LLVM_HEAD_VERSION}"
       agents:
         queue: "libcxx-builders"
         os: "linux"
       retry:
         automatic:
           - exit_status: -1  # Agent was lost
             limit: 2
       timeout_in_minutes: 120
 
     - label: "No experimental features"
       command: "libcxx/utils/ci/run-buildbot generic-no-experimental"
       artifact_paths:
         - "**/test-results.xml"
         - "**/*.abilist"
+      env:
+          CC: "clang-${LLVM_HEAD_VERSION}"
+          CXX: "clang++-${LLVM_HEAD_VERSION}"
       agents:
         queue: "libcxx-builders"
         os: "linux"
       retry:
         automatic:
           - exit_status: -1  # Agent was lost
             limit: 2
       timeout_in_minutes: 120
 
     - label: "No exceptions"
       command: "libcxx/utils/ci/run-buildbot generic-noexceptions"
       artifact_paths:
         - "**/test-results.xml"
         - "**/*.abilist"
+      env:
+          CC: "clang-${LLVM_HEAD_VERSION}"
+          CXX: "clang++-${LLVM_HEAD_VERSION}"
       agents:
         queue: "libcxx-builders"
         os: "linux"
       retry:
         automatic:
           - exit_status: -1  # Agent was lost
             limit: 2
       timeout_in_minutes: 120
 
   - label: "Unstable ABI"
     command: "libcxx/utils/ci/run-buildbot generic-abi-unstable"
     artifact_paths:
       - "**/test-results.xml"
       - "**/*.abilist"
+    env:
+        CC: "clang-${LLVM_HEAD_VERSION}"
+        CXX: "clang++-${LLVM_HEAD_VERSION}"
     agents:
       queue: "libcxx-builders"
       os: "linux"
     retry:
       automatic:
         - exit_status: -1  # Agent was lost
           limit: 2
     timeout_in_minutes: 120
 
   # Other non-testing CI jobs
   - label: "Benchmarks"
     command: "libcxx/utils/ci/run-buildbot benchmarks"
     artifact_paths:
       - "**/test-results.xml"
       - "**/*.abilist"
+    env:
+        CC: "clang-${LLVM_HEAD_VERSION}"
+        CXX: "clang++-${LLVM_HEAD_VERSION}"
     agents:
       queue: "libcxx-builders"
       os: "linux"
     retry:
       automatic:
         - exit_status: -1  # Agent was lost
           limit: 2
     timeout_in_minutes: 120
steps:
   - group: "AIX"
     steps:
     - label: "AIX (32-bit)"
       command: "libcxx/utils/ci/run-buildbot aix"
       artifact_paths:
         - "**/test-results.xml"
         - "**/*.abilist"
       env:
+          CC: "clang"
+          CXX: "clang++"
           OBJECT_MODE: "32"
       agents:
           queue: libcxx-builders
           os: aix
       retry:
         automatic:
           - exit_status: -1  # Agent was lost
             limit: 2
       timeout_in_minutes: 120
 
     - label: "AIX (64-bit)"
       command: "libcxx/utils/ci/run-buildbot aix"
       artifact_paths:
         - "**/test-results.xml"
         - "**/*.abilist"
       env:
+          CC: "clang"
+          CXX: "clang++"
           OBJECT_MODE: "64"
       agents:
           queue: libcxx-builders
           os: aix
       retry:
         automatic:
           - exit_status: -1  # Agent was lost
             limit: 2
       timeout_in_minutes: 120

libcxx/utils/ci/run-buildbot

 generic-merged)
     clean
     generate-cmake -C "${MONOREPO_ROOT}/libcxx/cmake/caches/Generic-merged.cmake" \
                    -DLIBCXX_TEST_CONFIG="llvm-libc++-shared.cfg.in" \
                    -DLIBCXXABI_TEST_CONFIG="llvm-libc++abi-merged.cfg.in" \
                    -DLIBUNWIND_TEST_CONFIG="llvm-libunwind-merged.cfg.in"
     check-runtimes
 ;;
-generic-clang-13)
-    export CC=clang-13
-    export CXX=clang++-13
-    clean
-    generate-cmake
-    check-runtimes
-    check-abi-list
-;;
-generic-clang-14)
-    export CC=clang-14
-    export CXX=clang++-14
-    clean
-    generate-cmake
-    check-runtimes
-    check-abi-list
-;;
 generic-gcc)
-    export CC=gcc-12
-    export CXX=g++-12
     clean
     generate-cmake -DLIBCXX_ENABLE_WERROR=NO \
                    -DLIBCXXABI_ENABLE_WERROR=NO \
                    -DLIBUNWIND_ENABLE_WERROR=NO
     check-runtimes
 ;;
 generic-gcc-cxx11)
-    export CC=gcc-12
-    export CXX=g++-12
     clean
     generate-cmake -C "${MONOREPO_ROOT}/libcxx/cmake/caches/Generic-cxx11.cmake" \
                    -DLIBCXX_ENABLE_WERROR=NO \
                    -DLIBCXXABI_ENABLE_WERROR=NO \
                    -DLIBUNWIND_ENABLE_WERROR=NO
     check-runtimes
 ;;
 generic-asan)
mingw-dll-i686)
     generate-cmake \
           -DCMAKE_C_COMPILER=i686-w64-mingw32-clang \
           -DCMAKE_CXX_COMPILER=i686-w64-mingw32-clang++ \
           -C "${MONOREPO_ROOT}/libcxx/cmake/caches/MinGW.cmake"
     echo "+++ Running the libc++ tests"
     ${NINJA} -vC "${BUILD_DIR}" check-cxx
 ;;
 aix)
-    export CC=clang
-    export CXX=clang++
     clean
     generate-cmake -C "${MONOREPO_ROOT}/libcxx/cmake/caches/AIX.cmake" \
                    -DLIBCXX_TEST_CONFIG="ibm-libc++-shared.cfg.in" \
                    -DLIBCXXABI_TEST_CONFIG="ibm-libc++abi-shared.cfg.in" \
                    -DLIBUNWIND_TEST_CONFIG="ibm-libunwind-shared.cfg.in"
     check-abi-list
     check-runtimes
 ;;