[libc++] Add a libc++ CI pipeline specific to Clang changes

[ldionne]

Summary

This will ensure that Clang changes get tested against libc++.

[ldionne]

This is a rough draft based on a discussion I had with @erichkeane over lunch today. This should provide customized jobs for testing Clang changes against libc++.

[aaron.ballman]

Thank you for working on this, I think this is going to be fantastic!!

libcxx/utils/ci/buildkite-pipeline-clang.yml

18

Do we have/need something to keep this value from drifting away from the trunk version?

42

Please excuse my utter ignorance of buildkite, but... this looks identical to the previous step with the only change being to the label (same for the rest of the entries). How does the CI know which version to test against? Does it pull that out of the label automatically or something?

[Mordante]

Thanks for working on this!

[erichkeane]

Thank you for doing this! I have no way of reviewing these, but I'm hopeful someone more knowledgeable will be able to.

[Mordante]

In D137759#3919733, @erichkeane wrote:

Thank you for doing this! I have no way of reviewing these, but I'm hopeful someone more knowledgeable will be able to.

I'm familiar with libc++'s Buildkite configuration so I'm happy to review, but I'll wait until it's ready for review.

[ldionne]

In D137759#3919733, @erichkeane wrote:

Thank you for doing this! I have no way of reviewing these, but I'm hopeful someone more knowledgeable will be able to.

Yeah no problem, this was mostly for awareness because we talked about it.

libcxx/utils/ci/buildkite-pipeline-clang.yml

18

We do this manually every release when we bump the base compiler version we're using. This is a deliberate act and I think it's fine. I do agree it's a bit annoying to duplicate this here and in the other libc++ CI configuration.

42

The answer is that all the jobs do the same thing as of this version of the patch. I just wanted to get something out real quick, I was sitting in EWG and wanted to make sure I didn't forget my conversation with @erichkeane.

Actually, let's discuss which jobs you folks want. What configurations do you want to test? I would suggest:

• A few standard modes -- what would make the most sense for clang?
• Modules enabled

That's probably enough, but I'd like to know what you and @erichkeane think about it. Oh and let's ping @mizvekov who has been needing these facilities too.

[ldionne]

Try to build Clang in its own pipeline step

[ldionne]

Try to fix install step

[ldionne]

Try to fix CI

[ldionne]

Experiment w/ CI

[ldionne]

Attempt 4/N. Sorry for the spam, this is going to keep going until I finally figure this out.

[ldionne]

Try again.

[aaron.ballman]

libcxx/utils/ci/buildkite-pipeline-clang.yml

18

Okie dokie, that's fine by me, but we might someday want to script that so there's less risk of missing a spot to update. (Nothing to be done for this patch though!)

42

Ah, okay, I wasn't stupid then, wahoo! :-D

In terms of what configurations to test, I think the important ones are:

1. Modules
2. Target OS differences (Windows vs Linux), especially where sizeof(size_t) or sizeof(long) differs
3. C++latest

+++ ones past here are more questionable +++

4. C++98? Alternatively: user-controlled standard version through some magic?
5. Target hardware differences (ARM vs IA vs PPC)?
6. (Someday, perhaps) Experimental constexpr interpreter vs current constexpr evaluator?

We might want to combine some of these options together as well, so you can do C++latest + modules + Windows and C++latest + no modules + Linux, but I don't know that we need to try for testing the full combinatorial matrix.

[ldionne]

Fix glob patterns

[ldionne]

Build clang for real now that artifacts work

[ldionne]

libcxx/utils/ci/buildkite-pipeline-clang.yml

42

Sounds good, we'll have modules, C++03 and C++2b for the time being. I am not sure how to set it up for Windows but we can add it later.

For other architectures, the problem is that our fast builders are all on Linux and using the same Docker image. Builders for other funky architectures are controlled by other folks and have less bandwidth, so I would not tackle that in this initial patch.

[ldionne]

Change jobs and try to fix env var for finding clang

[aaron.ballman]

libcxx/utils/ci/buildkite-pipeline-clang.yml

42

SGTM, I think it's totally fine to roll this out incrementally.

[ldionne]

Try to fix env paths

[ldionne]

Another attempt

[ldionne]

Use $(pwd) instead of $PWD

[ldionne]

Fix path

[ldionne]

I think $PWD is set incorrectly in the build

[ldionne]

Artifacts should work now

[ldionne]

Debug

[ldionne]

Make sure clang is executable.

[ldionne]

Fix call to chmod

[ldionne]

Remove temporary comments. This should be good to go now. I'll merge once the CI is green.

Once I merge, we can do a test Clang review to confirm that the right jobs are triggered. Also, I'd like to note that
we're using the libc++ CI resources right now. If this ends up taking too much resources away from libc++ CI, we'll have
to negotiate something because I don't want to starve our own pipeline.

[ldionne]

Actually, CI's going to be green for sure since we are running the libc++ CI. I'll merge this.

[Mordante]

Nice! Some nits. But I think we need to make sure that change to both Clang and libc++ still run the Clang CI.

libcxx/utils/ci/buildkite-pipeline-clang.yml

50

Isn't it build by default?

83

To avoid confusion with C++20 modules.

libcxx/utils/ci/generate-buildkite-pipeline

17

Note when there are changes to clang and libc++ this pipeline isn't executed. This might be an issue:

• A clang modular build change affects libc++
• libc++ is fixed and the Clang modular build is no longer tested, so it's unclear whether the fix works

Should we instead have one build script and set environment variable which CI test should be executed?

[ldionne]

libcxx/utils/ci/buildkite-pipeline-clang.yml

50

No, I don't think it is.

libcxx/utils/ci/generate-buildkite-pipeline

17

I agree. Trying something out in https://reviews.llvm.org/D138042.

libcxx/utils/ci/buildkite-pipeline-clang.yml

+#===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===##
+
+#
+# This file describes the various pre-commit CI bots used to test Clang against
+# libc++ under various configurations. Unlike the usual libc++ CI pipeline,
+# which aims to test libc++ itself, this pipeline aims to test Clang by
+# compiling libc++ and running its test suite against the just-built Clang,
+# in various configurations.
+#
+env:
+    LLVM_HEAD_VERSION: "16"
+steps:
+  - label: "Building clang"
+    commands:
+      - "mkdir install"
+      - "cmake -S llvm -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_INSTALL_PREFIX=install -DLLVM_ENABLE_PROJECTS=\"clang;compiler-rt\""
+      - "ninja -C build install-clang install-clang-resource-headers"
+      - "buildkite-agent artifact upload 'install/**/*'"
+    env:
+        CC: "clang-${LLVM_HEAD_VERSION}"
+        CXX: "clang++-${LLVM_HEAD_VERSION}"
+    agents:
+      queue: "libcxx-builders"
+      os: "linux"
+    retry:
+      automatic:
+        - exit_status: -1  # Agent was lost
+          limit: 2
+    timeout_in_minutes: 120
+
+  - wait
+
+  - label: "C++03"
+    commands:
+      - "buildkite-agent artifact download 'install/**' ."
+      - "export CC=$(pwd)/install/bin/clang"
+      - "export CXX=$(pwd)/install/bin/clang++"
+      - "chmod +x install/bin/clang install/bin/clang++"
+      - "libcxx/utils/ci/run-buildbot generic-cxx03"
+    artifact_paths:
+      - "**/test-results.xml"
+      - "**/crash_diagnostics/*"
+    env:
+        LLVM_SYMBOLIZER_PATH: "/usr/bin/llvm-symbolizer-${LLVM_HEAD_VERSION}" # TODO: Should we build that from scratch?
+        CLANG_CRASH_DIAGNOSTICS_DIR: "crash_diagnostics"
+    agents:
+      queue: "libcxx-builders"
+      os: "linux"
+    retry:
+      automatic:
+        - exit_status: -1  # Agent was lost
+          limit: 2
+    timeout_in_minutes: 120
+
+  - label: "C++2b"
+    commands:
+      - "buildkite-agent artifact download 'install/**' ."
+      - "export CC=$(pwd)/install/bin/clang"
+      - "export CXX=$(pwd)/install/bin/clang++"
+      - "chmod +x install/bin/clang install/bin/clang++"
+      - "libcxx/utils/ci/run-buildbot generic-cxx2b"
+    artifact_paths:
+      - "**/test-results.xml"
+      - "**/crash_diagnostics/*"
+    env:
+        LLVM_SYMBOLIZER_PATH: "/usr/bin/llvm-symbolizer-${LLVM_HEAD_VERSION}" # TODO: Should we build that from scratch?
+        CLANG_CRASH_DIAGNOSTICS_DIR: "crash_diagnostics"
+    agents:
+      queue: "libcxx-builders"
+      os: "linux"
+    retry:
+      automatic:
+        - exit_status: -1  # Agent was lost
+          limit: 2
+    timeout_in_minutes: 120
+
+  - label: "Modules"
+    commands:
+      - "buildkite-agent artifact download 'install/**' ."
+      - "export CC=$(pwd)/install/bin/clang"
+      - "export CXX=$(pwd)/install/bin/clang++"
+      - "chmod +x install/bin/clang install/bin/clang++"
+      - "libcxx/utils/ci/run-buildbot generic-modules"
+    artifact_paths:
+      - "**/test-results.xml"
+      - "**/crash_diagnostics/*"
+    env:
+        LLVM_SYMBOLIZER_PATH: "/usr/bin/llvm-symbolizer-${LLVM_HEAD_VERSION}" # TODO: Should we build that from scratch?
+        CLANG_CRASH_DIAGNOSTICS_DIR: "crash_diagnostics"
+    agents:
+      queue: "libcxx-builders"
+      os: "linux"
+    retry:
+      automatic:
+        - exit_status: -1  # Agent was lost
+          limit: 2
+    timeout_in_minutes: 120

libcxx/utils/ci/buildkite-pipeline-premerge.sh

 #
 # This file generates a Buildkite pipeline that triggers the libc++ CI
 # job(s) if needed.
 # See https://buildkite.com/docs/agent/v3/cli-pipeline#pipeline-format.
 #
 # Invoked by CI on pre-merge check for a commit.
 #
 
-DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
-
-if ! git diff --name-only HEAD~1 | grep -q -E "^libcxx/|^libcxxabi/|^libunwind/|^runtimes/|^cmake/"; then
-  # libcxx/, libcxxabi/, libunwind/, runtimes/ or cmake/ are not affected
+if ! git diff --name-only HEAD~1 | grep -q -E "^libcxx/|^libcxxabi/|^libunwind/|^runtimes/|^cmake/|^clang/"; then
+  # libcxx/, libcxxabi/, libunwind/, runtimes/, cmake/ or clang/ are not affected
   exit 0
 fi
 
 reviewID="$(git log --format=%B -n 1 | sed -nE 's/^Review-ID:[[:space:]]*(.+)$/\1/p')"
 if [[ "${reviewID}" != "" ]]; then
   buildMessage="https://llvm.org/${reviewID}"
 else
   buildMessage="Push to branch ${BUILDKITE_BRANCH}"

libcxx/utils/ci/generate-buildkite-pipeline

 #!/usr/bin/env bash
 #===----------------------------------------------------------------------===##
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 #
 #===----------------------------------------------------------------------===##
 
+#
+# This script generates the appropriate libc++ CI pipeline based on which project(s) were changed.
+#
+
+if git diff --name-only HEAD~1 | grep -q -E "^libcxx/|^libcxxabi/|^libunwind/|^runtimes/|^cmake/"; then
   cat libcxx/utils/ci/buildkite-pipeline.yml
+elif git diff --name-only HEAD~1 | grep -q -E "^clang/"; then
+  cat libcxx/utils/ci/buildkite-pipeline-clang.yml
+fi