[ci] Make libc++ and Clang CI scripts independent

[ldionne]

Summary

This simply moves existing CI-related scripts for Clang outside of libc++
to make the two projects independent with respect to CI.

The reason for making this change is that libc++'s CI pipeline history
was full of Clang runs, making it a lot harder to figure out information
about libc++'s own jobs. I created a pipeline for Clang and this patch
will run pre-commit CI for Clang inside that pipeline instead.

[ldionne]

Update commit message.

[ldionne]

Try to fix path when generating the CI pipelines.

[ldionne]

Touch a file inside Clang to test the triggering.

[ldionne]

Poke CI after fixing pipeline settings.

[ldionne]

Ok, this is ready to go. More context around this change:

I created a clang-specific CI pipeline (https://buildkite.com/llvm-project/clang-ci) and Clang CI jobs will now trigger inside that pipeline instead of inside the libc++ pipeline (https://buildkite.com/llvm-project/libcxx-ci). The problem this solves is that 90% of the jobs of the libc++ CI pipelines are otherwise Clang jobs, which makes it nearly impossible for libc++ to figure out information about the jobs we're actually running.

• For now, this does not have any impact on the fact that Clang CI is still utilizing libc++ resources (although this is next on my list of things to address).
• This doesn't have any impact on whether we'll eventually want to move to Github Actions or some other CI system. This simply fixes the current system before the LLVM 17 release. In particular, I would like to avoid bikeshedding the location of the generate-buildkite-pipeline scripts under .ci -- it doesn't matter, I just wanted a neutral (non-libcxx-specific) location and that can be changed later once CI has stabilized.

[ldionne]

Poke CI after fixing Clang CI pipeline clone URL.

[philnik]

I like this a lot! LGTM.

[ldionne]

See https://github.com/google/llvm-premerge-checks/pull/452 for the other side of this change. https://github.com/google/llvm-premerge-checks/pull/452 must land after this patch.

[aaron.ballman]

In D155078#4493468, @ldionne wrote:

Ok, this is ready to go. More context around this change:

I created a clang-specific CI pipeline (https://buildkite.com/llvm-project/clang-ci) and Clang CI jobs will now trigger inside that pipeline instead of inside the libc++ pipeline (https://buildkite.com/llvm-project/libcxx-ci). The problem this solves is that 90% of the jobs of the libc++ CI pipelines are otherwise Clang jobs, which makes it nearly impossible for libc++ to figure out information about the jobs we're actually running.

• For now, this does not have any impact on the fact that Clang CI is still utilizing libc++ resources (although this is next on my list of things to address).
• This doesn't have any impact on whether we'll eventually want to move to Github Actions or some other CI system. This simply fixes the current system before the LLVM 17 release. In particular, I would like to avoid bikeshedding the location of the generate-buildkite-pipeline scripts under .ci -- it doesn't matter, I just wanted a neutral (non-libcxx-specific) location and that can be changed later once CI has stabilized.

Please excuse my ignorance, but does this change precommit CI test coverage when reviews are created/updated? e.g., if a review goes up with changes only to Clang, will the review still get a libc++ test run to ensure the Clang changes don't break libc++?

[ldionne]

In D155078#4493995, @aaron.ballman wrote:

In D155078#4493468, @ldionne wrote:

Ok, this is ready to go. More context around this change:

I created a clang-specific CI pipeline (https://buildkite.com/llvm-project/clang-ci) and Clang CI jobs will now trigger inside that pipeline instead of inside the libc++ pipeline (https://buildkite.com/llvm-project/libcxx-ci). The problem this solves is that 90% of the jobs of the libc++ CI pipelines are otherwise Clang jobs, which makes it nearly impossible for libc++ to figure out information about the jobs we're actually running.

• For now, this does not have any impact on the fact that Clang CI is still utilizing libc++ resources (although this is next on my list of things to address).
• This doesn't have any impact on whether we'll eventually want to move to Github Actions or some other CI system. This simply fixes the current system before the LLVM 17 release. In particular, I would like to avoid bikeshedding the location of the generate-buildkite-pipeline scripts under .ci -- it doesn't matter, I just wanted a neutral (non-libcxx-specific) location and that can be changed later once CI has stabilized.

Please excuse my ignorance, but does this change precommit CI test coverage when reviews are created/updated? e.g., if a review goes up with changes only to Clang, will the review still get a libc++ test run to ensure the Clang changes don't break libc++?

It actually fixes it. Previously, if both libc++ and Clang were modified in a review, only the libc++ CI was triggered. After this patch:

• If Clang is modified, the Clang pre-commit CI is triggered (which includes running the libc++ test suite w/ just-built clang in C++03, C++20 and Modules IIRC)
• If Libc++ is modified, the libc++ pre-commit CI is triggered (which includes the 60ish jobs we have)
• If both are modified, both are triggered.

[ldionne]

I will land this because I can tell this is correct after following the CI pipeline's execution, and https://github.com/google/llvm-premerge-checks/pull/452 has already landed so the CI could start becoming red anytime now. If there are concerns about Clang having its own CI pipeline (which I think is only a huge improvement), please raise them and we can address that concern post-commit.

[Mordante]

Thanks for working on this! I really like the direction. I mainly skimmed the code since the patch as already landed.

[aaron.ballman]

In D155078#4494167, @ldionne wrote:

In D155078#4493995, @aaron.ballman wrote:

In D155078#4493468, @ldionne wrote:

Ok, this is ready to go. More context around this change:

I created a clang-specific CI pipeline (https://buildkite.com/llvm-project/clang-ci) and Clang CI jobs will now trigger inside that pipeline instead of inside the libc++ pipeline (https://buildkite.com/llvm-project/libcxx-ci). The problem this solves is that 90% of the jobs of the libc++ CI pipelines are otherwise Clang jobs, which makes it nearly impossible for libc++ to figure out information about the jobs we're actually running.

• For now, this does not have any impact on the fact that Clang CI is still utilizing libc++ resources (although this is next on my list of things to address).
• This doesn't have any impact on whether we'll eventually want to move to Github Actions or some other CI system. This simply fixes the current system before the LLVM 17 release. In particular, I would like to avoid bikeshedding the location of the generate-buildkite-pipeline scripts under .ci -- it doesn't matter, I just wanted a neutral (non-libcxx-specific) location and that can be changed later once CI has stabilized.

Please excuse my ignorance, but does this change precommit CI test coverage when reviews are created/updated? e.g., if a review goes up with changes only to Clang, will the review still get a libc++ test run to ensure the Clang changes don't break libc++?

It actually fixes it. Previously, if both libc++ and Clang were modified in a review, only the libc++ CI was triggered. After this patch:

• If Clang is modified, the Clang pre-commit CI is triggered (which includes running the libc++ test suite w/ just-built clang in C++03, C++20 and Modules IIRC)
• If Libc++ is modified, the libc++ pre-commit CI is triggered (which includes the 60ish jobs we have)
• If both are modified, both are triggered.

Thank you, this sounds great to me then! LGTM. :-)

.ci/generate-buildkite-pipeline-premerge

 #!/usr/bin/env bash
 #===----------------------------------------------------------------------===##
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 #
 #===----------------------------------------------------------------------===##
 
 #
-# This file generates a Buildkite pipeline that triggers the libc++ CI
-# job(s) if needed.
-# See https://buildkite.com/docs/agent/v3/cli-pipeline#pipeline-format.
+# This file generates a Buildkite pipeline that triggers the various CI jobs for
+# the LLVM project during pre-commit CI (each time a Phabricator diff is uploaded).
 #
-# Invoked by CI on pre-merge check for a commit.
+# See https://buildkite.com/docs/agent/v3/cli-pipeline#pipeline-format.
 #
 
 if ! git diff --name-only HEAD~1 | grep -q -E "^libcxx/|^libcxxabi/|^libunwind/|^runtimes/|^cmake/|^clang/"; then
   # libcxx/, libcxxabi/, libunwind/, runtimes/, cmake/ or clang/ are not affected
   exit 0
 fi
 
 reviewID="$(git log --format=%B -n 1 | sed -nE 's/^Review-ID:[[:space:]]*(.+)$/\1/p')"
 if [[ "${reviewID}" != "" ]]; then
   buildMessage="https://llvm.org/${reviewID}"
 else
   buildMessage="Push to branch ${BUILDKITE_BRANCH}"
 fi
 
-
 cat <<EOF
 steps:
+EOF
+
+# If libc++ or one of the runtimes directories changed
+if git diff --name-only HEAD~1 | grep -q -E "^libcxx/|^libcxxabi/|^libunwind/|^runtimes/|^cmake/"; then
+  cat <<EOF
   - trigger: "libcxx-ci"
     build:
       message: "${buildMessage}"
       commit: "${BUILDKITE_COMMIT}"
       branch: "${BUILDKITE_BRANCH}"
 EOF
+fi
+
+# If clang changed
+if git diff --name-only HEAD~1 | grep -q -E "^clang/"; then
+  cat <<EOF
+  - trigger: "clang-ci"
+    build:
+      message: "${buildMessage}"
+      commit: "${BUILDKITE_COMMIT}"
+      branch: "${BUILDKITE_BRANCH}"
+EOF
+fi

.ci/generate-buildkite-pipeline-scheduled

 #!/usr/bin/env bash
 #===----------------------------------------------------------------------===##
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 #
 #===----------------------------------------------------------------------===##
 
 #
-# This file generates a Buildkite pipeline that triggers the libc++ CI jobs.
-# See https://buildkite.com/docs/agent/v3/cli-pipeline#pipeline-format.
+# This file generates a Buildkite pipeline that triggers the various CI jobs for
+# the LLVM project on scheduled builds.
 #
-# Invoked by CI on full builds.
+# See https://buildkite.com/docs/agent/v3/cli-pipeline#pipeline-format.
 #
 
 cat <<EOF
 steps:
   - trigger: "libcxx-ci"
     build:
       message: "${BUILDKITE_MESSAGE}"
       commit: "${BUILDKITE_COMMIT}"
       branch: "${BUILDKITE_BRANCH}"
+
+  - trigger: "clang-ci"
+    build:
+      message: "${BUILDKITE_MESSAGE}"
+      commit: "${BUILDKITE_COMMIT}"
+      branch: "${BUILDKITE_BRANCH}"
 EOF

libcxx/utils/ci/buildkite-pipeline-premerge.sh

 #!/usr/bin/env bash
 #===----------------------------------------------------------------------===##
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 #
 #===----------------------------------------------------------------------===##
 
-#
-# This file generates a Buildkite pipeline that triggers the libc++ CI
-# job(s) if needed.
-# See https://buildkite.com/docs/agent/v3/cli-pipeline#pipeline-format.
-#
-# Invoked by CI on pre-merge check for a commit.
-#
-
-if ! git diff --name-only HEAD~1 | grep -q -E "^libcxx/|^libcxxabi/|^libunwind/|^runtimes/|^cmake/|^clang/"; then
-  # libcxx/, libcxxabi/, libunwind/, runtimes/, cmake/ or clang/ are not affected
-  exit 0
-fi
-
-reviewID="$(git log --format=%B -n 1 | sed -nE 's/^Review-ID:[[:space:]]*(.+)$/\1/p')"
-if [[ "${reviewID}" != "" ]]; then
-  buildMessage="https://llvm.org/${reviewID}"
-else
-  buildMessage="Push to branch ${BUILDKITE_BRANCH}"
-fi
-
-
-cat <<EOF
-steps:
-  - trigger: "libcxx-ci"
-    build:
-      message: "${buildMessage}"
-      commit: "${BUILDKITE_COMMIT}"
-      branch: "${BUILDKITE_BRANCH}"
-EOF
+# TODO: Remove this once the premerge checks invoke the script below directly.
+./.ci/generate-buildkite-pipeline-premerge

libcxx/utils/ci/buildkite-pipeline-snapshot.sh

 #!/usr/bin/env bash
 #===----------------------------------------------------------------------===##
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 #
 #===----------------------------------------------------------------------===##
 
-#
-# This file generates a Buildkite pipeline that triggers the libc++ CI jobs.
-# See https://buildkite.com/docs/agent/v3/cli-pipeline#pipeline-format.
-#
-# Invoked by CI on full builds.
-#
-
-cat <<EOF
-steps:
-  - trigger: "libcxx-ci"
-    build:
-      message: "${BUILDKITE_MESSAGE}"
-      commit: "${BUILDKITE_COMMIT}"
-      branch: "${BUILDKITE_BRANCH}"
-EOF
+# TODO: Remove this once the premerge checks invoke the script below directly.
+./.ci/generate-buildkite-pipeline-scheduled

libcxx/utils/ci/generate-buildkite-pipeline

 #!/usr/bin/env bash
 #===----------------------------------------------------------------------===##
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 #
 #===----------------------------------------------------------------------===##
 
 #
 # This script generates the appropriate libc++ CI pipeline based on which project(s) were changed.
+# TODO: Remove this once the libcxx CI pipeline uploads the buildkite-pipeline.yml file directly.
 #
 
 if git diff --name-only HEAD~1 | grep -q -E "^libcxx/|^libcxxabi/|^libunwind/|^runtimes/|^cmake/"; then
-  LIBCXX_CHANGED=true
-fi
-
-if git diff --name-only HEAD~1 | grep -q -E "^clang/"; then
-  CLANG_CHANGED=true
-fi
-
-if [[ "${CLANG_CHANGED}" == "true" && "${LIBCXX_CHANGED}" != "true" ]]; then
-  cat clang/utils/ci/buildkite-pipeline.yml
-else
   cat libcxx/utils/ci/buildkite-pipeline.yml
 fi