Add buildbot with C++20 configuration

[ilya-biryukov]

Summary

The worker runs inside GCE alongside other Google buildbots.
Actual build uses Clang 15. The Dockerfiles are tested locally.

Prior discussion on Discourse:
https://discourse.llvm.org/t/adding-a-c-20-buildbot/67156

[ilya-biryukov]

Sending this out early to have it reviewed by the time I get the required credentials.
This will also require the corresponding configuration changes in our GCP changes before being submitted.

buildbot/google/terraform/main.tf

345 ↗

(On Diff #483991)

This will require extra configuration in our GCP project after I recieve a password for the new worker.

[kadircet]

buildbot/google/docker/buildbot-cpp20/run.sh

29

you probably want to run $CC and $LD here instead?

buildbot/google/terraform/main.tf

318 ↗

(On Diff #483991)

i think 2 cs are enough here

buildbot/osuosl/master/config/builders.py

2470

do we really aim to detect any runtime regressions by running tests ? otherwise i'd just leave them out.

2471

we're only building clangd here, but running check-clang-tools.

2476

if we end up not running tests, i guess we can also get rid of assertions

[ilya-biryukov]

• fix a typo in /vol/ccache
• use password from clangd worker (approved by Galina in our email exchange)
• leave out the targets and checks, use defaults that build and test everything
• Use $CC and $LD instead of the wrong compilers in run.sh

[ilya-biryukov]

buildbot/google/docker/buildbot-cpp20/run.sh

29

Nice catch, thanks! I was copy-pasting without giving too much thought to it.

buildbot/google/terraform/main.tf

318 ↗

(On Diff #483991)

Wow, nice catch. I'll send a patch for clangd worker to update this (this code is a copy-paste).
We probably don't persist the cache among runs because of this, not sure how severe this is for the running time (assuming that the worker runs continuously and does not spawn per build)

buildbot/osuosl/master/config/builders.py

2470

Sure, running the tests also seems useful to catch any discrepancies in runtime behavior between C++20 and C++17.
There are some cases where the code may silently change behavior.

2471

Yeah, good point. I would like to default to building as much as possible to start with. I have removed the targets and checks completely, it seems the code can just build and run all tests by default. I do not have a good of the build latency trade-offs there. Please let me know if there is some good subset of targets that would provide good coverage and yet reduce the build times significantly.

PS also initially leaving out compiler-rt, lld and other subprojects that I am less familiar with. But that's something that we may add in the future.

2476

I think we should still keep assertions on. If we enable assertions, we build more code that could potentially not compile in C++20 mode and catch more potential issues.

[ilya-biryukov]

buildbot/osuosl/master/config/builders.py

2471

typo: I do not have a good understanding of the build latency trade-offs there

[gkistanova]

LGTM.

Thanks for the patch, Ilya!
And thanks for helping with the review, Kadir! All good comments.

Having "llvm", "clang", "clang-tools-extra" is a good staring point. You can always add more later if the allocated hardware will handle more with a reasonable response time.
Building with assertion is a good default. UnifiedTreeBuilder will run check-all, so assertions will be handy.

[ilya-biryukov]

• Use kubernetes resource YAML for deployment instead of terraform config. I had problems updating our terraform deployment, it seems broken. I will follow up by moving existing clangd buildbot out of terrafrom too, the kubectl-based workflow seems prefereable.
• Set version of the new docker image to 0 (script updates it on build).

buildbot/google/docker/buildbot-cpp20/Dockerfile

+#===-- Dockerfile --------------------------------------------------------===//
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===//
+# Environment variables configurable at runtime:
+#    BUILDBOT_PORT - server port to connect to
+#===----------------------------------------------------------------------===//
+
+FROM debian:bullseye
+
+# Install build tools.
+RUN apt-get update && \
+    apt-get install -y \
+        software-properties-common apt-transport-https ca-certificates \
+        git wget gnupg \
+        cmake ninja-build  \
+        ccache \
+        python3 python3-pip python3-psutil && \
+    # Install clang-15 to make the minimal C++20 support in the toolchain.
+    echo "deb http://apt.llvm.org/bullseye/ llvm-toolchain-bullseye-15 main" >> /etc/apt/sources.list.d/llvm-15.list && \
+    wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key >> /etc/apt/trusted.gpg.d/llvm.asc && \
+    apt-get update && \
+    apt-get install -y clang-15 lld-15 && \
+    apt-get clean
+
+# Install build bot (server was at 2.8.5-dev at time of writing).
+RUN pip3 install buildbot-worker==2.8.4
+
+# Workaround permissions issues when writing to named volumes
+# https://github.com/docker/compose/issues/3270#issuecomment-206214034
+RUN mkdir -p /vol/test /vol/ccache /vol/worker ; \
+    chmod -R 777 /vol
+
+# Volume to mount secrets into the container.
+VOLUME /vol/secrets
+# Volume to store data for local, manual testing of the container.
+VOLUME /vol/test
+# Volume to store ccache.
+VOLUME /vol/ccache
+ENV CCACHE_DIR=/vol/ccache
+# Volume for worker working directory.
+VOLUME /vol/worker
+
+# Create user account, some tests fail if run as root.
+RUN useradd buildbot --create-home
+WORKDIR /vol/worker
+
+# Copy startup script.
+COPY run.sh /home/buildbot/
+RUN chmod a+rx /home/buildbot/run.sh
+
+USER buildbot
+ENV WORKER_NAME="clang-debian-cpp20"
+
+# Allow the server port of this agent to be configurable during deployment.
+# This way we can connect the same image to production and integration.
+# Ports:
+#   9990 - production
+#   9994 - integration
+ENV BUILDBOT_PORT="9994"
+
+# Configure LLVM tools.
+ENV CC=clang-15
+ENV CXX=clang++-15
+ENV LD=lld-15
+
+# Run startup script.
+CMD /home/buildbot/run.sh

buildbot/google/docker/buildbot-cpp20/VERSION

+0

buildbot/google/docker/buildbot-cpp20/run.sh

+#!/bin/bash
+#===-- run.sh -------------------------------------------------------------===//
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===//
+# This script will start the buildbot worker
+#
+#===----------------------------------------------------------------------===//
+
+set -eu
+
+# Read the worker password from a mounted file.
+WORKER_PASSWORD=$(cat /vol/secrets/token)
+
+# Set up buildbot host and maintainer info.
+mkdir -p "${WORKER_NAME}/info/"
+echo "Ilya Biryukov <[e-mail]>" > "${WORKER_NAME}/info/admin"
+
+# generate the host information of this worker
+(
+  uname -a ; \
+  cat /proc/cpuinfo | grep "model name" | head -n1 | cut -d " " -f 3- ;\
+  echo "number of cores: $(nproc)" ;\
+  cat /proc/meminfo | grep MemTo ;\
+  lsb_release -d | cut -f 2- ; \
+  $CC --version | head -n1 ; \
+  $LD --version ; \
+  cmake --version | head -n1 ; \
+) > ${WORKER_NAME}/info/host
+
+echo "Host information:"
+cat ${WORKER_NAME}/info/host
+
+# create the folder structure
+echo "creating worker ${WORKER_NAME} at port ${BUILDBOT_PORT}..."
+buildbot-worker create-worker --keepalive=200 "${WORKER_NAME}" \
+  lab.llvm.org:${BUILDBOT_PORT} "${WORKER_NAME}" "${WORKER_PASSWORD}"
+
+# start the daemon, this command returns immediately
+echo "starting worker..."
+set +e
+if ! buildbot-worker start "${WORKER_NAME}"; then
+  echo ERROR: starting worker failed. contents of twistd.log:
+  cat "${WORKER_NAME}/twistd.log"
+  exit 1
+fi
+
+# To keep the container running and produce log outputs: dump the worker
+# log to stdout
+echo "Following worker log..."
+tail -f ${WORKER_NAME}/twistd.log

buildbot/google/terraform/deployment-clang-debian-cpp20.yaml

+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  labels:
+    app: clang-debian-cpp20
+  name: clang-debian-cpp20
+  namespace: default
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: clang-debian-cpp20
+  strategy:
+    rollingUpdate:
+      maxSurge: 0
+      maxUnavailable: 1
+    type: RollingUpdate
+  template:
+    metadata:
+      labels:
+        app: clang-debian-cpp20
+    spec:
+      automountServiceAccountToken: false
+      containers:
+      - env:
+        - name: BUILDBOT_PORT
+          value: "9994" # 9994 is staging, 9990 is production.
+        image: gcr.io/sanitizer-bots/buildbot-cpp20:1
+        name: buildbot-clang-debian-cpp20
+        resources:
+          limits:
+            cpu: "15"
+            memory: 45G
+          requests:
+            cpu: "15"
+            memory: 45G
+        volumeMounts:
+        - mountPath: /vol/secrets
+          mountPropagation: None
+          name: buildbot-token
+        - mountPath: /vol/cccache
+          mountPropagation: None
+          name: ccache-vol
+        - mountPath: /vol/worker
+          mountPropagation: None
+          name: worker-vol
+      nodeSelector:
+        pool: linux-16-core-pool
+      restartPolicy: Always
+      volumes:
+      - name: buildbot-token
+        secret:
+          optional: false
+          secretName: password-clangd-ubuntu-clang
+      - emptyDir: {}
+        name: ccache-vol
+      - emptyDir: {}
+        name: worker-vol

buildbot/osuosl/master/config/builders.py

'factory': UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
              "-DLLVM_CCACHE_BUILD=ON",
              "-DLLVM_USE_SANITIZER=Thread",
              "-DCMAKE_BUILD_TYPE=Release",
              "-DCLANGD_ENABLE_REMOTE=ON",
              "-DLLVM_ENABLE_ASSERTIONS=ON",
              "-DGRPC_INSTALL_PATH=/usr/local/lib/grpc"
          ])},
 
+    # Build in C++20 configuration.
+    {'name': "clang-debian-cpp20",
+     'tags': ["clang", "c++20"],
+     'workernames': ["clang-debian-cpp20"],
+     'builddir': "clang-debian-cpp20",
+     'factory': UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
+         clean=True,
+         depends_on_projects=["llvm", "clang", "clang-tools-extra"],
+         extra_configure_args=[
+             "-DCMAKE_CXX_STANDARD=20"
+             "-DLLVM_CCACHE_BUILD=ON",
+             "-DCMAKE_BUILD_TYPE=Release",
+             "-DLLVM_ENABLE_ASSERTIONS=ON",
+         ])},
+
     # Target ARC from Synopsys
     {'name': "arc-builder",
      'tags': ["clang", "lld"],
      'collapseRequests': False,
      'workernames' : ["arc-worker"],
      'builddir': "arc-folder",
      'factory': UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
         depends_on_projects=["llvm", "clang", "lld"],

buildbot/osuosl/master/config/workers.py

return [
         create_worker("mlir-ubuntu-worker2", max_builds=1),
         create_worker("mlir-ubuntu-worker3", max_builds=1),
         create_worker("mlir-ubuntu-worker4", max_builds=1),
 
         # Ubuntu 18.04 on Google Cloud, for machine configuration check
         # buildbot/google/terraform/main.tf
         create_worker("clangd-ubuntu-clang", max_builds=1),
 
+        # Debian bullseye, build in C++20 configuration, for worker
+        # configuration check buildbot/google/terraform/main.tf
+        create_worker("clang-debian-cpp20", max_builds=1),
+
         # Ubuntu 18.04.LTS x86_64, GCE instance
         create_worker("polly-x86_64-gce1", properties={'jobs': 2}, max_builds=1),
         create_worker("polly-x86_64-gce2", properties={'jobs': 2}, max_builds=1),
 
         # Ubuntu 18.04.LTS x86_64, Intel(R) Xeon(R) CPU X3460 @ 2.80GHz, 32 GiB RAM
         create_worker("polly-x86_64-fdcserver", properties={'jobs': 8, 'loadaverage': 8}, max_builds=1),
 
         # Windows 10, AMD Ryzen 5 PRO 4650G, 16 GiB RAM