initial terraform configuration for Google buildbot workers

[kuhnel]

Summary

This patch adds an initial terraform configuration for the new buildbot workers run at Google, on top of D81737.

This defined the cluster configuration and how the Docker images are deployed to the cluster.

[kuhnel]

• first version of terraform configuration

[kuhnel]

cleaned up dependencies

[tra]

buildbot/google/README.md

16

past -> paste

buildbot/google/gcloud_config.sh

4 ↗

(On Diff #270408)

config.sh?

If the script is executed while the current directory is not buildbot/google you may need to infer full file name for the config.

buildbot/google/terraform/README.md

16 ↗

(On Diff #270408)

This will not work as it's implemented now, because config will not be found. See the comment in gcloud_config.sh

28 ↗

(On Diff #270408)

*to* enable Kubernetes?

40 ↗

(On Diff #270408)

What's expected to be in the file name ?
Is buildbot-token-mlir-nvidia special in any way. I do not see it mantioned anywhere else in the patch. Who/where/how is going to use this secret?

buildbot/google/terraform/main.tf

121 ↗

(On Diff #270408)

Does it have something to do with the buildbot-token-mlir-nvidia we create as described in the Secrets section of the README?
I'm not quite sure how exactly those two are connected. Some comments with the details would be helpful here.

124–125 ↗

(On Diff #270408)

Pardon my ignorance -- what is taint and why does it matter for us?

[PaulkaToast]

buildbot/google/README.md

32

I'd maybe add a note here about the ability to reproduce failures the buildbot encounters locally using the docker containers for debugging. As in, if an LLVM contributor breaks a Google buildbot by accident they can try to reproduce it locally without having access to our infrastructure.

40

nit: add a newline at the end of files. (:

buildbot/google/terraform/README.md

28 ↗

(On Diff #270408)

If I'm not mistaken for new GCP projects, the Container Registry API must be enabled once as well.

[kuhnel]

• improved documentation of secrets and taints
• fixed missing new lines
• fixed config file paths

buildbot/google/terraform/README.md

16 ↗

(On Diff #270408)

fixed the script.

40 ↗

(On Diff #270408)

I fixed the name of the secrets and added more documentation on how they work.

buildbot/google/terraform/main.tf

121 ↗

(On Diff #270408)

yes, that should be buildbot-token-mlir-nvidia, i seem to have mixed different versions of the files...

124–125 ↗

(On Diff #270408)

tl;dr: This is a safe guard to only deploy container that require GPUs to machines with GPUs.

The long story:
https://cloud.google.com/kubernetes-engine/docs/how-to/gpus
https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/

I added more documentation there.

buildbot/google/README.md

+# LLVM buildbot workers configuration
+
+This folder contains some of the configuration of the buildbots managed
+at Google. The workers are deployed on Google Cloud.
+

buildbot/google/docker/README.md

+This folder contains the Dockerfiles and scripts used for some of the 
+buildbot workers. 
+
+# Scripts
+
+This folder also contains some scripts that are useful in working with the
+docker images.
+
+## build_run.sh
+Build a docker image and run it locally
+
+## build_deploy.sh
+Build a docker image, increment the version number, tag it and upload it to
+the registry.
+
+# Secrets
+
+The buildbot workers need a password to authenticate with the buildbot server. 
+This password needs to be kept a secert. The usual way to handle it is to store
+the secrets in a secure place and during runtime mount that secure place into 
+the container. The secret file shall just contain the password in plain text.
+
+Kubernetes offers a [mechanism to handle secrets](https://kubernetes.io/docs/concepts/configuration/secret/).

buildbot/google/docker/build_deploy.sh

+#!/bin/bash
+#===-- build_deploy.sh ---------------------------------------------------===//
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===//
+# This script will deploy a docker image to the registry.
+# Arguments: <path to Dockerfile> 
+# 
+# This updates the `VERSION` file with the latest version number.
+#===----------------------------------------------------------------------===//
+
+set -eu
+
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+IMAGE_NAME="${1%/}"
+
+# increment version number
+cd "${DIR}/${IMAGE_NAME}"
+
+# get version numbers from repository
+# FIXME: use variables to configure URL
+ALL_VERSIONS=$(gcloud container images list-tags gcr.io/sanitizer-bots/${IMAGE_NAME} --format=text | \
+  awk '/tags.*:\W+[0-9]+/ {print $2}')
+# read local version number from file and add it to the array
+ALL_VERSIONS+=($(cat VERSION))
+# find maximum version number and increment it
+VERSION=$(echo "${ALL_VERSIONS[*]}" | sort -nr | head -n1)
+VERSION=$(( ${VERSION} + 1 ))
+
+# get the git hash and add some suffixes
+GIT_HASH=$(git rev-parse HEAD)
+if [[ $(git diff --stat) != '' ]]; then
+  # if working copy is dirty
+  GIT_HASH+="-dirty-${USER}"
+elif [[ $(git --no-pager diff origin/master | wc -l) > 0 ]]; then
+  # if the hash has not been uploaded to origin/master yet
+  GIT_HASH+="-local-${USER}"
+fi
+
+# fully qualified image name
+# FIXME: use variables to configure URL
+QUALIFIED_NAME="gcr.io/sanitizer-bots/${IMAGE_NAME}"
+# tags to be added to the image and pushed to the repository
+TAGS=(
+  "${QUALIFIED_NAME}:latest" 
+  "${QUALIFIED_NAME}:${VERSION}"
+  "${QUALIFIED_NAME}:${GIT_HASH}"
+  )
+
+# build the image and tag it locally
+docker build -t ${IMAGE_NAME}:latest -t ${IMAGE_NAME}:${VERSION} .
+
+# print the list of tags to be pushed
+echo "-----------------------------------------"
+echo "image version: ${VERSION}"
+echo "tags:"
+printf '   %s\n' "${TAGS[@]}"
+echo "-----------------------------------------"
+read -p "Push to registry? [yN]" -n 1 -r
+echo
+
+if [[ $REPLY =~ ^[Yy]$ ]]
+then
+  for TAG in "${TAGS[@]}"
+  do
+    docker tag ${IMAGE_NAME}:${VERSION} "${TAG}"
+    docker push "${TAG}"
+  done
+  # store the version number
+  echo "${VERSION}" > VERSION
+fi

buildbot/google/docker/build_run.sh

+#!/bin/bash
+#===-- build_run.sh ------------------------------------------------------===//
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===//
+# This script will deploy a docker image to the registry.
+# Arguments:
+#     <path to Dockerfile>
+#     <path containing secrets>
+#     optional: <command to be executed in the container>
+#===----------------------------------------------------------------------===//
+
+set -eux
+
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+IMAGE_NAME="${1%/}"
+SECRET_STORAGE="$2"
+CMD=
+if [ "$#" -eq 3 ];
+then
+    CMD="$3"
+fi
+
+cd "${DIR}/${IMAGE_NAME}"
+
+docker build -t "${IMAGE_NAME}:latest" .
+docker run -it -v "${SECRET_STORAGE}":/secrets "${IMAGE_NAME}" ${CMD}

buildbot/google/docker/buildbot-mlir-nvidia/Dockerfile

+#===-- Dockerfile --------------------------------------------------------===//
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===//
+# Docker image used for the mlir-nvidia builder
+#===----------------------------------------------------------------------===//
+
+# Use the image from NVIDIA as base
+FROM nvidia/cuda:10.2-base-ubuntu18.04
+
+
+# install build tools
+RUN apt-get update; \
+    apt-get install -y software-properties-common apt-transport-https ca-certificates \
+      clang-8 lld-8 ninja-build git wget gnupg ccache \
+      python python-pip python-psutil ;\
+    update-alternatives --install /usr/bin/clang clang /usr/bin/clang-8 100 ;\
+    update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-8 100 ;\
+    update-alternatives --install /usr/bin/lld lld /usr/bin/lld-8 100
+
+# install cuda
+# avoid popups for keyboard configurations
+RUN DEBIAN_FRONTEND=noninteractive apt-get install -y cuda
+
+# Ubuntu ships with old cmake version, install the latest one
+# from https://apt.kitware.com/
+RUN wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | \
+       gpg --dearmor - | \
+       tee /etc/apt/trusted.gpg.d/kitware.gpg >/dev/null ;\
+    apt-add-repository 'deb https://apt.kitware.com/ubuntu/ bionic main' ;\
+    apt-get update ;\
+    apt-get install -y cmake
+
+# install (old) build bot version
+# this version of build bot requires python2!
+RUN pip install buildbot-slave==0.8.11
+
+# Volume to mount secrets into the container
+VOLUME /secrets
+
+# create user account, some test fail if run as root
+RUN useradd buildbot --create-home
+WORKDIR /home/buildbot
+USER buildbot
+
+# copy startup script
+COPY run.sh /home/buildbot/
+
+ENV WORKER_NAME="mlir-nvidia"
+
+# Set up buildbot host and maintainer info.
+RUN mkdir -p "${WORKER_NAME}/info/" ;\
+    echo "Christian Kühnel <[e-mail]>" > "${WORKER_NAME}/info/admin"
+
+CMD ./run.sh

buildbot/google/docker/buildbot-mlir-nvidia/VERSION

+1
 No newline at end of file

buildbot/google/docker/buildbot-mlir-nvidia/run.sh

+#!/bin/bash
+#===-- run.sh -------------------------------------------------------------===//
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===//
+# This script will start the buildbot worker
+#===----------------------------------------------------------------------===//
+
+set -eu
+
+# Read the worker password from a mounted file.
+WORKER_PASSWORD=$(cat /secrets/token)
+
+# generate the host information of this worker
+( 
+  uname -a ; \
+  cat /proc/cpuinfo | grep "model name" | head -n1 | cut -d " " -f 3- ;\
+  echo "number of cores: $(nproc)" ;\
+  nvidia-smi -L | cut -d "(" -f 1 ;\
+  lsb_release -d | cut -f 2- ; \
+  clang --version | head -n1 ; \
+  ld.lld-8 --version ; \
+  cmake --version | head -n1 
+) > ${WORKER_NAME}/info/host 
+
+# FIXME(kuhnel):
+# It looks like GKE sometimes deploys the container before the NVIDIA drivers 
+# are loaded on the host. In this case the GPU is not available during the 
+# entire lifecycle of the container. Not sure how to fix this properly. 
+# Maybe the above entry is enough as it depends on a working `nvidia-smi`. 
+# If not a workaround might be to check for the graphics card in this script and 
+# exit immediately if it's not available.
+
+# create the folder structure
+buildslave create-slave --keepalive=200 "${WORKER_NAME}" \
+  lab.llvm.org:9994 "${WORKER_NAME}" "${WORKER_PASSWORD}"
+
+# start the daemon, this command return immetiately
+buildslave start "${WORKER_NAME}"
+
+# To keep the container running and produce log outputs: dump the worker
+# log to stdout
+tail -f ${WORKER_NAME}/twistd.log