docker images for mlir-nvidia

[kuhnel]

Summary

A bunch of people at Google are working on standardising the way we run our buildbots. We also want to add new ones. As part of this effort we want to upstream the configuration and documentation of our buildbots. This way other buildbot owners can benefit from our setup.

This first patch adds the docker image and some scripts for mlir-nvidia buildbot.

Future patches will add more documentation, Terraform/kubernetes configuration files, scripts, additional build bots

Please check if

• Is this the right repository for such things?
• Is this the right place in the folder structure?

[kuhnel]

removed stray edit

[mehdi_amini]

I don't know where is the best place for the Docker config to live, but there are some in the monorepo already FYI:

llvm-project/libcxx/utils/docker/debian9/buildbot/Dockerfile
llvm-project/clang/tools/clang-fuzzer/Dockerfile
llvm-project/libc/utils/buildbot/Dockerfile
llvm-project/llvm/utils/docker/debian8/Dockerfile
llvm-project/llvm/utils/docker/example/Dockerfile
llvm-project/llvm/utils/docker/nvidia-cuda/Dockerfile

[tra]

LGTM.

buildbot/google/docker/build_deploy.sh

23–26

Nit: This looks somewhat fragile -- anyone building from the same checked out tree will end up with the same version and will potentially clobber each other.
I think it would be more robust to always use checked-in version (or just git hash of the commit) and add "+local changes" to the tag/version if the checked out tree has been modified.
Not a big deal in practice considering that there's likely to be only one owner.

buildbot/google/docker/buildbot-mlir-nvidia/run.sh

15

We should probably describe what's expected to be in the secrets directory in the README file.

[PaulkaToast]

In D81737#2089966, @mehdi_amini wrote:

I don't know where is the best place for the Docker config to live, but there are some in the monorepo already FYI:

llvm-project/libcxx/utils/docker/debian9/buildbot/Dockerfile
llvm-project/clang/tools/clang-fuzzer/Dockerfile
llvm-project/libc/utils/buildbot/Dockerfile
llvm-project/llvm/utils/docker/debian8/Dockerfile
llvm-project/llvm/utils/docker/example/Dockerfile
llvm-project/llvm/utils/docker/nvidia-cuda/Dockerfile

I think it is debatable where the Dockerfiles should live. As mendi mentioned there are examples in the monorepo.
This links them closely with the project the belong to which might make more sense for contributors to individual projects.
However I also see the benefit of this centralized location as well for standardizing the setup.

I think there is an importance to keeping the Dockerfiles decoupled from the Google specific infrastructure. This means that if someone outside of Google would like to setup their own buildbot, they can re-use the project-specific Dockerfiles instead of starting from scratch. They might wish to set it up without terraform or GCP.
If our Dockerfiles need something google specific (e.g our particular secret handling) we could make use of the FROM directive to pull in the generic docker config for the llvm project and then customize it specifically to our needs.

buildbot/google/docker/README.md

24

nit: newline.

[kuhnel]

In D81737#2089966, @mehdi_amini wrote:

I don't know where is the best place for the Docker config to live, but there are some in the monorepo already FYI:

Yes, I'm aware of this. @asl recommended putting it in llvm-zorg, as this is there buildbot related things should go.
He also proposed moving the other buildbot Dockerfiles here.

[kuhnel]

• replaced version file with number from registry
• updated readme on handling secrets

buildbot/google/docker/build_deploy.sh

23–26

I can get the version number from the registry. Will add a patch for that. However then it's more difficult to trace changes between the docker repo and the git commits.

[tra]

buildbot/google/docker/build_deploy.sh

23–26

I don't think it's worth complicating it yet. I guess something as simple as adding a hash-based tag + 'dirty' bit would do.
This should allow unambiguously identifying uploaded images.
e.g. something like this:

tags = ${IMAGE_NAME}:latest ${IMAGE_NAME}:${VERSION} ${IMAGE_NAME}:${HASH} 
if <tree is dirty>; then
  tags += ${IMAGE_NAME}:${HASH}-${USER}-${BUILDNO}
  BUILDNO++
fi

36

What if I happen to run it in a tree I haven't updated for a while? We probably don't want accidentally overwriting the currently used images with an out-of-date version. Should we check if the image with particular version already exists and inform the user that they are about to overwrite it?

[mehdi_amini]

In D81737#2092635, @kuhnel wrote:

In D81737#2089966, @mehdi_amini wrote:

I don't know where is the best place for the Docker config to live, but there are some in the monorepo already FYI:

Yes, I'm aware of this. @asl recommended putting it in llvm-zorg, as this is there buildbot related things should go.
He also proposed moving the other buildbot Dockerfiles here.

To me this is fine, as long as it provides easy way for reproduction locally when a failure happens: I don't mind checking another repo to get the Docker images, but ideally I should just run a script and point to my local repo to get a reproduced build.

[kuhnel]

improved tagging of images

[kuhnel]

fixed last patch

buildbot/google/docker/build_deploy.sh

36

For that we do need to check the remote repository. So I'll keep that code.

It's surprisingly hard to come up with a solid way to tag images :)

buildbot/google/README.md

+# LLVM buildbot workers configuration
+
+This folder contains some of the configuration of the buildbots managed
+at Google. The workers are deployed on Google Cloud.
+

buildbot/google/docker/README.md

+This folder contains the Dockerfiles and scripts used for some of the 
+buildbot workers. 
+
+# Scripts
+
+This folder also contains some scripts that are useful in working with the
+docker images.
+
+## build_run.sh
+Build a docker image and run it locally
+
+## build_deploy.sh
+Build a docker image, increment the version number, tag it and upload it to
+the registry.
+
+# Secrets
+
+The buildbot workers need a password to authenticate with the buildbot server. 
+This password needs to be kept a secert. The usual way to handle it is to store
+the secrets in a secure place and during runtime mount that secure place into 
+the container. The secret file shall just contain the password in plain text.
+
+Kubernetes offers a [mechanism to handle secrets](https://kubernetes.io/docs/concepts/configuration/secret/).

buildbot/google/docker/build_deploy.sh

+#!/bin/bash
+#===-- build_deploy.sh ---------------------------------------------------===//
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===//
+# This script will deploy a docker image to the registry.
+# Arguments: <path to Dockerfile> 
+# 
+# This updates the `VERSION` file with the latest version number.
+#===----------------------------------------------------------------------===//
+
+set -eu
+
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+
+# get config options
+
+IMAGE_NAME="${1%/}"
+
+# increment version number
+cd "${DIR}/${IMAGE_NAME}"
+
+# get the latest tag number from the repository
+# FIXME: use configuration variables instead of hard-coded path
+VERSION=$(gcloud container images list-tags gcr.io/sanitizer-bots/${IMAGE_NAME} --format=text | \
+  awk '/tags.*:\W+[0-9]+/ {print $2}' | \
+  sort -nr | \
+  head -n1)
+
+# increment the version number
+VERSION=$(( ${VERSION} + 1 ))
+echo "image version: ${VERSION}"
+
+docker build -t ${IMAGE_NAME}:latest -t ${IMAGE_NAME}:${VERSION} .
+
+read -p "Push to registry? [yN]" -n 1 -r
+echo
+if [[ $REPLY =~ ^[Yy]$ ]]
+then
+  QUALIFIED_NAME="gcr.io/sanitizer-bots/${IMAGE_NAME}"
+  docker tag ${IMAGE_NAME}:${VERSION} ${QUALIFIED_NAME}:${VERSION}
+  docker tag ${IMAGE_NAME}:latest ${QUALIFIED_NAME}:latest
+  docker push ${QUALIFIED_NAME}:${VERSION}
+  docker push ${QUALIFIED_NAME}:latest
+fi

buildbot/google/docker/build_run.sh

+#!/bin/bash
+#===-- build_run.sh ------------------------------------------------------===//
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===//
+# This script will deploy a docker image to the registry.
+# Arguments:
+#     <path to Dockerfile>
+#     <path containing secrets>
+#     optional: <command to be executed in the container>
+#===----------------------------------------------------------------------===//
+
+set -eux
+
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+IMAGE_NAME="${1%/}"
+SECRET_STORAGE="$2"
+CMD=
+if [ "$#" -eq 3 ];
+then
+    CMD="$3"
+fi
+
+cd "${DIR}/${IMAGE_NAME}"
+
+docker build -t "${IMAGE_NAME}" .
+docker run -it -v "${SECRET_STORAGE}":/secrets "${IMAGE_NAME}" ${CMD}

buildbot/google/docker/buildbot-mlir-nvidia/Dockerfile

+#===-- Dockerfile --------------------------------------------------------===//
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===//
+# Docker image used for the mlir-nvidia builder
+#===----------------------------------------------------------------------===//
+
+# Use the image from NVIDIA as base
+FROM nvidia/cuda:10.2-base-ubuntu18.04
+
+
+# install build tools
+RUN apt-get update; \
+    apt-get install -y software-properties-common apt-transport-https ca-certificates \
+      clang-8 lld-8 ninja-build git wget gnupg ccache \
+      python python-pip python-psutil ;\
+    update-alternatives --install /usr/bin/clang clang /usr/bin/clang-8 100 ;\
+    update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-8 100 ;\
+    update-alternatives --install /usr/bin/lld lld /usr/bin/lld-8 100
+
+# install cuda
+# avoid popups for keyboard configurations
+RUN DEBIAN_FRONTEND=noninteractive apt-get install -y cuda
+
+# Ubuntu ships with old cmake version, install the latest one
+# from https://apt.kitware.com/
+RUN wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | \
+       gpg --dearmor - | \
+       tee /etc/apt/trusted.gpg.d/kitware.gpg >/dev/null ;\
+    apt-add-repository 'deb https://apt.kitware.com/ubuntu/ bionic main' ;\
+    apt-get update ;\
+    apt-get install -y cmake
+
+# install (old) build bot version
+# this version of build bot requires python2!
+RUN pip install buildbot-slave==0.8.11
+
+# Volume to mount secrets into the container
+VOLUME /secrets
+
+# create user account, some test fail if run as root
+RUN useradd buildbot --create-home
+WORKDIR /home/buildbot
+USER buildbot
+
+# copy startup script
+COPY run.sh /home/buildbot/
+
+ENV WORKER_NAME="mlir-nvidia"
+
+# Set up buildbot host and maintainer info.
+RUN mkdir -p "${WORKER_NAME}/info/" ;\
+    echo "Christian Kühnel <[e-mail]>" > "${WORKER_NAME}/info/admin"
+
+CMD ./run.sh

buildbot/google/docker/buildbot-mlir-nvidia/run.sh

+#!/bin/bash
+#===-- run.sh -------------------------------------------------------------===//
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===//
+# This script will start the buildbot worker
+#===----------------------------------------------------------------------===//
+
+set -eu
+
+# Read the worker password from a mounted file.
+WORKER_PASSWORD=$(cat /secrets/token)
+
+# generate the host information of this worker
+( 
+  uname -a ; \
+  cat /proc/cpuinfo | grep "model name" | head -n1 | cut -d " " -f 3- ;\
+  echo "number of cores: $(nproc)" ;\
+  nvidia-smi -L | cut -d "(" -f 1 ;\
+  lsb_release -d | cut -f 2- ; \
+  clang --version | head -n1 ; \
+  ld.lld-8 --version ; \
+  cmake --version | head -n1 
+) > ${WORKER_NAME}/info/host 
+
+# FIXME(kuhnel):
+# It looks like GKE sometimes deploys the container before the NVIDIA drivers 
+# are loaded on the host. In this case the GPU is not available during the 
+# entire lifecycle of the container. Not sure how to fix this properly. 
+# Maybe the above entry is enough as it depends on a working `nvidia-smi`. 
+# If not a workaround might be to check for the graphics card in this script and 
+# exit immediately if it's not available.
+
+# create the folder structure
+buildslave create-slave --keepalive=200 "${WORKER_NAME}" \
+  lab.llvm.org:9994 "${WORKER_NAME}" "${WORKER_PASSWORD}"
+
+# start the daemon, this command return immetiately
+buildslave start "${WORKER_NAME}"
+
+# To keep the container running and produce log outputs: dump the worker
+# log to stdout
+tail -f ${WORKER_NAME}/twistd.log