[buildbot] Moved MLIR buildbot config under buildbot/google/mlir

[tra]

Summary

buildbot/google itself will also hold configs for other bots owned by Google.

[kuhnel]

Sorry I did not understand why do you want to move these files.
Why not keep the new docker files for the existing machines right next to the existing ones?

So my thought on the directory structure was:

• /buildbot/google/
  • docker/
    • mlir-nvidia/
    • win32/
    • your-docker-dir/
    • some-other-bot/
    • <common scripting>
  • terraform/
    • <one terraform config for all machines>

[tra]

In D84256#2166061, @kuhnel wrote:

Sorry I did not understand why do you want to move these files.
Why not keep the new docker files for the existing machines right next to the existing ones?

So my thought on the directory structure was:

• /buildbot/google/
  • docker/
    • mlir-nvidia/
    • win32/
    • your-docker-dir/
    • some-other-bot/
    • <common scripting>

I'm fine with this part.

• terraform/
  • <one terraform config for all machines>

That's the part I'm not quite sure about. Terraform is an all-or-nothing kind of thing. I.e. terraform apply will try to set thing up for *everything* described in main.tf. I'm not convinced that we should mix all our bot setups here. I don't want MLIR folks and myself stepping on each other's toes and affecting each others' cluster setups. We may eventually consolidate them, but I don't think we're there yet.

Hence, I have my own terraform setup. We could structure it similarly to your suggestion for the docker files -- move current files under terraform/mlir and mine under terraform/cudabot, but with mlir and cudabot setups being split everywhere, we may as well maintain mlir and cudabot separately.

[mehdi_amini]

What is strange to me right now is the asymmetry:

google/docker/...
google/mlir/docker/...

google/terraform/
google/mlir/terraform/

[tra]

In D84256#2167664, @mehdi_amini wrote:

What is strange to me right now is the asymmetry:

google/docker/...
google/mlir/docker/...

google/terraform/
google/mlir/terraform/

I'm not sure what exactly you have in mind. Could you elaborate?
The stuff that's currently in google/ will move under google/mlir and there will be a new set of configs under google/cuda in the patch up the stack.

[mehdi_amini]

In D84256#2167804, @tra wrote:

I'm not sure what exactly you have in mind. Could you elaborate?
The stuff that's currently in google/ will move under google/mlir and there will be a new set of configs under google/cuda in the patch up the stack.

I don't have this context so I was just seeing this revision alone which setup google/mlir/docker/ and still has google/docker.
It makes sense if after your stack of revisions if there is no google/docker left.

[tra]

In D84256#2168058, @mehdi_amini wrote:

I don't have this context so I was just seeing this revision alone which setup google/mlir/docker/ and still has google/docker.
It makes sense if after your stack of revisions if there is no google/docker left.

File list is indeed somewhat hard to parse. After all the changes directory structure looks like this:

$ find buildbot/google -type d
buildbot/google
buildbot/google/cuda
buildbot/google/cuda/terraform
buildbot/google/cuda/docker
buildbot/google/cuda/docker/cudabot
buildbot/google/mlir
buildbot/google/mlir/terraform
buildbot/google/mlir/docker
buildbot/google/mlir/docker/buildbot-mlir-nvidia

[kuhnel]

Well, then let's go with your proposal of merging the docker directory and separate the terraform directory.
At least for now. If we later figure out a way to work in the same terraform config, we can always invest some time to merge that later on.

[tra]

Updated directory structure.

[tra]

Really updated directory structure.

[tra]

Phabricator seems to be confused by the renames. The change just moved terraform -> terraform/buildbot-mlir-nvidia
The directory structure looks like this now:

buildbot/google/terraform
buildbot/google/terraform/buildbot-mlir-nvidia
buildbot/google/docker
buildbot/google/docker/windows-base-vscode2019
buildbot/google/docker/buildbot-windows10-vs2019
buildbot/google/docker/buildbot-mlir-nvidia

buildbot/google/README.md

-# LLVM buildbot workers configuration
+# Configuration files for various buildbots run by Google.
 
-This folder contains some of the configuration of the buildbots managed
-at Google. The workers are deployed on Google Cloud.
-
-# The cloud stack
-
-To deploy build bots workers, we need to create a bunch of virtual machines 
-on Google Cloud. There are multiple ways to do this. *Terraform* is convenient 
-as it offers to declare the required machines in config files and then 
-create/update the machines in the cloud. 
-
-This way we have version control over the infrastructure 
-and we can review changes before applying them. In case something goes wrong, 
-we can easily revert changes. It also allows us to copy & paste parts of the 
-infrastructure for additional machines.
-
-Internally, Terraform is using *Kubernetes* to manage the deployment of software 
-to machines. The software installed on the build machines is defined
-in *Docker* images. An image is a (layered) file system with all the tools and
-settings required for the worker. 
-
-The images are stored in a "registry" (gcr.io in this case) and are then 
-pulled from the machines where they are executed. The 
-images can be versioned so that we can pick exactly which version of the image
-we want to run.
-
-The contents of a Docker image is again defined in a config file called 
-`Dockerfile`. A Dockerfile is a sort of script defining on how to install and
-configure the software for a machine. We keep those files in this repository as 
-well so we can review changes and revert changes if something breaks.
-
-The docker images also allow contributors to reproduce a failing test locally, 
-as they will get the same machine configuration as used on the server.
-
-# Folder structure
-
-* `docker` - Dockerfiles for the workers and some scripting
-* `terraform` - cluster configuration and deployment
-* `config.sh` - variables used in other scripts
-* `gcloud_config.sh` - configure cloud tooling
+* mlir

buildbot/google/mlir/README.md

 # LLVM buildbot workers configuration
 
 This folder contains some of the configuration of the buildbots managed
 at Google. The workers are deployed on Google Cloud.
 
 # The cloud stack
 
 To deploy build bots workers, we need to create a bunch of virtual machines 
 on Google Cloud. There are multiple ways to do this. *Terraform* is convenient 
 as it offers to declare the required machines in config files and then 
 create/update the machines in the cloud. 
 
 This way we have version control over the infrastructure 
 and we can review changes before applying them. In case something goes wrong, 
 we can easily revert changes. It also allows us to copy & paste parts of the 
 infrastructure for additional machines.
 
 Internally, Terraform is using *Kubernetes* to manage the deployment of software 
 to machines. The software installed on the build machines is defined
 in *Docker* images. An image is a (layered) file system with all the tools and
 settings required for the worker. 
 
 The images are stored in a "registry" (gcr.io in this case) and are then 
 pulled from the machines where they are executed. The 
 images can be versioned so that we can pick exactly which version of the image
 we want to run.
 
 The contents of a Docker image is again defined in a config file called 
 `Dockerfile`. A Dockerfile is a sort of script defining on how to install and
 configure the software for a machine. We keep those files in this repository as 
 well so we can review changes and revert changes if something breaks.
 
 The docker images also allow contributors to reproduce a failing test locally, 
 as they will get the same machine configuration as used on the server.
 
 # Folder structure
 
 * `docker` - Dockerfiles for the workers and some scripting
 * `terraform` - cluster configuration and deployment
 * `config.sh` - variables used in other scripts
 * `gcloud_config.sh` - configure cloud tooling

buildbot/google/mlir/config.sh

 # config parameters for the Google Cloud, this is used by other scripts
 GCP_PROJECT="sanitizer-bots"
 GCP_ZONE="us-central1-a"
 GCP_CLUSTER="buildbot-cluster"
 GCR_PREFIX="gcr.io/${GCP_PROJECT}"

buildbot/google/mlir/docker/README.md

 This folder contains the Dockerfiles and scripts used for some of the 
 buildbot workers. 
 
 # Scripts
 
 This folder also contains some scripts that are useful in working with the
 docker images.
 
 ## build_run.sh
 Build a docker image and run it locally
 
 ## build_deploy.sh
 Build a docker image, increment the version number, tag it and upload it to
 the registry. This updates the `VERSION` file to track the version numbers.
 
 # Secrets
 
 The buildbot workers need a password to authenticate with the buildbot server. 
 This password needs to be kept a secert. The usual way to handle it is to store
 the secrets in a secure place and during runtime mount that secure place into 
 the container. The secret file shall just contain the password in plain text.
 
 Kubernetes offers a [mechanism to handle secrets](https://kubernetes.io/docs/concepts/configuration/secret/).

buildbot/google/mlir/docker/build_deploy.sh

 #!/bin/bash
 #===-- build_deploy.sh ---------------------------------------------------===//
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 #
 #===----------------------------------------------------------------------===//
 # This script will deploy a docker image to the registry.
 # Arguments: <path to Dockerfile> 
 # 
 # This updates the `VERSION` file with the latest version number.
 #===----------------------------------------------------------------------===//
 
 set -eu
 
 DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
 IMAGE_NAME="${1%/}"
 
 # increment version number
 cd "${DIR}/${IMAGE_NAME}"
 
 # get version numbers from repository
 # FIXME: use variables to configure URL
 ALL_VERSIONS=$(gcloud container images list-tags gcr.io/sanitizer-bots/${IMAGE_NAME} --format=text | \
   awk '/tags.*:\W+[0-9]+$/ {print $2}')
 # read local version number from file and add it to the array
 ALL_VERSIONS+=($(cat VERSION))
 # find maximum version number and increment it
 VERSION=$(echo "${ALL_VERSIONS[*]}" | sort -nr | head -n1)
 VERSION=$(( ${VERSION} + 1 ))
 
 # get the git hash and add some suffixes
 GIT_HASH=$(git rev-parse HEAD)
 if [[ $(git diff --stat) != '' ]]; then
   # if working copy is dirty
   GIT_HASH+="-dirty-${USER}"
 elif [[ $(git --no-pager diff origin/master | wc -l) > 0 ]]; then
   # if the hash has not been uploaded to origin/master yet
   GIT_HASH+="-local-${USER}"
 fi
 
 # fully qualified image name
 # FIXME: use variables to configure URL
 QUALIFIED_NAME="gcr.io/sanitizer-bots/${IMAGE_NAME}"
 # tags to be added to the image and pushed to the repository
 TAGS=(
   "${QUALIFIED_NAME}:latest" 
   "${QUALIFIED_NAME}:${VERSION}"
   "${QUALIFIED_NAME}:${GIT_HASH}"
   )
 
 # build the image and tag it locally
 docker build -t ${IMAGE_NAME}:latest -t ${IMAGE_NAME}:${VERSION} .
 
 # print the list of tags to be pushed
 echo "-----------------------------------------"
 echo "image version: ${VERSION}"
 echo "tags:"
 printf '   %s\n' "${TAGS[@]}"
 echo "-----------------------------------------"
 read -p "Push to registry? [yN]" -n 1 -r
 echo
 
 if [[ $REPLY =~ ^[Yy]$ ]]
 then
   for TAG in "${TAGS[@]}"
   do
     docker tag ${IMAGE_NAME}:${VERSION} "${TAG}"
     docker push "${TAG}"
   done
   # store the version number
   echo "${VERSION}" > VERSION
 fi

buildbot/google/mlir/docker/build_run.sh

 #!/bin/bash
 #===-- build_run.sh ------------------------------------------------------===//
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 #
 #===----------------------------------------------------------------------===//
 # This script will deploy a docker image to the registry.
 # Arguments:
 #     <path to Dockerfile>
 #     <path containing secrets>
 #     optional: <command to be executed in the container>
 #===----------------------------------------------------------------------===//
 
 set -eux
 
 DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
 IMAGE_NAME="${1%/}"
 SECRET_STORAGE="$2"
 CMD=
 if [ "$#" -eq 3 ];
 then
     CMD="$3"
 fi
 
 cd "${DIR}/${IMAGE_NAME}"
 
 docker build -t "${IMAGE_NAME}:latest" .
 docker run -it -v "${SECRET_STORAGE}":/secrets "${IMAGE_NAME}" ${CMD}

buildbot/google/mlir/docker/buildbot-mlir-nvidia/Dockerfile

 #===-- Dockerfile --------------------------------------------------------===//
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 #
 #===----------------------------------------------------------------------===//
 # Docker image used for the mlir-nvidia builder
 #===----------------------------------------------------------------------===//
 
 # Use the image from NVIDIA as base
 FROM nvidia/cuda:10.2-base-ubuntu18.04
 
 
 # install build tools
 RUN apt-get update; \
     apt-get install -y software-properties-common apt-transport-https ca-certificates \
       clang-8 lld-8 ninja-build git wget gnupg ccache \
       python python-pip python-psutil ;\
     update-alternatives --install /usr/bin/clang clang /usr/bin/clang-8 100 ;\
     update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-8 100 ;\
     update-alternatives --install /usr/bin/lld lld /usr/bin/lld-8 100
 
 # install cuda
 # avoid popups for keyboard configurations
 RUN DEBIAN_FRONTEND=noninteractive apt-get install -y cuda
 
 # Ubuntu ships with old cmake version, install the latest one
 # from https://apt.kitware.com/
 RUN wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | \
        gpg --dearmor - | \
        tee /etc/apt/trusted.gpg.d/kitware.gpg >/dev/null ;\
     apt-add-repository 'deb https://apt.kitware.com/ubuntu/ bionic main' ;\
     apt-get update ;\
     apt-get install -y cmake
 
 # install (old) build bot version
 # this version of build bot requires python2!
 RUN pip install buildbot-slave==0.8.11
 
 # Volume to mount secrets into the container
 VOLUME /secrets
 
 # create user account, some test fail if run as root
 RUN useradd buildbot --create-home
 WORKDIR /home/buildbot
 
 # copy startup script
 COPY run.sh /home/buildbot/
 RUN chmod a+rx /home/buildbot/run.sh
 
 USER buildbot
 ENV WORKER_NAME="mlir-nvidia"
 
 # Set up buildbot host and maintainer info.
 RUN mkdir -p "${WORKER_NAME}/info/" ;\
     echo "Christian Kühnel <[e-mail]>" > "${WORKER_NAME}/info/admin"
 
 CMD ./run.sh

buildbot/google/mlir/docker/buildbot-mlir-nvidia/VERSION

 9

buildbot/google/mlir/docker/buildbot-mlir-nvidia/run.sh

 #!/bin/bash
 #===-- run.sh -------------------------------------------------------------===//
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 #
 #===----------------------------------------------------------------------===//
 # This script will start the buildbot worker
 #===----------------------------------------------------------------------===//
 
 set -eu
 
 # Read the worker password from a mounted file.
 WORKER_PASSWORD=$(cat /secrets/token)
 
 # generate the host information of this worker
 ( 
   uname -a ; \
   cat /proc/cpuinfo | grep "model name" | head -n1 | cut -d " " -f 3- ;\
   echo "number of cores: $(nproc)" ;\
   nvidia-smi -L | cut -d "(" -f 1 ;\
   lsb_release -d | cut -f 2- ; \
   clang --version | head -n1 ; \
   ld.lld-8 --version ; \
   cmake --version | head -n1 
 ) > ${WORKER_NAME}/info/host 
 
 # It looks like GKE sometimes deploys the container before the NVIDIA drivers 
 # are loaded on the host. In this case the GPU is not available during the 
 # entire lifecycle of the container. Not sure how to fix this properly. 
 
 RETURN_CODE=$(nvidia-smi > /dev/null ; echo $?)
 if [[ "$RETURN_CODE" != "0" ]] ; then
   echo "ERROR: Failed to access NVIDIA graphics card."
   echo "Exiting in 30 secs..."
   sleep 30
   exit 1
 fi
 
 # create the folder structure
 # port 9990 for production
 # port 9994 for staging
 buildslave create-slave --keepalive=200 "${WORKER_NAME}" \
   lab.llvm.org:9990 "${WORKER_NAME}" "${WORKER_PASSWORD}"
 
 # start the daemon, this command return immetiately
 buildslave start "${WORKER_NAME}"
 
 # To keep the container running and produce log outputs: dump the worker
 # log to stdout
 tail -f ${WORKER_NAME}/twistd.log

buildbot/google/mlir/docker/pod_login.sh

 #! /bin/bash
 #===-- pod_login.sh ------------------------------------------------------===//
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 #
 #===----------------------------------------------------------------------===//
 # This script will give you a terminal in on of the pods. This is useful
 # to debug problems in a running container. This is much faster than going
 # through the Google Cloud UI.
 #  
 # Arguments:
 #     pod name fragement : 
 #             part of the name of the pod to log in, eg. name of the 
 #             deployment, if we have only one of them
 #===----------------------------------------------------------------------===//
 
 set -eu
 
 WORKLOAD_NAME=$1
 # get name of the pod
 POD=$(kubectl get pod -o name | grep "$1")
 
 # FIXME: exit if more than one pod is returned
 
 # login to the pod
 kubectl exec --stdin --tty "${POD}" -- /bin/bash

buildbot/google/mlir/docker/pod_logs.sh

 #! /bin/bash
 #===-- pod_login.sh ------------------------------------------------------===//
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 #
 #===----------------------------------------------------------------------===//
 # This script will give get the console logs form the container. 
 #  
 # Arguments:
 #     pod name fragement : 
 #             part of the name of the pod to log in, eg. name of the 
 #             deployment, if we have only one of them
 #===----------------------------------------------------------------------===//
 
 set -eu
 
 WORKLOAD_NAME=$1
 # get name of the pod
 POD=$(kubectl get pod -o name | grep "$1")
 
 # FIXME: exit if more than one pod is returned
 
 # fetch the logd for the pod
 kubectl logs "${POD}"

buildbot/google/mlir/gcloud_config.sh

 #!/bin/bash
 set -eux
 
 ROOT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
 
 # load project configuration
 source "${ROOT_DIR}/config.sh"
 
 gcloud config set project ${GCP_PROJECT}
 gcloud config set compute/zone ${GCP_ZONE}
 gcloud auth configure-docker
 gcloud container clusters get-credentials $GCP_CLUSTER

buildbot/google/mlir/terraform/README.md

 This folder contains the Terraform configuration to spawn the build bots.
 
 Before deploying anything new, use `terraform plan` to check that you're only 
 modifying the parts that you intended to.
 
 
 # Installation
 
 To set up your local machine to deploy changes to the cluster follow these 
 steps:
 
 1. Install these tools:
     1. [Terraform](https://learn.hashicorp.com/terraform/getting-started/install.html)
     1. [Google Cloud SDK](https://cloud.google.com/sdk/install)
     1. [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
 1. Run `llvm-zorg/buildbot/google/gcloud_config.sh` to configure the Google
    Cloud SDK.
 1. To configure the GCP credetianls for terraform run: 
    ```bash
     export GOOGLE_CREDENTIALS=~/.config/gcloud/legacy_credentials/<your email>/adc.json
     ```
 
 # Deploying to new Google Cloud project
 
 When deploying this cluster to a completely new Google Cloud project, these 
 manual steps are required:
 
 * You need to create the GCP project manually before Terraform works.
 * You also need to go to the Kubernetes page once, to enable Kubernetes and 
   Container Registry for that project.
 * GPUs need to be enabled on Kubernetes by following these
 [instructions](https://cloud.google.com/kubernetes-engine/docs/how-to/gpus#installing_drivers).
 
 
 # Secrets
 
 To keep secrets a secret, they MUST not be stored in version control. The right
 place on kubernetes is a "secret". To create a kubernetes secret for the agent
 token: 
 ```bash
 kubectl create secret generic buildbot-token-mlir-nvidia --from-file=token=<file name>
 ```
 The file in `<file name>` then must contain the password of the buildbot worker
 in plain text. In the `Deployment` of a container, the secret is defined as a 
 special type of volume and mounted in the specified path. During runtime the 
 secret can then be read from that file.
 
 An example:
 The secret `buildbot-token-mlir-nvidia` is defined (as above) in Kubernetes. 
 In the [deployment](buildbot/google/terraform/main.tf) `mlir-nvidia` it is 
 used as a volume of type `secret` and then mounted at `/secrets`. During the 
 runtime of the docker container, the script 
 [run.sh](../docker/buildbot-mlir-nvidia/run.sh) reads the secret from the file
 `/secrets/token` and uses it to create the worker configuration.
 
 
 # Using GPUs on Google Cloud
 
 Terraform does not support deployments on GCP using a GPU at the moment.
 So we need to deploy such cases using plain Kubernetes configuration files.
 See this [issue](https://github.com/terraform-providers/terraform-provider-kubernetes/issues/149) 
 for more details.
 The buildbot mlir-nvidia is configured in `deployment-mlir-nvidia.yaml` in this
 folder. 
 
 For all non-GPU cases add a `"kubernetes_deployment"` to `main.tf`. 
 The contents is identical to the the Kubernetes file, just the markup is 
 different.
 
 Kubernetes files are also declarative, so you can re-deploy them when you made
 a change. They can be deployed with:
 ```bash
 kubectl apply -f myfile.yaml
 ```

buildbot/google/mlir/terraform/deployment-mlir-nvidia.yaml

 ---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: mlir-nvidia
 spec:
   # number of instances we want to run
   replicas: 1
   selector:
     matchLabels:
       app: buildbot-mlir-nvidia
   # define strategy for updating the images
   strategy:
     rollingUpdate:
       # do not deploy more replicas, as the buildbot server 
       # can't handle multiple workers with the same credentials
       maxSurge: 0
       # Allow to have 0 replicas during updates. 
       maxUnavailable: 1
     type: RollingUpdate
   template:
     metadata:
       labels:
         app: buildbot-mlir-nvidia
     spec:
       containers:
       # the image and version we want to run   
       - image: gcr.io/sanitizer-bots/buildbot-mlir-nvidia:9
         name: mlir-nvidia
         # reserve "<number of cores>-1" for this image, kubernetes also
         # needs <1 core for management tools
         resources:
           limits:
             cpu: "15"
             memory: 10Gi
             # also request to use the GPU
             # Note: this does not work in terraform at the moment
             nvidia.com/gpu: "1"
           requests:
             cpu: "15"
             memory: 10Gi
             nvidia.com/gpu: "1"
         volumeMounts:
         # mount the secrets into a folder  
         - mountPath: /secrets
           mountPropagation: None
           name: buildbot-token
       # specify the nood pool on which to deploy
       nodeSelector:
         pool: nvidia-16core-pool
       restartPolicy: Always
       # FIXME: do we need this if we requested a GPU?
       #tolerations:
       #- effect: NoSchedule
       #  key: nvidia.com/gpu
       #  operator: Equal
       #  value: present
       volumes:
       # declare the secret as a volume so we can mount it
       - name: buildbot-token
         secret:
           optional: false
           secretName: password-mlir-nvidia

buildbot/google/mlir/terraform/main.tf

 
 # configure Google Cloud project
 provider "google" {
   project  = var.gcp_config.project
   region   = var.gcp_config.region
 }
 
 # Create the cluster runningn all Kubernetes services
 resource "google_container_cluster" "primary" {
   name     = "buildbot-cluster"
   # maybe have a regional cluster for Kubernetes, as we depend on this...
   location = var.gcp_config.zone_a
 
   # one node is enough (at the moment)
   initial_node_count = 1
 
   node_config {
     # FIXME(kuhnel): turn this into a private cluster, without external IP
     # We need at least 2 vCPU to run all kubernetes services
     machine_type = "n1-standard-2"
     # use preemptible, as this saves costs
     preemptible = true
   }
 }
 
 # Create machines for mlir-nvidia
 # Note: The buildbot mlir-nividia is deployed using a kubernetes file. See
 # the README.md for details on GPUs.
 
 resource "google_container_node_pool" "nvidia_16core_pool_nodes" {
   name       = "nvidia-16core-pool"
   # specify a zone here (e.g. "-a") to avoid a redundant deployment
   location   = var.gcp_config.zone_a
   cluster    = google_container_cluster.primary.name
   
   # use autoscaling to only create a machine when there is a deployment
   autoscaling {
     min_node_count = 0
     max_node_count = 1
   }
   
   node_config {
     # use preemptible, as this saves costs
     preemptible  = true
     machine_type = "n1-highcpu-16"
     disk_size_gb = 100
     # FIXME: test if SSDs are actually faster than HDDs for our use case
     disk_type = "pd-ssd"
     guest_accelerator {
       type = "nvidia-tesla-t4"
       count= 1
     }
 
     # set the premissions required for the deployment later
     oauth_scopes = [
       "https://www.googleapis.com/auth/logging.write",
       "https://www.googleapis.com/auth/monitoring",
       "https://www.googleapis.com/auth/devstorage.read_only",
     ]
 
     # add a label to all machines of this type, so we can select them 
     # during deployment
     labels = {
       pool = "nvidia-16core-pool"
     }
   }
 }

buildbot/google/mlir/terraform/variables.tf

 # configuration parameter for Google Cloud
 variable "gcp_config" {
   type = object({
     project     = string
     region      = string
     zone_a      = string
     gcr_prefix  = string
    })
   default = {
     project     = "sanitizer-bots"
     region      = "us-central1"
     zone_a      = "us-central1-a"
     gcr_prefix  = "gcr.io/sanitizer-bots"
   }
 }