[buildbot] Moved MLIR buildbot config under buildbot/google/mlir

[tra]

Summary

buildbot/google itself will also hold configs for other bots owned by Google.

[kuhnel]

Sorry I did not understand why do you want to move these files.
Why not keep the new docker files for the existing machines right next to the existing ones?

So my thought on the directory structure was:

• /buildbot/google/
  • docker/
    • mlir-nvidia/
    • win32/
    • your-docker-dir/
    • some-other-bot/
    • <common scripting>
  • terraform/
    • <one terraform config for all machines>

[tra]

In D84256#2166061, @kuhnel wrote:

Sorry I did not understand why do you want to move these files.
Why not keep the new docker files for the existing machines right next to the existing ones?

So my thought on the directory structure was:

• /buildbot/google/
  • docker/
    • mlir-nvidia/
    • win32/
    • your-docker-dir/
    • some-other-bot/
    • <common scripting>

I'm fine with this part.

• terraform/
  • <one terraform config for all machines>

That's the part I'm not quite sure about. Terraform is an all-or-nothing kind of thing. I.e. terraform apply will try to set thing up for *everything* described in main.tf. I'm not convinced that we should mix all our bot setups here. I don't want MLIR folks and myself stepping on each other's toes and affecting each others' cluster setups. We may eventually consolidate them, but I don't think we're there yet.

Hence, I have my own terraform setup. We could structure it similarly to your suggestion for the docker files -- move current files under terraform/mlir and mine under terraform/cudabot, but with mlir and cudabot setups being split everywhere, we may as well maintain mlir and cudabot separately.

[mehdi_amini]

What is strange to me right now is the asymmetry:

google/docker/...
google/mlir/docker/...

google/terraform/
google/mlir/terraform/

[tra]

In D84256#2167664, @mehdi_amini wrote:

What is strange to me right now is the asymmetry:

google/docker/...
google/mlir/docker/...

google/terraform/
google/mlir/terraform/

I'm not sure what exactly you have in mind. Could you elaborate?
The stuff that's currently in google/ will move under google/mlir and there will be a new set of configs under google/cuda in the patch up the stack.

[mehdi_amini]

In D84256#2167804, @tra wrote:

I'm not sure what exactly you have in mind. Could you elaborate?
The stuff that's currently in google/ will move under google/mlir and there will be a new set of configs under google/cuda in the patch up the stack.

I don't have this context so I was just seeing this revision alone which setup google/mlir/docker/ and still has google/docker.
It makes sense if after your stack of revisions if there is no google/docker left.

[tra]

In D84256#2168058, @mehdi_amini wrote:

I don't have this context so I was just seeing this revision alone which setup google/mlir/docker/ and still has google/docker.
It makes sense if after your stack of revisions if there is no google/docker left.

File list is indeed somewhat hard to parse. After all the changes directory structure looks like this:

$ find buildbot/google -type d
buildbot/google
buildbot/google/cuda
buildbot/google/cuda/terraform
buildbot/google/cuda/docker
buildbot/google/cuda/docker/cudabot
buildbot/google/mlir
buildbot/google/mlir/terraform
buildbot/google/mlir/docker
buildbot/google/mlir/docker/buildbot-mlir-nvidia

[kuhnel]

Well, then let's go with your proposal of merging the docker directory and separate the terraform directory.
At least for now. If we later figure out a way to work in the same terraform config, we can always invest some time to merge that later on.

[tra]

Updated directory structure.

[tra]

Really updated directory structure.

[tra]

Phabricator seems to be confused by the renames. The change just moved terraform -> terraform/buildbot-mlir-nvidia
The directory structure looks like this now:

buildbot/google/terraform
buildbot/google/terraform/buildbot-mlir-nvidia
buildbot/google/docker
buildbot/google/docker/windows-base-vscode2019
buildbot/google/docker/buildbot-windows10-vs2019
buildbot/google/docker/buildbot-mlir-nvidia

buildbot/google/terraform/buildbot-mlir-nvidia/README.md

 This folder contains the Terraform configuration to spawn the build bots.
 
 Before deploying anything new, use `terraform plan` to check that you're only 
 modifying the parts that you intended to.
 
 
 # Installation
 
 To set up your local machine to deploy changes to the cluster follow these 
 steps:
 
 1. Install these tools:
     1. [Terraform](https://learn.hashicorp.com/terraform/getting-started/install.html)
     1. [Google Cloud SDK](https://cloud.google.com/sdk/install)
     1. [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
 1. Run `llvm-zorg/buildbot/google/gcloud_config.sh` to configure the Google
    Cloud SDK.
 1. To configure the GCP credetianls for terraform run: 
    ```bash
     export GOOGLE_CREDENTIALS=~/.config/gcloud/legacy_credentials/<your email>/adc.json
     ```
 
 # Deploying to new Google Cloud project
 
 When deploying this cluster to a completely new Google Cloud project, these 
 manual steps are required:
 
 * You need to create the GCP project manually before Terraform works.
 * You also need to go to the Kubernetes page once, to enable Kubernetes and 
   Container Registry for that project.
 * GPUs need to be enabled on Kubernetes by following these
 [instructions](https://cloud.google.com/kubernetes-engine/docs/how-to/gpus#installing_drivers).
 
 
 # Secrets
 
 To keep secrets a secret, they MUST not be stored in version control. The right
 place on kubernetes is a "secret". To create a kubernetes secret for the agent
 token: 
 ```bash
 kubectl create secret generic buildbot-token-mlir-nvidia --from-file=token=<file name>
 ```
 The file in `<file name>` then must contain the password of the buildbot worker
 in plain text. In the `Deployment` of a container, the secret is defined as a 
 special type of volume and mounted in the specified path. During runtime the 
 secret can then be read from that file.
 
 An example:
 The secret `buildbot-token-mlir-nvidia` is defined (as above) in Kubernetes. 
 In the [deployment](buildbot/google/terraform/main.tf) `mlir-nvidia` it is 
 used as a volume of type `secret` and then mounted at `/secrets`. During the 
 runtime of the docker container, the script 
 [run.sh](../docker/buildbot-mlir-nvidia/run.sh) reads the secret from the file
 `/secrets/token` and uses it to create the worker configuration.
 
 
 # Using GPUs on Google Cloud
 
 Terraform does not support deployments on GCP using a GPU at the moment.
 So we need to deploy such cases using plain Kubernetes configuration files.
 See this [issue](https://github.com/terraform-providers/terraform-provider-kubernetes/issues/149) 
 for more details.
 The buildbot mlir-nvidia is configured in `deployment-mlir-nvidia.yaml` in this
 folder. 
 
 For all non-GPU cases add a `"kubernetes_deployment"` to `main.tf`. 
 The contents is identical to the the Kubernetes file, just the markup is 
 different.
 
 Kubernetes files are also declarative, so you can re-deploy them when you made
 a change. They can be deployed with:
 ```bash
 kubectl apply -f myfile.yaml
 ```

buildbot/google/terraform/buildbot-mlir-nvidia/deployment-mlir-nvidia.yaml

 ---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: mlir-nvidia
 spec:
   # number of instances we want to run
   replicas: 1
   selector:
     matchLabels:
       app: buildbot-mlir-nvidia
   # define strategy for updating the images
   strategy:
     rollingUpdate:
       # do not deploy more replicas, as the buildbot server 
       # can't handle multiple workers with the same credentials
       maxSurge: 0
       # Allow to have 0 replicas during updates. 
       maxUnavailable: 1
     type: RollingUpdate
   template:
     metadata:
       labels:
         app: buildbot-mlir-nvidia
     spec:
       containers:
       # the image and version we want to run   
       - image: gcr.io/sanitizer-bots/buildbot-mlir-nvidia:9
         name: mlir-nvidia
         # reserve "<number of cores>-1" for this image, kubernetes also
         # needs <1 core for management tools
         resources:
           limits:
             cpu: "15"
             memory: 10Gi
             # also request to use the GPU
             # Note: this does not work in terraform at the moment
             nvidia.com/gpu: "1"
           requests:
             cpu: "15"
             memory: 10Gi
             nvidia.com/gpu: "1"
         volumeMounts:
         # mount the secrets into a folder  
         - mountPath: /secrets
           mountPropagation: None
           name: buildbot-token
       # specify the nood pool on which to deploy
       nodeSelector:
         pool: nvidia-16core-pool
       restartPolicy: Always
       # FIXME: do we need this if we requested a GPU?
       #tolerations:
       #- effect: NoSchedule
       #  key: nvidia.com/gpu
       #  operator: Equal
       #  value: present
       volumes:
       # declare the secret as a volume so we can mount it
       - name: buildbot-token
         secret:
           optional: false
           secretName: password-mlir-nvidia

buildbot/google/terraform/buildbot-mlir-nvidia/main.tf

 
 # configure Google Cloud project
 provider "google" {
   project  = var.gcp_config.project
   region   = var.gcp_config.region
 }
 
 # Create the cluster runningn all Kubernetes services
 resource "google_container_cluster" "primary" {
   name     = "buildbot-cluster"
   # maybe have a regional cluster for Kubernetes, as we depend on this...
   location = var.gcp_config.zone_a
 
   # one node is enough (at the moment)
   initial_node_count = 1
 
   node_config {
     # FIXME(kuhnel): turn this into a private cluster, without external IP
     # We need at least 2 vCPU to run all kubernetes services
     machine_type = "n1-standard-2"
     # use preemptible, as this saves costs
     preemptible = true
   }
 }
 
 # Create machines for mlir-nvidia
 # Note: The buildbot mlir-nividia is deployed using a kubernetes file. See
 # the README.md for details on GPUs.
 
 resource "google_container_node_pool" "nvidia_16core_pool_nodes" {
   name       = "nvidia-16core-pool"
   # specify a zone here (e.g. "-a") to avoid a redundant deployment
   location   = var.gcp_config.zone_a
   cluster    = google_container_cluster.primary.name
   
   # use autoscaling to only create a machine when there is a deployment
   autoscaling {
     min_node_count = 0
     max_node_count = 1
   }
   
   node_config {
     # use preemptible, as this saves costs
     preemptible  = true
     machine_type = "n1-highcpu-16"
     disk_size_gb = 100
     # FIXME: test if SSDs are actually faster than HDDs for our use case
     disk_type = "pd-ssd"
     guest_accelerator {
       type = "nvidia-tesla-t4"
       count= 1
     }
 
     # set the premissions required for the deployment later
     oauth_scopes = [
       "https://www.googleapis.com/auth/logging.write",
       "https://www.googleapis.com/auth/monitoring",
       "https://www.googleapis.com/auth/devstorage.read_only",
     ]
 
     # add a label to all machines of this type, so we can select them 
     # during deployment
     labels = {
       pool = "nvidia-16core-pool"
     }
   }
 }

buildbot/google/terraform/buildbot-mlir-nvidia/variables.tf

 # configuration parameter for Google Cloud
 variable "gcp_config" {
   type = object({
     project     = string
     region      = string
     zone_a      = string
     gcr_prefix  = string
    })
   default = {
     project     = "sanitizer-bots"
     region      = "us-central1"
     zone_a      = "us-central1-a"
     gcr_prefix  = "gcr.io/sanitizer-bots"
   }
 }