[buildbot] Moved MLIR buildbot config under buildbot/google/mlir

[tra]

Summary

buildbot/google itself will also hold configs for other bots owned by Google.

[kuhnel]

Sorry I did not understand why do you want to move these files.
Why not keep the new docker files for the existing machines right next to the existing ones?

So my thought on the directory structure was:

• /buildbot/google/
  • docker/
    • mlir-nvidia/
    • win32/
    • your-docker-dir/
    • some-other-bot/
    • <common scripting>
  • terraform/
    • <one terraform config for all machines>

[tra]

In D84256#2166061, @kuhnel wrote:

Sorry I did not understand why do you want to move these files.
Why not keep the new docker files for the existing machines right next to the existing ones?

So my thought on the directory structure was:

• /buildbot/google/
  • docker/
    • mlir-nvidia/
    • win32/
    • your-docker-dir/
    • some-other-bot/
    • <common scripting>

I'm fine with this part.

• terraform/
  • <one terraform config for all machines>

That's the part I'm not quite sure about. Terraform is an all-or-nothing kind of thing. I.e. terraform apply will try to set thing up for *everything* described in main.tf. I'm not convinced that we should mix all our bot setups here. I don't want MLIR folks and myself stepping on each other's toes and affecting each others' cluster setups. We may eventually consolidate them, but I don't think we're there yet.

Hence, I have my own terraform setup. We could structure it similarly to your suggestion for the docker files -- move current files under terraform/mlir and mine under terraform/cudabot, but with mlir and cudabot setups being split everywhere, we may as well maintain mlir and cudabot separately.

[mehdi_amini]

What is strange to me right now is the asymmetry:

google/docker/...
google/mlir/docker/...

google/terraform/
google/mlir/terraform/

[tra]

In D84256#2167664, @mehdi_amini wrote:

What is strange to me right now is the asymmetry:

google/docker/...
google/mlir/docker/...

google/terraform/
google/mlir/terraform/

I'm not sure what exactly you have in mind. Could you elaborate?
The stuff that's currently in google/ will move under google/mlir and there will be a new set of configs under google/cuda in the patch up the stack.

[mehdi_amini]

In D84256#2167804, @tra wrote:

I'm not sure what exactly you have in mind. Could you elaborate?
The stuff that's currently in google/ will move under google/mlir and there will be a new set of configs under google/cuda in the patch up the stack.

I don't have this context so I was just seeing this revision alone which setup google/mlir/docker/ and still has google/docker.
It makes sense if after your stack of revisions if there is no google/docker left.

[tra]

In D84256#2168058, @mehdi_amini wrote:

I don't have this context so I was just seeing this revision alone which setup google/mlir/docker/ and still has google/docker.
It makes sense if after your stack of revisions if there is no google/docker left.

File list is indeed somewhat hard to parse. After all the changes directory structure looks like this:

$ find buildbot/google -type d
buildbot/google
buildbot/google/cuda
buildbot/google/cuda/terraform
buildbot/google/cuda/docker
buildbot/google/cuda/docker/cudabot
buildbot/google/mlir
buildbot/google/mlir/terraform
buildbot/google/mlir/docker
buildbot/google/mlir/docker/buildbot-mlir-nvidia

[kuhnel]

Well, then let's go with your proposal of merging the docker directory and separate the terraform directory.
At least for now. If we later figure out a way to work in the same terraform config, we can always invest some time to merge that later on.

[tra]

Updated directory structure.

[tra]

Really updated directory structure.

[tra]

Phabricator seems to be confused by the renames. The change just moved terraform -> terraform/buildbot-mlir-nvidia
The directory structure looks like this now:

buildbot/google/terraform
buildbot/google/terraform/buildbot-mlir-nvidia
buildbot/google/docker
buildbot/google/docker/windows-base-vscode2019
buildbot/google/docker/buildbot-windows10-vs2019
buildbot/google/docker/buildbot-mlir-nvidia

buildbot/google/docker/README.md

 # Secrets
 
 The buildbot workers need a password to authenticate with the buildbot server. 
 This password needs to be kept a secert. The usual way to handle it is to store
 the secrets in a secure place and during runtime mount that secure place into 
 the container. The secret file shall just contain the password in plain text.
 
 Kubernetes offers a [mechanism to handle secrets](https://kubernetes.io/docs/concepts/configuration/secret/).
+
+# Setting up Windows VM for development
+See [windows.md](windows.md).

buildbot/google/docker/build_deploy.sh

 # 
 # This updates the `VERSION` file with the latest version number.
 #===----------------------------------------------------------------------===//
 
 set -eu
 
 DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
 IMAGE_NAME="${1%/}"
-
-# increment version number
 cd "${DIR}/${IMAGE_NAME}"
 
-# get version numbers from repository
-# FIXME: use variables to configure URL
-ALL_VERSIONS=$(gcloud container images list-tags gcr.io/sanitizer-bots/${IMAGE_NAME} --format=text | \
-  awk '/tags.*:\W+[0-9]+$/ {print $2}')
-# read local version number from file and add it to the array
-ALL_VERSIONS+=($(cat VERSION))
-# find maximum version number and increment it
-VERSION=$(echo "${ALL_VERSIONS[*]}" | sort -nr | head -n1)
-VERSION=$(( ${VERSION} + 1 ))
+# read local version number from file and increment it
+# TODO: maybe also get latest version number from repository and take the 
+#       maximum of local and repo versions.
+VERSION=$(( $(cat VERSION) + 1 ))
+
+# on Windows: USER=$USERNAME
+if [[ "${OS}" == "Windows_NT" ]] ; then
+  USER="${USERNAME}"
+fi
 
 # get the git hash and add some suffixes
 GIT_HASH=$(git rev-parse HEAD)
 if [[ $(git diff --stat) != '' ]]; then
   # if working copy is dirty
   GIT_HASH+="-dirty-${USER}"
 elif [[ $(git --no-pager diff origin/master | wc -l) > 0 ]]; then
   # if the hash has not been uploaded to origin/master yet

buildbot/google/docker/build_run.sh

 #!/bin/bash
 #===-- build_run.sh ------------------------------------------------------===//
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 #
 #===----------------------------------------------------------------------===//
 # This script will deploy a docker image to the registry.
 # Arguments:
 #     <path to Dockerfile>
 #     <path containing secrets>
 #     optional: <command to be executed in the container>
 #===----------------------------------------------------------------------===//
 
-set -eux
+set -eu
 
 DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
 IMAGE_NAME="${1%/}"
 SECRET_STORAGE="$2"
 CMD=
 if [ "$#" -eq 3 ];
 then
     CMD="$3"
 fi
 
 cd "${DIR}/${IMAGE_NAME}"
 
+# Mount a volume "workertest" to persit across test runs.
+# Use this to keep e.g.  a git checkout or partial build across runs
+if [[ $(docker volume ls | grep workertest | wc -l) == 0 ]] ; then
+    docker volume create workertest
+fi
+
+# Volume to presist the build cache e.g. ccache or sccache.
+# This will speed up local testing.
+if [[ $(docker volume ls | grep workercache | wc -l) == 0 ]] ; then
+    docker volume create workercache
+fi
+
+# Define arguments for mounting the volumes
+# These differ on Windows and Linux
+VOLUMES="-v ${SECRET_STORAGE}:/secrets -v workertest:/test"
+if [[ "${OS}" == "Windows_NT" ]] ; then
+    VOLUMES="-v ${SECRET_STORAGE}:c:\\volumes\\secrets -v workertest:c:\volumes\\test -v workercache:c:\sccache"
+fi
+
 docker build -t "${IMAGE_NAME}:latest" .
-docker run -it -v "${SECRET_STORAGE}":/secrets "${IMAGE_NAME}" ${CMD}
+docker run -it ${VOLUMES} "${IMAGE_NAME}:latest" ${CMD}

buildbot/google/docker/buildbot-windows10-vs2019/Dockerfile

+# escape=`
+FROM gcr.io/sanitizer-bots/windows-base-vscode2019:5
+
+ENV WORKER_NAME=windows10-vs2019
+
+# copy admin information
+RUN md %WORKER_NAME%\info
+COPY admin ${WORKER_NAME}\info
+
+# copy script to start the agent
+COPY run.ps1 .
+
+# Set location for sccache cache
+ENV SCCACHE_DIR="C:\volumes\sccache"
+
+# Set the maximum sccache local cache size
+ENV SCCACHE_CACHE_SIZE="5G"
+
+# Configure 32bit tools ("x86") instead of 64bit ("amd64")
+ENTRYPOINT ["C:\\BuildTools\\Common7\\Tools\\VsDevCmd.bat", "-arch=x86", "-host_arch=amd64", "&&", "powershell.exe", "-NoLogo", "-ExecutionPolicy", "Bypass"]
+
+CMD ["c:\\buildbot\\run.ps1"]

buildbot/google/docker/buildbot-windows10-vs2019/VERSION

+4

buildbot/google/docker/buildbot-windows10-vs2019/admin

+Christian Kühnel <[e-mail]>

buildbot/google/docker/buildbot-windows10-vs2019/run.ps1

+# Stop the script on the first error
+$ErrorActionPreference = "Stop"
+
+# Read password from file
+$WORKER_PASSWORD=Get-Content "C:\volumes\secrets\token"
+
+# Write host information
+# configure powershell output to use UTF-8, otherwiese buildbot can't read the file
+$HOST_FILE="$env:WORKER_NAME\info\host"
+$PSDefaultParameterValues['Out-File:Encoding'] = 'utf8'; `
+Write-Output "Windows version: $(cmd /c ver)" > $HOST_FILE
+Write-Output "VCTools version: $env:VCToolsVersion" >> $HOST_FILE
+Write-Output "Cores          : $((Get-CimInstance Win32_ComputerSystem).NumberOfLogicalProcessors)" >> $HOST_FILE
+Write-Output "RAM            : $([int][Math]::Round((Get-CimInstance Win32_ComputerSystem).TotalPhysicalMemory / 1073741824)) GB" >> $HOST_FILE
+ 
+# create the worker
+Write-Output "creating worker..."
+buildslave create-slave --keepalive=200 "$env:WORKER_NAME" `
+  "lab.llvm.org:9994" "$env:WORKER_NAME" "$WORKER_PASSWORD"
+
+# Note: Powershell does NOT exit on non-zero return codes, so we need to check manually.
+if ($LASTEXITCODE -ne 0) { throw "Exit code is $LASTEXITCODE" }
+
+# start the daemon, as this does not print and logs to std out, run it in the background
+Write-Output "starting worker..."
+cmd /c Start /B buildslave start "$env:WORKER_NAME"
+
+# Wait a bit until the logfile exists
+Start-Sleep -s 5
+
+# To keep the container running and produce log outputs: dump the worker
+# log to stdout, this is the windows equivalent of `tail -f`
+Get-Content -Path "$env:WORKER_NAME\twistd.log" -Wait

buildbot/google/docker/pod_login.sh

 #! /bin/bash
 #===-- pod_login.sh ------------------------------------------------------===//
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 #
 #===----------------------------------------------------------------------===//
 # This script will give you a terminal in on of the pods. This is useful
 # to debug problems in a running container. This is much faster than going
 # through the Google Cloud UI.
 #  
 # Arguments:
 #     pod name fragement : 
 #             part of the name of the pod to log in, eg. name of the 
 #             deployment, if we have only one of them
+#     command (optional):
+#             Command to be run in the container. Default: /bin/bash
 #===----------------------------------------------------------------------===//
 
 set -eu
 
 WORKLOAD_NAME=$1
+CMD="/bin/bash"
+if [ "$#" -eq 2 ];
+then
+    CMD="$2"
+fi
+
 # get name of the pod
 POD=$(kubectl get pod -o name | grep "$1")
 
 # FIXME: exit if more than one pod is returned
 
 # login to the pod
-kubectl exec --stdin --tty "${POD}" -- /bin/bash
+kubectl exec --stdin --tty "${POD}" -- "${CMD}"

buildbot/google/docker/windows-base-vscode2019/Dockerfile

+# escape=`
+# Windows is picky about host and container OS versions. Recommondation:
+# Use "LTSC" releases for both.
+# https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/version-compatibility
+# https://cloud.google.com/kubernetes-engine/docs/how-to/creating-a-cluster-windows#choose_your_windows_server_node_image
+FROM mcr.microsoft.com/dotnet/framework/sdk:4.8-windowsservercore-ltsc2019
+
+# Restore the default Windows shell for correct batch processing.
+SHELL ["cmd", "/S", "/C"]
+
+# Download the Build Tools bootstrapper.
+ADD https://aka.ms/vs/16/release/vs_buildtools.exe C:\TEMP\vs_buildtools.exe
+
+# Download channel for fixed install.
+ARG CHANNEL_URL=https://aka.ms/vs/16/release/channel
+ADD ${CHANNEL_URL} C:\TEMP\VisualStudio.chman
+
+# Install Build Tools with C++ workload.
+#   - Documentation for docker installation 
+#     https://docs.microsoft.com/en-us/visualstudio/install/build-tools-container?view=vs-2019
+#   - Documentation on workloads 
+#     https://docs.microsoft.com/en-us/visualstudio/install/workload-component-id-vs-build-tools?view=vs-2019#c-build-tools
+#   - Documentation on flags
+#     https://docs.microsoft.com/en-us/visualstudio/install/use-command-line-parameters-to-install-visual-studio?view=vs-2019
+RUN C:\TEMP\vs_buildtools.exe --quiet --wait --norestart --nocache `
+    --channelUri C:\TEMP\VisualStudio.chman `
+    --installChannelUri C:\TEMP\VisualStudio.chman `
+    --installPath C:\BuildTools `
+    --add Microsoft.VisualStudio.Workload.VCTools --includeRecommended `
+    --add Microsoft.VisualStudio.Component.VC.ATL `
+    || IF "%ERRORLEVEL%"=="3010" EXIT 0
+
+# install chocolately as package manager
+RUN powershell -NoProfile -InputFormat None -Command `
+    iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1')) ; `
+    choco feature disable --name showDownloadProgress
+
+# install tools as described in https://llvm.org/docs/GettingStartedVS.html
+# and a few more that were not documented...
+RUN choco install -y ninja 
+RUN choco install -y git
+RUN choco install -y cmake
+RUN choco install -y gnuwin
+# lit runs best with Python3, Buildbot worker needs Python2
+RUN choco install -y python2 python3
+RUN choco install -y sccache
+# Install psutils for python3 for lit
+RUN pip3 install psutil
+# Install buildbot using python2
+RUN pip2 install --trusted-host pythonhosted.org buildbot-slave==0.8.11 pywin32
+
+# configure Python encoding
+ENV PYTHONIOENCODING=UTF-8
+
+# update the path variable    
+RUN powershell -NoProfile -InputFormat None  -Command `
+    $path = $env:path + ';c:\Program Files (x86)\GnuWin32\bin;C:\Program Files\CMake\bin'; `
+    Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager\Environment\' -Name Path -Value $path
+
+# use this folder to store the worksapce'
+WORKDIR C:\buildbot
+
+# Root folder for all docker volumes. Make it explicite that all subfolders
+# are volumes. Volumes are a common pitfall for config errors.
+RUN md c:\volumes
+
+# storage for cached builds
+VOLUME C:\volumes\sccache
+
+# storage for secrets
+VOLUME C:\volumes\secrets
+
+# support long file names during git checkout
+RUN git config --system core.longpaths true & `
+    git config --global core.autocrlf false
+
+# Define the entry point for the docker container.
+# This entry point starts the developer command prompt and launches the PowerShell shell.
+#
+# For running manually:
+# C:\BuildTools\Common7\Tools\VsDevCmd.bat -arch=amd64 -host_arch=amd64
+ENTRYPOINT ["C:\\BuildTools\\Common7\\Tools\\VsDevCmd.bat", "-arch=amd64", "-host_arch=amd64", "&&", "powershell.exe", "-NoLogo", "-ExecutionPolicy", "Bypass"]

buildbot/google/docker/windows-base-vscode2019/VERSION

+5

buildbot/google/docker/windows.md

+# Overview
+
+Running a Windows buildbot differs from Linux a bit. So this document will give 
+hints on the differences.
+
+# Create VM
+To create a new Dockerfile and you do not have a local Windows machine, create 
+a VM in the the cloud.
+
+* Use the latest Windows image with "Desktop experience" in the description.
+* Use something with at least 16 cores, to get reasonable build speed while 
+  testing your images.
+* Get 200GB of persistent SSD, this is much faster than the normal presistent 
+  storage.
+* In the "Cloud API access scopes" select "Read Write" for "Storage" to be able
+  to upload docker images form the VM.
+
+# setup your workstation
+For frequent access to your Windows VM it's conveneint to install Remmina, a 
+remote desktop client for Linux, to connect the the Windows VM. 
+
+# Configure the VM
+1. install [Chocolately](https://chocolatey.org/docs/installation) as package 
+   manager.
+1. Install your development tools, a good starting point is:
+    ```
+    choco install -y git vscode googlechrome arcanist vim
+    ``
+1. Setup your github account and push access. Start "Git bash" and run 
+   `ssh-keygen`, upload your public key to Github.
+1. Setup your Phabricator account for `arc diff`.
+1. `git clone ssh://[e-mail]/llvm/llvm-zorg.git`
+1. To install docker, run in an powershell with admin rights:
+    ```powershell
+    Install-PackageProvider -Name NuGet -Force
+    Install-Module -Name DockerMsftProvider -Repository PSGallery -Force
+    Install-Package -Name docker -ProviderName DockerMsftProvider -Force
+    sc.exe config docker start=delayed-auto
+    ```
+    Then reboot your VM to start the Docker service.
+1. Maybe disable Windows Defender anti vrius "realtime protection" to get better
+   IO performance.
+1. Install [Google Cloud SDK](https://cloud.google.com/sdk/install). Installing
+   via chocolately is sometimes broken. Then run `gcloud auth login` and 
+   `gcloud auth configure-docker` to be able to push images to the registry.
+1. Create a file `token` somewhere and store your worker password there. You 
+   will need this to test your worker locally.
+
+# General Hints
+
+## Get ideas from LLVM premerge-checks
+[Premerge-testing](https://github.com/google/llvm-premerge-checks/) is also 
+running LLVM builds on Windows. You might be able to get some ideas from there.
+
+## Bash on Windows
+Git on Windows includes a bash. Right-click the Start menu und launch a
+"Windows Powershell (Admin)" (with admin rights, as docker requires that) and 
+run `bash` from there. Then you should be able to use the helper scripts 
+(e.g. `build_run.sh`) on Windows as well.
+
+## Resource Usage
+If Windows seems stuck, use the `Resource Monitor` to check what the processes
+are doing. If you need even more insight, e.g. debugging builds or test, use the
+`Process Explorer` (`choco install procexp`) to figure out what's going on.
+
+## Windows base image with Visual Studio 2019
+Reuse the `windows-base-vscode2019` image, it contains everything you need to
+build and test with Visual Studio 2019.

buildbot/google/terraform/buildbot-mlir-nvidia/README.md

 This folder contains the Terraform configuration to spawn the build bots.
 
 Before deploying anything new, use `terraform plan` to check that you're only 
 modifying the parts that you intended to.
 
 
 # Installation
 
 To set up your local machine to deploy changes to the cluster follow these 
 steps:
 
 1. Install these tools:
     1. [Terraform](https://learn.hashicorp.com/terraform/getting-started/install.html)
     1. [Google Cloud SDK](https://cloud.google.com/sdk/install)
     1. [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
 1. Run `llvm-zorg/buildbot/google/gcloud_config.sh` to configure the Google
    Cloud SDK.
 1. To configure the GCP credetianls for terraform run: 
    ```bash
     export GOOGLE_CREDENTIALS=~/.config/gcloud/legacy_credentials/<your email>/adc.json
     ```
 
 # Deploying to new Google Cloud project
 
 When deploying this cluster to a completely new Google Cloud project, these 
 manual steps are required:
 
 * You need to create the GCP project manually before Terraform works.
 * You also need to go to the Kubernetes page once, to enable Kubernetes and 
   Container Registry for that project.
 * GPUs need to be enabled on Kubernetes by following these
 [instructions](https://cloud.google.com/kubernetes-engine/docs/how-to/gpus#installing_drivers).
 
 
 # Secrets
 
 To keep secrets a secret, they MUST not be stored in version control. The right
 place on kubernetes is a "secret". To create a kubernetes secret for the agent
 token: 
 ```bash
 kubectl create secret generic buildbot-token-mlir-nvidia --from-file=token=<file name>
 ```
 The file in `<file name>` then must contain the password of the buildbot worker
 in plain text. In the `Deployment` of a container, the secret is defined as a 
 special type of volume and mounted in the specified path. During runtime the 
 secret can then be read from that file.
 
 An example:
 The secret `buildbot-token-mlir-nvidia` is defined (as above) in Kubernetes. 
 In the [deployment](buildbot/google/terraform/main.tf) `mlir-nvidia` it is 
 used as a volume of type `secret` and then mounted at `/secrets`. During the 
 runtime of the docker container, the script 
 [run.sh](../docker/buildbot-mlir-nvidia/run.sh) reads the secret from the file
 `/secrets/token` and uses it to create the worker configuration.
 
 
 # Using GPUs on Google Cloud
 
 Terraform does not support deployments on GCP using a GPU at the moment.
 So we need to deploy such cases using plain Kubernetes configuration files.
 See this [issue](https://github.com/terraform-providers/terraform-provider-kubernetes/issues/149) 
 for more details.
 The buildbot mlir-nvidia is configured in `deployment-mlir-nvidia.yaml` in this
 folder. 
 
 For all non-GPU cases add a `"kubernetes_deployment"` to `main.tf`. 
 The contents is identical to the the Kubernetes file, just the markup is 
 different.
 
 Kubernetes files are also declarative, so you can re-deploy them when you made
 a change. They can be deployed with:
 ```bash
 kubectl apply -f myfile.yaml
 ```

buildbot/google/terraform/buildbot-mlir-nvidia/deployment-mlir-nvidia.yaml

 ---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: mlir-nvidia
 spec:
   # number of instances we want to run
   replicas: 1
   selector:
     matchLabels:
       app: buildbot-mlir-nvidia
   # define strategy for updating the images
   strategy:
     rollingUpdate:
       # do not deploy more replicas, as the buildbot server 
       # can't handle multiple workers with the same credentials
       maxSurge: 0
       # Allow to have 0 replicas during updates. 
       maxUnavailable: 1
     type: RollingUpdate
   template:
     metadata:
       labels:
         app: buildbot-mlir-nvidia
     spec:
       containers:
       # the image and version we want to run   
       - image: gcr.io/sanitizer-bots/buildbot-mlir-nvidia:9
         name: mlir-nvidia
         # reserve "<number of cores>-1" for this image, kubernetes also
         # needs <1 core for management tools
         resources:
           limits:
             cpu: "15"
             memory: 10Gi
             # also request to use the GPU
             # Note: this does not work in terraform at the moment
             nvidia.com/gpu: "1"
           requests:
             cpu: "15"
             memory: 10Gi
             nvidia.com/gpu: "1"
         volumeMounts:
         # mount the secrets into a folder  
         - mountPath: /secrets
           mountPropagation: None
           name: buildbot-token
       # specify the nood pool on which to deploy
       nodeSelector:
         pool: nvidia-16core-pool
       restartPolicy: Always
       # FIXME: do we need this if we requested a GPU?
       #tolerations:
       #- effect: NoSchedule
       #  key: nvidia.com/gpu
       #  operator: Equal
       #  value: present
       volumes:
       # declare the secret as a volume so we can mount it
       - name: buildbot-token
         secret:
           optional: false
           secretName: password-mlir-nvidia

buildbot/google/terraform/buildbot-mlir-nvidia/main.tf

 
 # configure Google Cloud project
 provider "google" {
   project  = var.gcp_config.project
   region   = var.gcp_config.region
 }
 
 # Create the cluster runningn all Kubernetes services
 resource "google_container_cluster" "primary" {
   name     = "buildbot-cluster"
   # maybe have a regional cluster for Kubernetes, as we depend on this...
   location = var.gcp_config.zone_a
 
   # one node is enough (at the moment)
   initial_node_count = 1
 
   node_config {
     # FIXME(kuhnel): turn this into a private cluster, without external IP
     # We need at least 2 vCPU to run all kubernetes services
     machine_type = "n1-standard-2"
     # use preemptible, as this saves costs
     preemptible = true
   }
 }
 
 # Create machines for mlir-nvidia
 # Note: The buildbot mlir-nividia is deployed using a kubernetes file. See
 # the README.md for details on GPUs.
 
 resource "google_container_node_pool" "nvidia_16core_pool_nodes" {
   name       = "nvidia-16core-pool"
   # specify a zone here (e.g. "-a") to avoid a redundant deployment
   location   = var.gcp_config.zone_a
   cluster    = google_container_cluster.primary.name
   
   # use autoscaling to only create a machine when there is a deployment
   autoscaling {
     min_node_count = 0
     max_node_count = 1
   }
   
   node_config {
     # use preemptible, as this saves costs
     preemptible  = true
     machine_type = "n1-highcpu-16"
     disk_size_gb = 100
     # FIXME: test if SSDs are actually faster than HDDs for our use case
     disk_type = "pd-ssd"
     guest_accelerator {
       type = "nvidia-tesla-t4"
       count= 1
     }
 
     # set the premissions required for the deployment later
     oauth_scopes = [
       "https://www.googleapis.com/auth/logging.write",
       "https://www.googleapis.com/auth/monitoring",
       "https://www.googleapis.com/auth/devstorage.read_only",
     ]
 
     # add a label to all machines of this type, so we can select them 
     # during deployment
     labels = {
       pool = "nvidia-16core-pool"
     }
   }
 }

buildbot/google/terraform/buildbot-mlir-nvidia/variables.tf

 # configuration parameter for Google Cloud
 variable "gcp_config" {
   type = object({
     project     = string
     region      = string
     zone_a      = string
     gcr_prefix  = string
    })
   default = {
     project     = "sanitizer-bots"
     region      = "us-central1"
     zone_a      = "us-central1-a"
     gcr_prefix  = "gcr.io/sanitizer-bots"
   }
 }

buildbot/osuosl/master/config/builders.py

 from buildbot.process.properties import WithProperties
 
 from zorg.buildbot.builders import ClangBuilder
-from zorg.buildbot.builders import LLVMBuilder
 from zorg.buildbot.builders import PollyBuilder
 from zorg.buildbot.builders import LLDBBuilder
-from zorg.buildbot.builders import LLDBuilder
-from zorg.buildbot.builders import ClangAndLLDBuilder
 from zorg.buildbot.builders import SanitizerBuilder
-from zorg.buildbot.builders import SanitizerBuilderII
 from zorg.buildbot.builders import OpenMPBuilder
 from zorg.buildbot.builders import LibcxxAndAbiBuilder
 from zorg.buildbot.builders import SphinxDocsBuilder
 from zorg.buildbot.builders import ABITestsuitBuilder
-from zorg.buildbot.builders import ClangLTOBuilder3Stage
 from zorg.buildbot.builders import ClangLTOBuilder
 from zorg.buildbot.builders import UnifiedTreeBuilder
 from zorg.buildbot.builders import CUDATestsuiteBuilder
 from zorg.buildbot.builders import AOSPBuilder
 from zorg.buildbot.builders import AnnotatedBuilder
 from zorg.buildbot.builders import LLDPerformanceTestsuite
 from zorg.buildbot.builders import FuchsiaBuilder
 from zorg.buildbot.builders import XToolchainBuilder
return [
                         testStage1=True,
                         useTwoStage=True,
                         stage1_config='Release',
                         stage2_config='Release',
                         extra_cmake_args=['-DLLVM_ENABLE_ASSERTIONS=ON',
                                           '-DLLVM_TARGETS_TO_BUILD=X86',
                                           '-DCMAKE_C_COMPILER_LAUNCHER=sccache',
                                           '-DCMAKE_CXX_COMPILER_LAUNCHER=sccache',
+                                          # FIXME: DIA is not working on this machine, so disabling it until we 
+                                          #        have a proper fix
+                                          '-D LLVM_ENABLE_DIA_SDK=OFF',
                                           ])},
 
     ]
 
 # Polly builders.
 def _get_polly_builders():
     return [
         {'name': "polly-arm-linux",

 
 # Experimental and stopped builders
 def _get_on_demand_builders():
     return [
         ]
 
 def _get_experimental_scheduled_builders():
     return [
-        {'name' : "clang-cuda-build",
-         'slavenames' : ["cuda-build-test-01"],
-         'builddir' : "clang-cuda-build",
-         'factory' : CUDATestsuiteBuilder.getCUDATestsuiteBuildFactory(
-                     useTwoStage=False,
-                     test=True,
-                     stage1_config='Release',
-                     extra_cmake_args=[
-                         '-DLLVM_ENABLE_ASSERTIONS=ON',
-                         "-DCMAKE_C_COMPILER:FILEPATH=/usr/bin/clang",
-                         "-DCMAKE_CXX_COMPILER:FILEPATH=/usr/bin/clang++",
-                         "-DLIBUNWIND_LIBCXX_PATH=../llvm/libcxx",
-                     ],
-                     externals="/home/botanist/bots/externals",
-                     gpu_arch_list=["sm_35", "sm_61"],
-                     gpu_devices=["GPU-af66efa4",  # K40c(sm_35),
-                                  "GPU-44fe2444"   # GTX1070(sm_61)
-                     ],
-                     cuda_jobs = 4, # Speeds up test execution time by ~2x.
-                     extra_ts_cmake_args=[],
-                     enable_thrust_tests=False,
-         ),
+        {'name': "clang-cuda-k80",
+         'slavenames' :["cuda-k80-0"],
+         'builddir': "clang-cuda-k80",
+         'factory': AnnotatedBuilder.getAnnotatedBuildFactory(script="/buildbot/cuda-build",
+                                                              checkout_llvm_sources=False),
+         'category' : 'clang'},
+        {'name': "clang-cuda-p4",
+         'slavenames' :["cuda-p4-0"],
+         'builddir': "clang-cuda-p4",
+         'factory': AnnotatedBuilder.getAnnotatedBuildFactory(script="/buildbot/cuda-build",
+                                                              checkout_llvm_sources=False),
+         'category' : 'clang'},
+        {'name': "clang-cuda-t4",
+         'slavenames' :["cuda-t4-0"],
+         'builddir': "clang-cuda-t4",
+         'factory': AnnotatedBuilder.getAnnotatedBuildFactory(script="/buildbot/cuda-build",
+                                                              checkout_llvm_sources=False),
          'category' : 'clang'},
-
         {'name': "clang-ve-ninja",
          'slavenames':["nec-arrproto41"],
          'builddir':"clang-ve-ninja",
          'factory': UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
                         clean=True,
                         depends_on_projects=['llvm','clang','openmp'],
                         extra_configure_args=[
                             "-DLLVM_TARGETS_TO_BUILD=X86",

buildbot/osuosl/master/config/slaves.py

return [
         # Ubuntu 16.04.3 LTS x86_64-scei-ps4, 2 x Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
         create_slave("ps4-buildslave4"),
 
         # NetBSD amd64
         create_slave("netbsd-amd64", properties={'jobs': 8}, max_builds=1),
 
         # Ubuntu 16.04 x86_64, 2 x Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz, 64GB of RAM
         create_slave("cuda-build-test-01", properties={'jobs': 72}, max_builds=1),
+        # WIP migration of the CUDA buildbot to GCE.
+        create_slave("cuda-k80-0", max_builds=1),
+        create_slave("cuda-p4-0", max_builds=1),
+        create_slave("cuda-t4-0", max_builds=1),
 
         # Ubuntu 14.04 x86_64, AMD Athlon(tm) 5150 APU with Radeon(tm) R3, 8GiB RAM
         create_slave("am1i-slv1", properties={'jobs': 8}),
 
         # Ubuntu 14.04 x86_64, AMD Athlon(tm) 5150 APU with Radeon(tm) R3, 8GiB RAM
         create_slave("am1i-slv3", properties={'jobs': 8}),
 
         # Ubuntu 14.04 x86_64, AMD Athlon(tm) 5150 APU with Radeon(tm) R3, 8GiB RAM

buildbot/osuosl/master/config/status.py

return [
             addLogs=False,
             num_lines = 15),
         InformativeMailNotifier(
             fromaddr = "[e-mail]",
             sendToInterestedUsers= False,
             extraRecipients = ["[e-mail]"],
             subject="Build %(builder)s Failure",
             mode = "failing",
-            builders = ["clang-cuda-build"],
+            builders = ["clang-cuda-k80", "clang-cuda-p4", "clang-cuda-t4"],
             addLogs=False,
             num_lines = 15),
         InformativeMailNotifier(
             fromaddr = "[e-mail]",
             sendToInterestedUsers= False,
             extraRecipients = ["[e-mail]", "[e-mail]"],
             subject="Build %(builder)s Failure",
             mode = "failing",

test/buildbot/builders/Import.py

 # RUN: python %s
 
 import zorg
-from zorg.buildbot.builders import ClangBuilder, LLVMBuilder, LLVMGCCBuilder
-from zorg.buildbot.builders import NightlytestBuilder
+from zorg.buildbot.builders import ClangBuilder
 from zorg.buildbot.builders import SanitizerBuilder
 
 # Just check that we can instantiate the build factors, what else can we do?
 
 print ClangBuilder.getClangBuildFactory()
 
-print LLVMBuilder.getLLVMBuildFactory()
-
-print LLVMGCCBuilder.getLLVMGCCBuildFactory()
-
-print NightlytestBuilder.getFastNightlyTestBuildFactory('x86_64-apple-darwin10')
-
 print SanitizerBuilder.getSanitizerBuildFactory()

zorg/buildbot/builders/AnnotatedBuilder.py

 from buildbot.process.properties import WithProperties
 from buildbot.steps.shell import SetProperty
 from zorg.buildbot.commands.AnnotatedCommand import AnnotatedCommand
 from zorg.buildbot.process.factory import LLVMBuildFactory
 
 
 def getAnnotatedBuildFactory(
     script,
     clean=False,
     depends_on_projects=None,
     env=None,
     extra_args=None,
     timeout=1200,
-    is_legacy_mode=False):
+    is_legacy_mode=False,
+    checkout_llvm_sources=True):
     """
     Returns a new build factory that uses AnnotatedCommand, which
     allows the build to be run by version-controlled scripts that do
     not require a buildmaster restart to update.
 
     script: script under "builders/annotated" to be run by python
     clean: set to true for a clean build of llvm
     depends_on_projects: which subprojects to enable
def getAnnotatedBuildFactory(
 
     # Check out zorg so we can run the annotator scripts.
     f.addGetSourcecodeForProject(
         name='update-annotated-scripts',
         project='zorg',
         src_dir='llvm-zorg',
         alwaysUseLatest=True)
 
+    if checkout_llvm_sources:
       f.addGetSourcecodeSteps()
 
-
     extra_args_with_props = [WithProperties(arg) for arg in extra_args]
     # Explicitly use '/' as separator, because it works on *nix and Windows.
+    if script.startswith('/'):
+      command = [script]
+    else:
       script_path = "../llvm-zorg/zorg/buildbot/builders/annotated/%s" % (script)
+      command = ["python", script_path, WithProperties("--jobs=%(jobs:-)s")]
+    command += extra_args_with_props
+
     f.addStep(AnnotatedCommand(name="annotate",
                                description="annotate",
                                timeout=timeout,
                                haltOnFailure=True,
-                               command=["python",
-                                        script_path,
-                                        WithProperties("--jobs=%(jobs:-)s")]
-                                        + extra_args_with_props,
+                               command=command,
                                env=merged_env))
     return f

zorg/buildbot/builders/ClangAndLLDBuilder.py

-import os
-
-import buildbot
-import buildbot.process.factory
-from buildbot.steps.source import SVN
-from buildbot.steps.shell import ShellCommand
-from buildbot.steps.shell import WarningCountingShellCommand
-from buildbot.steps.shell import SetProperty
-from buildbot.process.properties import WithProperties, Property
-from zorg.buildbot.commands.LitTestCommand import LitTestCommand
-from zorg.buildbot.commands.NinjaCommand import NinjaCommand
-
-import zorg.buildbot.builders.Util as builders_util
-
-def getClangAndLLDBuildFactory(
-           clean=True,
-           env=None,
-           withLLD=True,
-           extraCmakeOptions=None,
-           extraCompilerOptions=None,
-           buildWithSanitizerOptions=None,
-           triple=None,
-           isMSVC=False,
-           # Choose VS tools to build with. For example,
-           # "autodetect" to find the latest installed Visual Studio, or
-           # %VS140COMNTOOLS% to selects the 2015 toolchain.
-           vs=None,
-           target_arch=None,
-           prefixCommand=["nice", "-n", "10"], # For backward compatibility.
-           extraLitArgs=None
-    ):
-
-    llvm_srcdir = "llvm.src"
-    llvm_objdir = "llvm.obj"
-
-    # Prepare environmental variables. Set here all env we want everywhere.
-    merged_env = {
-        'TERM' : 'dumb' # Make sure Clang doesn't use color escape sequences.
-                 }
-    if env is not None:
-        assert not (isMSVC and vs), "Can't have custom builder env vars with VS"
-        # Overwrite pre-set items with the given ones, so user can set anything.
-        merged_env.update(env)
-
-    f = buildbot.process.factory.BuildFactory()
-
-    # Determine the build directory.
-    f.addStep(buildbot.steps.shell.SetProperty(name="get_builddir",
-                                               command=["pwd"],
-                                               property="builddir",
-                                               description="set build dir",
-                                               workdir=".",
-                                               env=merged_env))
-    # Get LLVM, Clang and LLD.
-    f.addStep(SVN(name='svn-llvm',
-                  mode='update',
-                  baseURL='http://llvm.org/svn/llvm-project/llvm/',
-                  defaultBranch='trunk',
-                  workdir=llvm_srcdir))
-
-    # Sanitizer runtime in compiler-rt, and cannot be built with sanitizer compiler.
-    if buildWithSanitizerOptions is None:
-        f.addStep(SVN(name='svn-compiler-rt',
-                      mode='update',
-                      baseURL='http://llvm.org/svn/llvm-project/compiler-rt/',
-                      defaultBranch='trunk',
-                      workdir='%s/projects/compiler-rt' % llvm_srcdir))
-
-    f.addStep(SVN(name='svn-clang',
-                  mode='update',
-                  baseURL='http://llvm.org/svn/llvm-project/cfe/',
-                  defaultBranch='trunk',
-                  workdir='%s/tools/clang' % llvm_srcdir))
-    f.addStep(SVN(name='svn-clang-tools-extra',
-                  mode='update',
-                  baseURL='http://llvm.org/svn/llvm-project/clang-tools-extra/',
-                  defaultBranch='trunk',
-                  workdir='%s/tools/clang/tools/extra' % llvm_srcdir))
-    if withLLD:
-        f.addStep(SVN(name='svn-lld',
-                      mode='update',
-                      baseURL='http://llvm.org/svn/llvm-project/lld/',
-                      defaultBranch='trunk',
-                      workdir='%s/tools/lld' % llvm_srcdir))
-
-    # Clean directory, if requested.
-    if clean:
-        shellCommand = ["rm", "-rf", llvm_objdir]
-        if isMSVC:
-            shellCommand = ["rmdir", "/S", "/Q", llvm_objdir]
-        f.addStep(ShellCommand(name="rm-llvm_objdir",
-                               command=shellCommand,
-                               haltOnFailure=False,
-                               flunkOnFailure=False,
-                               description=["rm build dir", "llvm"],
-                               workdir=".",
-                               env=merged_env))
-
-    # Create configuration files with cmake.
-    options = ["-Wdocumentation", "-Wno-documentation-deprecated-sync"]
-    if isMSVC:
-        options = []
-    if extraCompilerOptions:
-        options += extraCompilerOptions
-
-    if buildWithSanitizerOptions:
-        options += buildWithSanitizerOptions
-
-    lit_args = ["-v"]
-    if extraLitArgs:
-        lit_args += extraLitArgs
-
-    lit_args = ["-DLLVM_LIT_ARGS=\"%s\"" % " ".join(lit_args)]
-
-    cmakeCommand = [
-        "cmake",
-        "-DCMAKE_BUILD_TYPE=Release",
-        "-DLLVM_ENABLE_ASSERTIONS=ON"
-    ]
-    if buildWithSanitizerOptions:
-        cmakeCommand += [
-            "-DCMAKE_C_COMPILER=clang",
-            "-DCMAKE_CXX_COMPILER=clang++"
-        ]
-    if triple:
-        cmakeCommand += ["-DLLVM_DEFAULT_TARGET_TRIPLE=%s" % triple]
-
-    if extraCmakeOptions:
-        assert not any(a.startswith('-DLLVM_LIT_ARGS=') for a in extraCmakeOptions), \
-            "Please use extraLitArgs for LIT arguments instead of defining them in extraCmakeOptions."
-        cmakeCommand += extraCmakeOptions
-
-    if not isMSVC:
-        cmakeCommand += [
-            "-DCMAKE_C_FLAGS=\"%s\"" % (" ".join(options)),
-            "-DCMAKE_CXX_FLAGS=\"-std=c++11 %s\"" % (" ".join(options)),
-        ]
-    cmakeCommand += lit_args
-    cmakeCommand += [
-       "-GNinja",
-        "../%s" % llvm_srcdir
-    ]
-
-    if isMSVC and vs:
-        # Set up VS environment, if requested.
-        f.addStep(SetProperty(
-            command=builders_util.getVisualStudioEnvironment(vs, target_arch),
-            extract_fn=builders_util.extractSlaveEnvironment))
-        merged_env = Property('slave_env')
-
-    # Note: ShellCommand does not pass the params with special symbols right.
-    # The " ".join is a workaround for this bug.
-    f.addStep(ShellCommand(name="cmake-configure",
-                               description=["cmake configure"],
-                               haltOnFailure=True,
-                               command=WithProperties(" ".join(cmakeCommand)),
-                               workdir=llvm_objdir,
-                               env=merged_env))
-
-    # Build everything.
-    f.addStep(NinjaCommand(name="build",
-                           prefixCommand=prefixCommand,
-                           haltOnFailure=True,
-                           description=["build"],
-                           workdir=llvm_objdir,
-                           env=merged_env))
-
-    # Test everything.
-    f.addStep(NinjaCommand(name="test",
-                           prefixCommand=prefixCommand,
-                           targets=["check-all"],
-                           haltOnFailure=True,
-                           description=["test"],
-                           workdir=llvm_objdir,
-                           env=merged_env))
-
-    return f

zorg/buildbot/builders/ClangBuilder.py

-import buildbot
-import buildbot.process.factory
 import copy
-import os
 from datetime import datetime
 
 from buildbot.process.properties import WithProperties, Property
-from buildbot.steps.shell import Configure, ShellCommand, SetProperty
+from buildbot.steps.shell import ShellCommand, SetProperty
 from buildbot.steps.shell import WarningCountingShellCommand
-from buildbot.steps.source import SVN
-from buildbot.steps.transfer import FileDownload
 
-import zorg.buildbot.util.artifacts as artifacts
 import zorg.buildbot.builders.Util as builders_util
-import zorg.buildbot.util.phasedbuilderutils as phasedbuilderutils
-import zorg.buildbot.commands as commands
-import zorg.buildbot.commands.BatchFileDownload as batch_file_download
-import zorg.buildbot.commands.LitTestCommand as lit_test_command
+
+from zorg.buildbot.commands.LitTestCommand import LitTestCommand
 from zorg.buildbot.conditions.FileConditions import FileDoesNotExist
 from zorg.buildbot.commands.CmakeCommand import CmakeCommand
 from zorg.buildbot.process.factory import LLVMBuildFactory
 
-# FIXME: This method is deprecated and will be removed. Please use getClangCMakeBuildFactory instead.
-def getClangBuildFactory(
-            triple=None,
-            clean=True,
-            test=True,
-            package_dst=None,
-            run_cxx_tests=False,
-            examples=False,
-            valgrind=False,
-            valgrindLeakCheck=False,
-            useTwoStage=False,
-            completely_clean=False, 
-            make='make',
-            jobs="%(jobs)s",
-            stage1_config='Debug+Asserts',
-            stage2_config='Release+Asserts',
-            env={}, # Environmental variables for all steps.
-            extra_configure_args=[],
-            stage2_extra_configure_args=[],
-            use_pty_in_tests=False,
-            trunk_revision=None,
-            force_checkout=False,
-            extra_clean_step=None,
-            checkout_compiler_rt=False,
-            checkout_lld=False,
-            run_gdb=False,
-            run_modern_gdb=False,
-            run_gcc=False):
-    # Prepare environmental variables. Set here all env we want everywhere.
-    merged_env = {
-        'TERM' : 'dumb' # Make sure Clang doesn't use color escape sequences.
-                 }
-    if env is not None:
-        # Overwrite pre-set items with the given ones, so user can set anything.
-        merged_env.update(env)
-
-    llvm_srcdir = "llvm.src"
-    llvm_1_objdir = "llvm.obj"
-    llvm_1_installdir = "llvm.install.1"
-    llvm_2_objdir = "llvm.obj.2"
-    llvm_2_installdir = "llvm.install"
-
-    f = buildbot.process.factory.BuildFactory()
-
-    # Determine the build directory.
-    f.addStep(buildbot.steps.shell.SetProperty(name="get_builddir",
-                                               command=["pwd"],
-                                               property="builddir",
-                                               description="set build dir",
-                                               workdir=".",
-                                               env=merged_env))
-
-    # Blow away completely, if requested.
-    if completely_clean:
-        f.addStep(ShellCommand(name="rm-llvm.src",
-                               command=["rm", "-rf", llvm_srcdir],
-                               haltOnFailure=True,
-                               description=["rm src dir", "llvm"],
-                               workdir=".",
-                               env=merged_env))
-
-    # Checkout sources.
-    if trunk_revision:
-        # The SVN build step provides no mechanism to check out a specific revision
-        # based on a property, so just run the commands directly here.
-        svn_co = ['svn', 'checkout']
-        if force_checkout:
-            svn_co += ['--force']
-        svn_co += ['--revision', WithProperties(trunk_revision)]
-
-        svn_co_llvm = svn_co + \
-          [WithProperties('http://llvm.org/svn/llvm-project/llvm/trunk@%s' %
-                          trunk_revision),
-           llvm_srcdir]
-        svn_co_clang = svn_co + \
-          [WithProperties('http://llvm.org/svn/llvm-project/cfe/trunk@%s' %
-                          trunk_revision),
-           '%s/tools/clang' % llvm_srcdir]
-        svn_co_clang_tools_extra = svn_co + \
-          [WithProperties('http://llvm.org/svn/llvm-project/clang-tools-extra/trunk@%s' %
-                          trunk_revision),
-           '%s/tools/clang/tools/extra' % llvm_srcdir]
-
-        f.addStep(ShellCommand(name='svn-llvm',
-                               command=svn_co_llvm,
-                               haltOnFailure=True,
-                               workdir='.'))
-        f.addStep(ShellCommand(name='svn-clang',
-                               command=svn_co_clang,
-                               haltOnFailure=True,
-                               workdir='.'))
-        f.addStep(ShellCommand(name='svn-clang-tools-extra',
-                               command=svn_co_clang_tools_extra,
-                               haltOnFailure=True,
-                               workdir='.'))
-    else:
-        f.addStep(SVN(name='svn-llvm',
-                      mode='update',
-                      baseURL='http://llvm.org/svn/llvm-project/llvm/',
-                      defaultBranch='trunk',
-                      workdir=llvm_srcdir))
-        f.addStep(SVN(name='svn-clang',
-                      mode='update',
-                      baseURL='http://llvm.org/svn/llvm-project/cfe/',
-                      defaultBranch='trunk',
-                      workdir='%s/tools/clang' % llvm_srcdir))
-        f.addStep(SVN(name='svn-clang-tools-extra',
-                      mode='update',
-                      baseURL='http://llvm.org/svn/llvm-project/clang-tools-extra/',
-                      defaultBranch='trunk',
-                      workdir='%s/tools/clang/tools/extra' % llvm_srcdir))
-        if checkout_compiler_rt:
-            f.addStep(SVN(name='svn-compiler-rt',
-                          mode='update',
-                          baseURL='http://llvm.org/svn/llvm-project/compiler-rt/',
-                          defaultBranch='trunk',
-                          workdir='%s/projects/compiler-rt' % llvm_srcdir))
-
-    # Clean up llvm (stage 1); unless in-dir.
-    if clean and llvm_srcdir != llvm_1_objdir:
-        f.addStep(ShellCommand(name="rm-llvm.obj.stage1",
-                               command=["rm", "-rf", llvm_1_objdir],
-                               haltOnFailure=True,
-                               description=["rm build dir", "llvm"],
-                               workdir=".",
-                               env=merged_env))
-
-    if not clean:
-        expected_makefile = 'Makefile'
-        f.addStep(SetProperty(name="Makefile_isready",
-                              workdir=llvm_1_objdir,
-                              command=["sh", "-c",
-                                       "test -e %s && echo OK || echo Missing" % expected_makefile],
-                              flunkOnFailure=False,
-                              property="exists_Makefile"))
-
-    cmake_triple_arg = []
-    if triple:
-        cmake_triple_arg = ['-DLLVM_HOST_TRIPLE=%s' % triple]
-    f.addStep(ShellCommand(name='cmake',
-                           command=['cmake',
-                                    '-DLLVM_BUILD_TESTS=ON',
-                                    '-DCMAKE_BUILD_TYPE=%s' % stage1_config] +
-                                   cmake_triple_arg +
-                                   extra_configure_args +
-                                   ["../" + llvm_srcdir],
-                           description='cmake stage1',
-                           workdir=llvm_1_objdir,
-                           env=merged_env,
-                           doStepIf=lambda step: step.build.getProperty("exists_Makefile") != "OK"))
-
-    # Make clean if using in-dir builds.
-    if clean and llvm_srcdir == llvm_1_objdir:
-        f.addStep(WarningCountingShellCommand(name="clean-llvm",
-                                              command=[make, "clean"],
-                                              haltOnFailure=True,
-                                              description="cleaning llvm",
-                                              descriptionDone="clean llvm",
-                                              workdir=llvm_1_objdir,
-                                              doStepIf=clean,
-                                              env=merged_env))
-
-    if extra_clean_step:
-        f.addStep(extra_clean_step)
-
-    f.addStep(WarningCountingShellCommand(name="compile",
-                                          command=['nice', '-n', '10',
-                                                   make, WithProperties("-j%s" % jobs)],
-                                          haltOnFailure=True,
-                                          flunkOnFailure=not run_gdb,
-                                          description=["compiling", stage1_config],
-                                          descriptionDone=["compile", stage1_config],
-                                          workdir=llvm_1_objdir,
-                                          env=merged_env))
-
-    if examples:
-        f.addStep(WarningCountingShellCommand(name="compile.examples",
-                                              command=['nice', '-n', '10',
-                                                       make, WithProperties("-j%s" % jobs),
-                                                       "BUILD_EXAMPLES=1"],
-                                              haltOnFailure=True,
-                                              description=["compiling", stage1_config, "examples"],
-                                              descriptionDone=["compile", stage1_config, "examples"],
-                                              workdir=llvm_1_objdir,
-                                              env=merged_env))
-
-    clangTestArgs = '-v -j %s' % jobs
-    if valgrind:
-        clangTestArgs += ' --vg'
-        if valgrindLeakCheck:
-            clangTestArgs += ' --vg-leak'
-        clangTestArgs += ' --vg-arg --suppressions=%(builddir)s/llvm/tools/clang/utils/valgrind/x86_64-pc-linux-gnu_gcc-4.3.3.supp --vg-arg --suppressions=%(builddir)s/llvm/utils/valgrind/x86_64-pc-linux-gnu.supp'
-    extraTestDirs = ''
-    if run_cxx_tests:
-        extraTestDirs += '%(builddir)s/llvm/tools/clang/utils/C++Tests'
-    if test:
-        f.addStep(lit_test_command.LitTestCommand(name='check-all',
-                                   command=[make, "check-all", "VERBOSE=1",
-                                            WithProperties("LIT_ARGS=%s" % clangTestArgs),
-                                            WithProperties("EXTRA_TESTDIRS=%s" % extraTestDirs)],
-                                   flunkOnFailure=not run_gdb,
-                                   description=["checking"],
-                                   descriptionDone=["checked"],
-                                   workdir=llvm_1_objdir,
-                                   usePTY=use_pty_in_tests,
-                                   env=merged_env))
-
-    # TODO: Install llvm and clang for stage1.
-
-    if run_gdb or run_gcc or run_modern_gdb:
-        ignores = getClangTestsIgnoresFromPath(os.path.expanduser('~/public/clang-tests'), 'clang-x86_64-darwin10')
-        install_prefix = "%%(builddir)s/%s" % llvm_1_installdir
-        if run_gdb:
-            addClangGDBTests(f, ignores, install_prefix)
-        if run_modern_gdb:
-            addModernClangGDBTests(f, jobs, install_prefix)
-        if run_gcc:
-            addClangGCCTests(f, ignores, install_prefix)
-
-    if not useTwoStage:
-        if package_dst:
-            name = WithProperties(
-                "%(builddir)s/" + llvm_1_objdir +
-                "/clang-r%(got_revision)s-b%(buildnumber)s.tgz")
-            f.addStep(ShellCommand(name='pkg.tar',
-                                   description="tar root",
-                                   command=["tar", "zcvf", name, "./"],
-                                   workdir=llvm_1_installdir,
-                                   warnOnFailure=True,
-                                   flunkOnFailure=False,
-                                   haltOnFailure=False,
-                                   env=merged_env))
-            f.addStep(ShellCommand(name='pkg.upload',
-                                   description="upload root",
-                                   command=["scp", name,
-                                            WithProperties(
-                            package_dst + "/%(buildername)s")],
-                                   workdir=".",
-                                   warnOnFailure=True,
-                                   flunkOnFailure=False,
-                                   haltOnFailure=False,
-                                   env=merged_env))
-
-        return f
-
-    # Clean up llvm (stage 2).
-    #
-    # We always cleanly build the stage 2. If the compiler has been
-    # changed on the stage 1, we cannot trust any of the intermediate file
-    # from the old compiler. And if the stage 1 compiler is the same, we should
-    # not build in the first place.
-    f.addStep(ShellCommand(name="rm-llvm.obj.stage2",
-                           command=["rm", "-rf", llvm_2_objdir],
-                           haltOnFailure=True,
-                           description=["rm build dir", "llvm", "(stage 2)"],
-                           workdir=".",
-                           env=merged_env))
-
-    # Configure llvm (stage 2).
-    f.addStep(ShellCommand(name='cmake',
-                           command=['cmake'] + stage2_extra_configure_args + [
-                                    '-DLLVM_BUILD_TESTS=ON',
-                                    WithProperties('-DCMAKE_C_COMPILER=%%(builddir)s/%s/bin/clang' % llvm_1_objdir), # FIXME use installdir
-                                    WithProperties('-DCMAKE_CXX_COMPILER=%%(builddir)s/%s/bin/clang++' % llvm_1_objdir),
-                                    '-DCMAKE_BUILD_TYPE=%s' % stage2_config,
-                                    "../" + llvm_srcdir],
-                           description='cmake stage2',
-                           workdir=llvm_2_objdir,
-                           env=merged_env))
-
-    # Build llvm (stage 2).
-    f.addStep(WarningCountingShellCommand(name="compile.llvm.stage2",
-                                          command=['nice', '-n', '10',
-                                                   make, WithProperties("-j%s" % jobs)],
-                                          haltOnFailure=True,
-                                          description=["compiling", "(stage 2)",
-                                                       stage2_config],
-                                          descriptionDone=["compile", "(stage 2)",
-                                                           stage2_config],
-                                          workdir=llvm_2_objdir,
-                                          env=merged_env))
-
-    if test:
-        f.addStep(lit_test_command.LitTestCommand(name='check-all',
-                                   command=[make, "check-all", "VERBOSE=1",
-                                            WithProperties("LIT_ARGS=%s" % clangTestArgs),
-                                            WithProperties("EXTRA_TESTDIRS=%s" % extraTestDirs)],
-                                   description=["checking"],
-                                   descriptionDone=["checked"],
-                                   workdir=llvm_2_objdir,
-                                   usePTY=use_pty_in_tests,
-                                   env=merged_env))
-
-    # TODO: Install llvm and clang for stage2.
-
-    if package_dst:
-        name = WithProperties(
-            "%(builddir)s/" + llvm_2_objdir +
-            "/clang-r%(got_revision)s-b%(buildnumber)s.tgz")
-        f.addStep(ShellCommand(name='pkg.tar',
-                               description="tar root",
-                               command=["tar", "zcvf", name, "./"],
-                               workdir=llvm_2_installdir,
-                               warnOnFailure=True,
-                               flunkOnFailure=False,
-                               haltOnFailure=False,
-                               env=merged_env))
-        f.addStep(ShellCommand(name='pkg.upload',
-                               description="upload root",
-                               command=["scp", name,
-                                        WithProperties(
-                        package_dst + "/%(buildername)s")],
-                               workdir=".",
-                               warnOnFailure=True,
-                               flunkOnFailure=False,
-                               haltOnFailure=False,
-                               env=merged_env))
-
-    return f
-
 def addGCSUploadSteps(f, package_name, install_prefix, gcs_directory, env,
                       gcs_url_property=None, use_pixz_compression=False,
                       xz_compression_factor=6):
     """
     Add steps to upload to the Google Cloud Storage bucket.
 
     f - The BuildFactory to modify.
     package_name - The name of this package for the descriptions (e.g.
f.addStep(WarningCountingShellCommand(name='build stage 1',
                                           command=ninja_cmd,
                                           haltOnFailure=True,
                                           description='ninja all',
                                           workdir=stage1_build,
                                           env=env))
 
     if test and testStage1:
         haltOnStage1Check = not useTwoStage and not runTestSuite
-        f.addStep(lit_test_command.LitTestCommand(name='ninja check 1',
-                                   command=ninja_check_cmd,
-                                   haltOnFailure=haltOnStage1Check,
-                                   description=["checking stage 1"],
-                                   descriptionDone=["stage 1 checked"],
-                                   workdir=stage1_build,
-                                   env=env))
+        f.addStep(LitTestCommand(name='ninja check 1',
 
     if useTwoStage or runTestSuite or stage1_upload_directory:
         f.addStep(ShellCommand(name='clean stage 1 install',
                                command=['rm','-rf',stage1_install],
                                warnOnFailure=True,
                                haltOnFailure=False,
                                flunkOnFailure=False,
                                description='cleaning stage 1 install',
if useTwoStage:
         f.addStep(WarningCountingShellCommand(name='build stage 2',
                                               command=ninja_cmd,
                                               haltOnFailure=True,
                                               description='ninja all',
                                               workdir=stage2_build,
                                               env=env))
 
         if test:
-            f.addStep(lit_test_command.LitTestCommand(name='ninja check 2',
-                                       command=ninja_check_cmd,
-                                       haltOnFailure=not runTestSuite,
-                                       description=["checking stage 2"],
-                                       descriptionDone=["stage 2 checked"],
-                                       workdir=stage2_build,
-                                       env=env))
+            f.addStep(LitTestCommand(name='ninja check 2',
 
     ############# TEST SUITE
     ## Test-Suite (stage 2 if built, stage 1 otherwise)
     if runTestSuite:
         compiler_path = stage1_install
         if useTwoStage:
             compiler_path=stage2_install
             f.addStep(ShellCommand(name='clean stage 2 install',
if runTestSuite:
                                workdir='test',
                                env=env))
         f.addStep(ShellCommand(name='setup lit',
                                command=[python, lnt_setup, 'develop'],
                                haltOnFailure=True,
                                description='setting up LNT in sandbox',
                                workdir='test/sandbox',
                                env=env))
-        f.addStep(commands.LitTestCommand.LitTestCommand(
+        f.addStep(LitTestCommand(
                                name='test-suite',
                                command=test_suite_cmd,
                                haltOnFailure=True,
                                description=['running the test suite'],
                                workdir='test/sandbox',
                                logfiles={'configure.log'   : 'build/configure.log',
                                          'build-tools.log' : 'build/build-tools.log',
                                          'test.log'        : 'build/test.log',
                                          'report.json'     : 'build/report.json'},
                                env=test_suite_env))
 
     return f
-
-# FIXME: Deprecated.
-def addClangGCCTests(f, ignores={}, install_prefix="%(builddir)s/llvm.install",
-                     languages = ('gcc', 'g++', 'objc', 'obj-c++')):
-    make_vars = [WithProperties(
-            'CC_UNDER_TEST=%s/bin/clang' % install_prefix),
-                 WithProperties(
-            'CXX_UNDER_TEST=%s/bin/clang++' % install_prefix)]
-    f.addStep(SVN(name='svn-clang-gcc-tests', mode='update',
-                  baseURL='http://llvm.org/svn/llvm-project/clang-tests/',
-                  defaultBranch='trunk', workdir='clang-tests'))
-    gcc_dg_ignores = ignores.get('gcc-4_2-testsuite', {})
-    for lang in languages:
-        f.addStep(commands.SuppressionDejaGNUCommand.SuppressionDejaGNUCommand(
-            name='test-gcc-4_2-testsuite-%s' % lang,
-            command=["make", "-k", "check-%s" % lang] + make_vars,
-            description="gcc-4_2-testsuite (%s)" % lang,
-            workdir='clang-tests/gcc-4_2-testsuite',
-            logfiles={ 'dg.sum' : 'obj/%s/%s.sum' % (lang, lang),
-                       '%s.log' % lang : 'obj/%s/%s.log' % (lang, lang)},
-            ignore=gcc_dg_ignores.get(lang, [])))
-
-# FIXME: Deprecated.
-def addClangGDBTests(f, ignores={}, install_prefix="%(builddir)s/llvm.install"):
-    make_vars = [WithProperties(
-            'CC_UNDER_TEST=%s/bin/clang' % install_prefix),
-                 WithProperties(
-            'CXX_UNDER_TEST=%s/bin/clang++' % install_prefix)]
-    f.addStep(SVN(name='svn-clang-gdb-tests', mode='update',
-                  baseURL='http://llvm.org/svn/llvm-project/clang-tests/',
-                  defaultBranch='trunk', workdir='clang-tests'))
-    f.addStep(commands.SuppressionDejaGNUCommand.SuppressionDejaGNUCommand(
-            name='test-gdb-1472-testsuite',
-            command=["make", "-k", "check"] + make_vars,
-            description="gdb-1472-testsuite",
-            workdir='clang-tests/gdb-1472-testsuite',
-            logfiles={ 'dg.sum' : 'obj/filtered.gdb.sum',
-                       'gdb.log' : 'obj/gdb.log' }))
-
-# FIXME: Deprecated.
-def addModernClangGDBTests(f, jobs, install_prefix):
-    make_vars = [WithProperties('RUNTESTFLAGS=CC_FOR_TARGET=\'{0}/bin/clang\' '
-                                'CXX_FOR_TARGET=\'{0}/bin/clang++\' '
-                                'CFLAGS_FOR_TARGET=\'-w -fno-limit-debug-info\''
-                                .format(install_prefix))]
-    f.addStep(SVN(name='svn-clang-modern-gdb-tests', mode='update',
-                  svnurl='http://llvm.org/svn/llvm-project/clang-tests-external/trunk/gdb/7.5',
-                  workdir='clang-tests/src'))
-    f.addStep(Configure(command='../src/configure',
-                        workdir='clang-tests/build/'))
-    f.addStep(WarningCountingShellCommand(name='gdb-75-build',
-                                          command=['make', WithProperties('-j%s' % jobs)],
-                                          haltOnFailure=True,
-                                          workdir='clang-tests/build'))
-    f.addStep(commands.DejaGNUCommand.DejaGNUCommand(
-            name='gdb-75-check',
-            command=['make', '-k', WithProperties('-j%s' % jobs), 'check'] + make_vars,
-            workdir='clang-tests/build',
-            logfiles={'dg.sum':'gdb/testsuite/gdb.sum', 
-                      'gdb.log':'gdb/testsuite/gdb.log'}))
-
-
-
-# FIXME: Deprecated.
-addClangTests = addClangGCCTests
-
-def getClangTestsIgnoresFromPath(path, key):
-    def readList(path):
-        if not os.path.exists(path):
-            return []
-
-        f = open(path)
-        lines = [ln.strip() for ln in f]
-        f.close()
-        return lines
-
-    ignores = {}
-
-    gcc_dg_ignores = {}
-    for lang in ('gcc', 'g++', 'objc', 'obj-c++'):
-        lang_path = os.path.join(path, 'gcc-4_2-testsuite', 'expected_results',
-                                 key, lang)
-        gcc_dg_ignores[lang] = (
-            readList(os.path.join(lang_path, 'FAIL.txt')) +
-            readList(os.path.join(lang_path, 'UNRESOLVED.txt')) +
-            readList(os.path.join(lang_path, 'XPASS.txt')))
-    ignores['gcc-4_2-testsuite' ] = gcc_dg_ignores
-
-    ignores_path = os.path.join(path, 'gdb-1472-testsuite', 'expected_results',
-                                key)
-    gdb_dg_ignores = (
-        readList(os.path.join(ignores_path, 'FAIL.txt')) +
-        readList(os.path.join(ignores_path, 'UNRESOLVED.txt')) +
-        readList(os.path.join(ignores_path, 'XPASS.txt')))
-    ignores['gdb-1472-testsuite' ] = gdb_dg_ignores
-
-    return ignores

zorg/buildbot/builders/ClangLTOBuilder3Stage.py

-from buildbot.steps.source import SVN
-from buildbot.steps.shell import SetProperty
-from buildbot.steps.shell import ShellCommand
-from buildbot.steps.shell import WarningCountingShellCommand
-from buildbot.process.factory import BuildFactory
-from buildbot.process.properties import WithProperties
-from zorg.buildbot.commands.NinjaCommand import NinjaCommand
-
-def get3StageClangLTOBuildFactory(
-           clean=True,
-           env=None,
-           build_gold=False,
-           cmake_cache_file=None,
-           extra_cmake_options=None,
-    ):
-
-    llvm_srcdir = "llvm.src"
-    llvm_objdir = "llvm-stage1.obj"
-
-    merged_env = {
-        'TERM' : 'dumb' # Make sure Clang doesn't use color escape sequences.
-    }
-    if env is not None:
-        merged_env.update(env)
-
-    f = BuildFactory()
-
-    f.addStep(
-        SVN(
-            name='svn-llvm',
-            mode='update',
-            baseURL='http://llvm.org/svn/llvm-project/llvm/',
-            defaultBranch='trunk',
-            workdir=llvm_srcdir
-        )
-    )
-
-    f.addStep(
-        SVN(
-            name='svn-clang',
-            mode='update',
-            baseURL='http://llvm.org/svn/llvm-project/cfe/',
-            defaultBranch='trunk',
-            workdir='%s/tools/clang' % llvm_srcdir
-        )
-    )
-
-    # Clean directory, if requested.
-    cleanBuildRequested = lambda step: step.build.getProperty("clean") or clean
-    f.addStep(
-        ShellCommand(
-            doStepIf=cleanBuildRequested,
-            name="rm-llvm_objdir",
-            command=["rm", "-rf", llvm_objdir],
-            haltOnFailure=True,
-            description=["rm build dir", "llvm"],
-            workdir=".",
-            env=merged_env
-        )
-    )
-
-    cmake_command = ["cmake"]
-
-    if cmake_cache_file:
-        cmake_command += ['-C', cmake_cache_file]
-
-    if extra_cmake_options:
-        cmake_command += extra_cmake_options
-
-    cmake_command += [
-        "../%s" % llvm_srcdir
-    ]
-
-    # Note: ShellCommand does not pass the params with special symbols right.
-    # The " ".join is a workaround for this bug.
-    f.addStep(
-        WarningCountingShellCommand(
-            name="cmake-configure",
-            description=["cmake configure"],
-            haltOnFailure=True,
-            command=WithProperties(" ".join(cmake_command)),
-            workdir=llvm_objdir,
-            env=merged_env
-        )
-    )
-
-    if build_gold:
-        f.addStep(
-            NinjaCommand(name='build-LLVMgold.so',
-                targets=['lib/LLVMgold.so'],
-                haltOnFailure=True,
-                warnOnWarnings=True,
-                description=["3 Stage Build Clang"],
-                workdir=llvm_objdir,
-                env=merged_env)
-        )
-
-    f.addStep(
-        NinjaCommand(name='build-stage3-clang',
-            targets=['stage3-clang'],
-            haltOnFailure=True,
-            warnOnWarnings=True,
-            description=["3 Stage Build Clang"],
-            timeout=3600, # Default value is not enough.
-            workdir=llvm_objdir,
-            env=merged_env)
-    )
-
-    f.addStep(
-        NinjaCommand(name='build-stage3-check-clang',
-            targets=['stage3-check-clang'],
-            haltOnFailure=True,
-            warnOnWarnings=True,
-            description=["Check Clang"],
-            timeout=3600, # Default value is not enough.
-            workdir=llvm_objdir,
-            env=merged_env)
-    )
-
-    # Compare stage2 & stage3 clang
-    shell_command = [
-        "diff",
-        "-q",
-        "tools/clang/stage2-bins/bin/clang",
-        "tools/clang/stage2-bins/tools/clang/stage3-bins/bin/clang"
-    ]
-    f.addStep(
-        ShellCommand(
-            name="compare-clang",
-            description=["comapre stage2 & stage3 clang"],
-            haltOnFailure=True,
-            command=WithProperties(" ".join(shell_command)),
-            workdir=llvm_objdir,
-            env=merged_env
-        )
-    )
-
-    return f

zorg/buildbot/builders/LLDBuilder.py

-import os
-
-import buildbot
-import buildbot.process.factory
-from buildbot.steps.source import SVN
-from buildbot.steps.shell import ShellCommand, SetProperty
-from buildbot.steps.slave import RemoveDirectory
-from buildbot.process.properties import WithProperties, Property
-from zorg.buildbot.builders.Util import getVisualStudioEnvironment
-from zorg.buildbot.builders.Util import extractSlaveEnvironment
-from zorg.buildbot.commands.CmakeCommand import CmakeCommand
-from zorg.buildbot.commands.NinjaCommand import NinjaCommand
-from zorg.buildbot.conditions.FileConditions import FileDoesNotExist
-from zorg.buildbot.process.factory import LLVMBuildFactory
-
-def getLLDBuildFactory(
-           clean = True,
-           jobs  = None,
-           extra_configure_args = None,
-           env   = None):
-
-    # Set defaults
-    if jobs is None:
-        jobs = "%(jobs)s"
-    if extra_configure_args is None:
-        extra_configure_args = []
-
-    # Prepare environmental variables. Set here all env we want everywhere.
-    merged_env = {
-                   'CC'   : "clang",
-                   'CXX'  : "clang++",
-                   'TERM' : 'dumb'     # Be cautious and disable color output from all tools.
-                 }
-    if env is not None:
-        # Overwrite pre-set items with the given ones, so user can set anything.
-        merged_env.update(env)
-
-    f = LLVMBuildFactory(
-            depends_on_projects=['llvm', 'lld'],
-            llvm_srcdir="llvm.src",
-            llvm_objdir="llvm.obj")
-
-    # Get LLVM and Lld
-    f.addSVNSteps()
-
-    # Clean directory, if requested.
-    cleanBuildRequested = lambda step: step.build.getProperty("clean") or clean
-    f.addStep(RemoveDirectory(name='clean ' + f.llvm_objdir,
-              dir=f.llvm_objdir,
-              haltOnFailure=False,
-              flunkOnFailure=False,
-              doStepIf=cleanBuildRequested
-              ))
-
-    # Create configuration files with cmake
-    f.addStep(CmakeCommand(name="cmake-configure",
-                           description=["cmake configure"],
-                           haltOnFailure=True,
-                           options=extra_configure_args,
-                           path="../%s" % f.llvm_srcdir,
-                           env=merged_env,
-                           workdir=f.llvm_objdir,
-                           doStepIf=FileDoesNotExist(
-                                        "./%s/CMakeCache.txt" % f.llvm_objdir)))
-
-    # Build Lld
-    f.addStep(ShellCommand(name="build_Lld",
-                           command=['nice', '-n', '10',
-                                    'make', WithProperties("-j%s" % jobs)],
-                           haltOnFailure=True,
-                           description=["build lld"],
-                           env=merged_env,
-                           workdir=f.llvm_objdir))
-
-    # Test Lld
-    f.addStep(ShellCommand(name="test_lld",
-                           command=["make", "lld-test"],
-                           haltOnFailure=True,
-                           description=["test lld"],
-                           env=merged_env,
-                           workdir=f.llvm_objdir))
-
-    return f
-
-
-def getLLDWinBuildFactory(
-           clean = True,
-
-           # Default values for VS devenv and build configuration
-           vs = None,          # What to run to configure Visual Studio utils.
-           target_arch = None, # Native.
-
-           extra_configure_args = None,
-           env   = None):
-
-    # Set defaults
-    if vs is None:
-        vs = r"""%VS140COMNTOOLS%"""   # Visual Studio 2015.
-    if extra_configure_args is None:
-        extra_configure_args = []
-    if env is None:
-        env = {}
-
-    f = LLVMBuildFactory(
-            depends_on_projects=['llvm', 'lld'],
-            llvm_srcdir="llvm.src",
-            llvm_objdir="llvm.obj")
-
-    # Get LLVM and Lld
-    f.addSVNSteps()
-
-    # Clean directory, if requested.
-    cleanBuildRequested = lambda step: step.build.getProperty("clean") or clean
-    f.addStep(RemoveDirectory(name='clean ' + f.llvm_objdir,
-              dir=f.llvm_objdir,
-              haltOnFailure=False,
-              flunkOnFailure=False,
-              doStepIf=cleanBuildRequested
-              ))
-
-    # If set up environment step is requested, do this now.
-    if vs:
-        f.addStep(SetProperty(
-            command=getVisualStudioEnvironment(vs, target_arch),
-            extract_fn=extractSlaveEnvironment))
-        assert not env, "Can't have custom builder env vars with VS"
-        env = Property('slave_env')
-
-    # Always build with ninja.
-    cmake_options = ["-G", "Ninja"]
-    # Reconsile configure args with the defaults we want.
-    if not any(a.startswith('-DCMAKE_BUILD_TYPE=')   for a in extra_configure_args):
-        cmake_options.append('-DCMAKE_BUILD_TYPE=Release')
-    if not any(a.startswith('-DLLVM_ENABLE_WERROR=') for a in extra_configure_args):
-        cmake_options.append('-DLLVM_ENABLE_WERROR=ON')
-    if not any(a.startswith('-DLLVM_ENABLE_ASSERTIONS=') for a in extra_configure_args):
-        cmake_options.append('-DLLVM_ENABLE_ASSERTIONS=ON')
-    if not any(a.startswith('-DLLVM_LIT_ARGS=') for a in extra_configure_args):
-        cmake_options.append('-DLLVM_LIT_ARGS=\"-v\"')
-
-    cmake_options += extra_configure_args
-
-    # Note: ShellCommand does not pass the params with special symbols right.
-    # The " ".join is a workaround for this bug.
-    f.addStep(CmakeCommand(name="cmake-configure",
-                           description=["cmake configure"],
-                           haltOnFailure=True,
-                           warnOnWarnings=True,
-                           options=cmake_options,
-                           path="../%s" % f.llvm_srcdir,
-                           env=env,
-                           workdir=f.llvm_objdir,
-                           doStepIf=FileDoesNotExist(
-                                        "./%s/CMakeCache.txt" % f.llvm_objdir)))
-
-    # Build Lld.
-    f.addStep(NinjaCommand(name='build lld',
-                           haltOnFailure=True,
-                           warnOnWarnings=True,
-                           description='build lld',
-                           workdir=f.llvm_objdir,
-                           env=env))
-
-    # Test Lld
-    f.addStep(NinjaCommand(name='test lld',
-                           targets=['lld-test'],
-                           haltOnFailure=True,
-                           warnOnWarnings=True,
-                           description='test lld',
-                           workdir=f.llvm_objdir,
-                           env=env))
-
-    return f

zorg/buildbot/builders/LLVMBuilder.py

-import os
-
-import buildbot
-import buildbot.process.factory
-from buildbot.steps.source import SVN
-from buildbot.steps.shell import Configure, ShellCommand
-from buildbot.steps.shell import WarningCountingShellCommand
-from buildbot.process.properties import WithProperties
-
-from zorg.buildbot.commands.LitTestCommand import LitTestCommand
-
-from Util import getConfigArgs
-
-def getLLVMCMakeBuildFactory(
-                  clean                = True,             # "clean-llvm" step is requested if true.
-                  test                 = True,             # "test-llvm" step is requested if true.
-                  jobs                 = '%(jobs)s',       # Number of concurrent jobs.
-                  timeout              = 20,               # Timeout if no activity seen (minutes).
-                  make                 = 'make',           # Make command.
-                  enable_shared        = False,            # Enable shared (-DBUILD_SHARED_LIBS=ON configure parameters added) if true.
-                  defaultBranch        = 'trunk',          # Branch to build.
-                  config_name          = 'Debug',          # Configuration name.
-                  env                  = None,             # Environmental variables for all steps.
-                  extra_cmake_args = []):                  # Extra args for the cmake step.
-    # Prepare environmental variables. Set here all env we want everywhere.
-    merged_env = {
-                   'TERM' : 'dumb'     # Make sure Clang doesn't use color escape sequences.
-                 }
-    if env is not None:
-        merged_env.update(env)  # Overwrite pre-set items with the given ones, so user can set anything.
-
-    llvm_srcdir = "llvm.src"
-    llvm_objdir = "llvm.obj"
-
-    f = buildbot.process.factory.BuildFactory()
-
-    # Determine the build directory.
-    f.addStep(
-        buildbot.steps.shell.SetProperty(
-            name        = "get_builddir",
-            command     = ["pwd"],
-            property    = "builddir",
-            description = "set build dir",
-            workdir     = ".",
-            env         = merged_env))
-
-    # Checkout sources.
-    f.addStep(
-        SVN(
-            name          = 'svn-llvm',
-            mode          = 'update', baseURL='http://llvm.org/svn/llvm-project/llvm/',
-            defaultBranch = defaultBranch,
-            workdir       = llvm_srcdir))
-
-    cmake_args = ['cmake']
-    cmake_args += ["-DCMAKE_BUILD_TYPE="+config_name]
-    if enable_shared:
-        cmake_args.append('-DBUILD_SHARED_LIBS=ON')
-    cmake_args.extend(extra_cmake_args)
-    cmake_args += ['../' + llvm_srcdir]
-    f.addStep(
-        Configure(
-            command         = cmake_args,
-            description     = ['configuring', config_name],
-            descriptionDone = ['configure',   config_name],
-            workdir         = llvm_objdir,
-            env             = merged_env))
-    if clean:
-        f.addStep(
-            WarningCountingShellCommand(
-                name            = "clean-llvm",
-                command         = [make, 'clean'],
-                haltOnFailure   = True,
-                description     = "cleaning llvm",
-                descriptionDone = "clean llvm",
-                workdir         = llvm_objdir,
-                env             = merged_env))
-    f.addStep(
-        WarningCountingShellCommand(
-            name            = "compile",
-            command         = ['nice', '-n', '10',
-                               make, WithProperties("-j%s" % jobs)],
-            haltOnFailure   = True,
-            description     = "compiling llvm",
-            descriptionDone = "compile llvm",
-            workdir         = llvm_objdir,
-            env             = merged_env,
-            timeout         = timeout * 60))
-    if test:
-        litTestArgs = '-v -j %s' % jobs
-        f.addStep(
-            LitTestCommand(
-                name            = 'test-llvm',
-                command         = [make, "check-all", "VERBOSE=1",
-                                   WithProperties("-j%s" % jobs),
-                                   WithProperties("LIT_ARGS=%s" % litTestArgs)],
-                description     = ["testing", "llvm"],
-                descriptionDone = ["test",    "llvm"],
-                workdir         = llvm_objdir,
-                env             = merged_env))
-    return f

zorg/buildbot/builders/LLVMGCCBuilder.py

-import buildbot
-from buildbot.steps.source import SVN
-from buildbot.steps.shell import Configure, WarningCountingShellCommand
-from buildbot.steps.shell import ShellCommand, SetProperty
-from buildbot.process.properties import WithProperties
-
-from zorg.buildbot.commands.LitTestCommand import LitTestCommand
-
-from Util import getConfigArgs
-
-def getLLVMGCCBuildFactory(jobs='%(jobs)s', update=True, clean=True,
-                           gxxincludedir=None,
-                           triple=None, build=None, host=None, target=None,
-                           useTwoStage=True, stage1_config='Release+Asserts',
-                           stage2_config='Release+Asserts', make='make',
-                           extra_configure_args=[], extra_languages=None,
-                           verbose=False, env = {}, defaultBranch='trunk',
-                           timeout=20, package_dst=None):
-  if build or host or target:
-    if not build or not host or not target:
-      raise ValueError,"Must specify all of 'build', 'host', 'target' if used."
-    if triple:
-      raise ValueError,"Cannot specify 'triple' and 'build', 'host', 'target' options."
-  elif triple:
-    build = host = target = triple
-
-    # Prepare environmental variables. Set here all env we want everywhere.
-    merged_env = {
-                   'TERM' : 'dumb'     # Make sure Clang doesn't use color escape sequences.
-                 }
-    if env is not None:
-        merged_env.update(env)  # Overwrite pre-set items with the given ones, so user can set anything.
-
-  f = buildbot.process.factory.BuildFactory()
-
-  # Determine the build directory.
-  f.addStep(buildbot.steps.shell.SetProperty(name="get_builddir",
-                                             command=["pwd"],
-                                             property    = "builddir",
-                                             description = "set build dir",
-                                             workdir     = ".",
-                                             env         = merged_env))
-
-  # Get the sources.
-  if update:
-    f.addStep(SVN(name='svn-llvm',
-                  mode='update', baseURL='http://llvm.org/svn/llvm-project/llvm/',
-                  defaultBranch = defaultBranch,
-                  workdir       = "llvm.src"))
-
-    f.addStep(SVN(name='svn-llvm-gcc',
-                  mode='update', baseURL='http://llvm.org/svn/llvm-project/llvm-gcc-4.2/',
-                  defaultBranch = defaultBranch,
-                  workdir       = "llvm-gcc.src"))
-
-  # Clean up llvm (stage 1).
-  if clean:
-    f.addStep(ShellCommand(name="rm-llvm.obj.stage1",
-                           command=["rm", "-rf", "llvm.obj"],
-                           haltOnFailure = True,
-                           description   = ["rm build dir",
-                                            "llvm",
-                                            "(stage 1)"],
-                           workdir       = ".",
-                           env           = merged_env))
-
-  # Configure llvm (stage 1).
-  base_llvm_configure_args = [WithProperties("%(builddir)s/llvm.src/configure")]
-  if build:
-    base_llvm_configure_args.append('--build=' + build)
-    base_llvm_configure_args.append('--host=' + host)
-    base_llvm_configure_args.append('--target=' + target)
-  stage_configure_args = getConfigArgs(stage1_config)
-  f.addStep(Configure(name='configure.llvm.stage1',
-                      command=base_llvm_configure_args +
-                              stage_configure_args +
-                              ["--without-llvmgcc",
-                               "--without-llvmgxx"],
-                      description = [ "configure",
-                                      "llvm",
-                                      "(stage 1)",
-                                      stage1_config ],
-                      workdir     = "llvm.obj",
-                      env         = merged_env))
-
-  # Build llvm (stage 1).
-  base_llvm_make_args = ['nice', '-n', '10',
-                         make, WithProperties("-j%s" % jobs)]
-  if verbose:
-    base_llvm_make_args.append('VERBOSE=1')
-  f.addStep(WarningCountingShellCommand(name = "compile.llvm.stage1",
-                                        command       = base_llvm_make_args,
-                                        haltOnFailure = True,
-                                        description   = ["compile",
-                                                         "llvm",
-                                                         "(stage 1)",
-                                                         stage1_config],
-                                        workdir       = "llvm.obj",
-                                        env           = merged_env,
-                                        timeout       = timeout * 60))
-
-  # Run LLVM tests (stage 1).
-  f.addStep(LitTestCommand(name = 'test.llvm.stage1',
-                             command = [make, "check-lit", "VERBOSE=1"],
-                             description     = ["testing", "llvm"],
-                             descriptionDone = ["test",    "llvm"],
-                             workdir         = 'llvm.obj',
-                             env             = merged_env))
-
-  # Clean up llvm-gcc.
-  if clean:
-    f.addStep(ShellCommand(name="rm-llvm-gcc.obj.stage1",
-                           command=["rm", "-rf", "llvm-gcc.obj"],
-                           haltOnFailure = True,
-                           description   = ["rm build dir",
-                                            "llvm-gcc"],
-                           workdir       = ".",
-                           env           = merged_env))
-
-  # Configure llvm-gcc.
-  base_llvmgcc_configure_args = ["../llvm-gcc.src/configure"]
-  llvmgcc_languages = "--enable-languages=c,c++"
-  if extra_languages:
-    llvmgcc_languages = llvmgcc_languages + "," + extra_languages
-  base_llvmgcc_configure_args.append(llvmgcc_languages)
-  if gxxincludedir:
-    base_llvmgcc_configure_args.append('--with-gxx-include-dir=' + gxxincludedir)
-  base_llvmgcc_configure_args.extend(extra_configure_args)
-  if build:
-    base_llvmgcc_configure_args.append('--build=' + build)
-    base_llvmgcc_configure_args.append('--host=' + host)
-    base_llvmgcc_configure_args.append('--target=' + target)
-  f.addStep(Configure(name='configure.llvm-gcc.stage1',
-                      command=(base_llvmgcc_configure_args +
-                               ["--program-prefix=llvm-",
-                                WithProperties("--prefix=%(builddir)s/llvm-gcc.install"),                      
-                                WithProperties("--enable-llvm=%(builddir)s/llvm.obj")]),
-                      haltOnFailure = True,
-                      description   = ["configure",
-                                       "llvm-gcc",
-                                       "(stage 1)"],
-                      workdir       = "llvm-gcc.obj",
-                      env           = merged_env))
-
-  # Build llvm-gcc.
-  f.addStep(WarningCountingShellCommand(name="compile.llvm-gcc.stage1",
-                                        command=['nice', '-n', '10',
-                                                 make, WithProperties("-j%s" % jobs)],
-                                        haltOnFailure = True,
-                                        description   = ["compile",
-                                                         "llvm-gcc"],
-                                        workdir       = "llvm-gcc.obj",
-                                        env           = merged_env,
-                                        timeout       = timeout * 60))
-
-  # Clean up llvm-gcc install.
-  if clean:
-    f.addStep(ShellCommand(name="rm-llvm-gcc.install.stage1",
-                           command=["rm", "-rf", "llvm-gcc.install"],
-                           haltOnFailure = True,
-                           description   = ["rm install dir",
-                                            "llvm-gcc"],
-                           workdir       = ".",
-                           env           = merged_env))
-
-  # Install llvm-gcc.
-  f.addStep(WarningCountingShellCommand(name="install.llvm-gcc.stage1",
-                                        command=['nice', '-n', '10',
-                                                 make, 'install'],
-                                        haltOnFailure = True,
-                                        description   = ["install",
-                                                         "llvm-gcc"],
-                                        workdir       = "llvm-gcc.obj",
-                                        env           = merged_env))
-
-  # We are done if not doing a two-stage build.
-  if not useTwoStage:
-    return f
-
-  # Clean up llvm (stage 2).
-  if clean:
-    f.addStep(ShellCommand(name="rm-llvm.obj.stage2",
-                           command=["rm", "-rf", "llvm.obj.2"],
-                           haltOnFailure = True,
-                           description   = ["rm build dir",
-                                            "llvm",
-                                            "(stage 2)"],
-                           workdir       = ".",
-                           env           = merged_env))
-
-  # Configure llvm (stage 2).
-  stage_configure_args = getConfigArgs(stage2_config)
-  local_env = dict(merged_env)
-  local_env['CC'] = WithProperties("%(builddir)s/llvm-gcc.install/bin/llvm-gcc")
-  local_env['CXX'] = WithProperties("%(builddir)s/llvm-gcc.install/bin/llvm-g++")
-  f.addStep(Configure(name="configure.llvm.stage2",
-                      command=base_llvm_configure_args + 
-                              stage_configure_args +
-                              [WithProperties("--with-llvmgcc=%(builddir)s/llvm-gcc.install/bin/llvm-gcc"),
-                               WithProperties("--with-llvmgxx=%(builddir)s/llvm-gcc.install/bin/llvm-g++")],
-                      haltOnFailure = True,
-                      description   = ["configure",
-                                       "llvm",
-                                       "(stage 2)",
-                                       stage2_config],
-                      workdir      = "llvm.obj.2",
-                      env          = local_env))
-
-  # Build LLVM (stage 2).
-  f.addStep(WarningCountingShellCommand(name = "compile.llvm.stage2",
-                                        command = base_llvm_make_args,
-                                        haltOnFailure = True,
-                                        description   = ["compile",
-                                                         "llvm",
-                                                         "(stage 2)",
-                                                         stage2_config],
-                                        workdir       = "llvm.obj.2",
-                                        env           = merged_env,
-                                        timeout       = timeout * 60))
-
-  # Run LLVM tests (stage 2).
-  f.addStep(LitTestCommand(name = 'test.llvm.stage2',
-                             command = [make, "check-lit", "VERBOSE=1"],
-                             description     = ["testing", "llvm", "(stage 2)"],
-                             descriptionDone = ["test",    "llvm", "(stage 2)"],
-                             workdir         = 'llvm.obj.2',
-                             env             = merged_env))
-
-  # Clean up llvm-gcc (stage 2).
-  if clean:
-    f.addStep(ShellCommand(name="rm-llvm-gcc.obj.stage2",
-                           command=["rm", "-rf", "llvm-gcc.obj.2"],
-                           haltOnFailure = True,
-                           description   = ["rm build dir",
-                                            "llvm-gcc",
-                                            "(stage 2)"],
-                           workdir       = ".",
-                           env           = merged_env))
-
-  # Configure llvm-gcc (stage 2).
-  local_env = dict(merged_env)
-  local_env['CC'] = WithProperties("%(builddir)s/llvm-gcc.install/bin/llvm-gcc")
-  local_env['CXX'] = WithProperties("%(builddir)s/llvm-gcc.install/bin/llvm-g++")
-  f.addStep(Configure(name = 'configure.llvm-gcc.stage2',
-                      command=base_llvmgcc_configure_args + [
-                               "--program-prefix=llvm.2-",
-                               WithProperties("--prefix=%(builddir)s/llvm-gcc.install.2"),
-                               WithProperties("--enable-llvm=%(builddir)s/llvm.obj.2")],
-                      haltOnFailure = True,
-                      description   = ["configure",
-                                       "llvm-gcc",
-                                       "(stage 2)"],
-                      workdir       = "llvm-gcc.obj.2",
-                      env           = local_env))
-
-  # Build llvm-gcc (stage 2).
-  f.addStep(WarningCountingShellCommand(name="compile.llvm-gcc.stage2",
-                                        command=['nice', '-n', '10',
-                                                 make, WithProperties("-j%s" % jobs)],
-                                        haltOnFailure = True,
-                                        description   = ["compile",
-                                                         "llvm-gcc",
-                                                         "(stage 2)"],
-                                        workdir       = "llvm-gcc.obj.2",
-                                        env           = merged_env,
-                                        timeout       = timeout * 60))
-
-  # Clean up llvm-gcc install (stage 2).
-  if clean:
-    f.addStep(ShellCommand(name="rm-llvm-gcc.install.stage2",
-                           command=["rm", "-rf", "llvm-gcc.install.2"],
-                           haltOnFailure = True,
-                           description   = ["rm install dir",
-                                            "llvm-gcc",
-                                            "(stage 2)"],
-                           workdir       = ".",
-                           env           = merged_env))
-
-  # Install llvm-gcc.
-  f.addStep(WarningCountingShellCommand(name="install.llvm-gcc.stage2",
-                                        command = ['nice', '-n', '10',
-                                                   make, 'install'],
-                                        haltOnFailure = True,
-                                        description   = ["install",
-                                                         "llvm-gcc",
-                                                         "(stage 2)"],
-                                        workdir       = "llvm-gcc.obj.2",
-                                        env           = merged_env))
-  if package_dst:
-    addPackageStep(f, package_dst, obj_path = 'llvm-gcc.install.2', env = merged_env)
-
-  return f
-
-import os
-def addPackageStep(f, package_dst,
-                   obj_path,
-                   info_string = '%(phase_id)s',
-                   env         = {}):
-
-  # Package and upload.
-    name = WithProperties(
-      os.path.join("%(builddir)s", obj_path,
-                   "llvm-gcc-%s.tar.gz" % info_string))
-
-    f.addStep(ShellCommand(name           = 'pkg.tar',
-                           description    = "tar root",
-                           command        = ["tar", "zcvf", name, "./"],
-                           workdir        = obj_path,
-                           env            = env,
-                           warnOnFailure  = True,
-                           flunkOnFailure = False,
-                           haltOnFailure  = False))
-
-    f.addStep(ShellCommand(name           = 'pkg.upload',
-                           description    = "upload root",
-                           command        = ["scp", name, package_dst],
-                           workdir        = ".",
-                           env            = env,
-                           warnOnFailure  = True,
-                           flunkOnFailure = False,
-                           haltOnFailure  = False))

zorg/buildbot/builders/LNTBuilder.py

-"""
-Builders for using LNT to test LLVM/Clang.
-"""
-
-import os
-
-import buildbot
-from buildbot.steps.source.svn import SVN
-from buildbot.process.properties import WithProperties
-
-import zorg
-from zorg.buildbot.builders import ClangBuilder
-from zorg.buildbot.util.phasedbuilderutils import getBuildDir, setProperty
-from zorg.buildbot.util.artifacts import GetCompilerRoot, package_url
-
-def _get_cc(status, stdin, stdout):
-    lines = filter(bool, stdin.split('\n'))
-    for line in lines:
-        if 'bin/clang' in line:
-            cc_path = line
-            return { 'cc_path' : cc_path }
-    return {}
-
-def _get_cxx(status, stdin, stdout):
-    lines = filter(bool, stdin.split('\n'))
-    for line in lines:
-        if 'bin/clang++' in line:
-            cxx_path = line
-            return { 'cxx_path' : cxx_path }
-    return {}
-
-def _get_liblto(status, stdin, stdout):
-    lines = filter(bool, stdin.split('\n'))
-    for line in lines:
-        if 'lib/libLTO.dylib' in line:
-            lto_path = line
-            return { 'lto_path' : lto_path }
-    return {}
-
-def getLNTFactory(triple, nt_flags, xfails=[], clean=True, test=False,
-                  reportBuildslave = True, **kwargs):
-    lnt_args = {}
-    lnt_arg_names = ['submitURL', 'package_cache', 'testerName',
-                     'reportBuildslave']
-
-    for argname in lnt_arg_names:
-        if argname in kwargs:
-            lnt_args[argname] = kwargs.pop(argname)
-
-    # Build compiler to test.  
-    f = ClangBuilder.getClangBuildFactory(
-        triple, clean=clean, test=test,
-        stage1_config='Release+Asserts', **kwargs)
-
-    # Add an LNT test runner.
-    AddLNTTestsToFactory(f, nt_flags,
-                         cc_path="llvm.install.1/bin/clang",
-                         cxx_path="llvm.install.1/bin/clang++",
-                         **lnt_args);
-
-    return f
-
-def AddLNTTestsToFactory(f, nt_flags, cc_path, cxx_path, **kwargs):
-    """
-    Add the buildbot steps necessary to run an LNT driven test of a compiler.
-
-    This assumes at a minimum that the factory has already been set up to
-    contain a builddir property which points at the full path to the build
-    directory.
-    """
-
-    parallel = kwargs.pop('parallel', False)
-    jobs = kwargs.pop('jobs', '$(jobs)s')
-    submitURL = kwargs.pop('submitURL', None)
-    package_cache = kwargs.pop('package_cache', 'http://lab.llvm.org/packages')
-    testerName = kwargs.pop('testerName', None)
-    reportBuildslave = kwargs.pop('reportBuildslave', True)
-    env = kwargs.pop('env', {})
-
-    # Create variables to refer to the compiler-under-test.
-    #
-    # We assume any relative paths are relative to the build directory (which
-    # prior steps will have presumably populated with a compiler).
-    cc_path = WithProperties(os.path.join('%(builddir)s', cc_path))
-    cxx_path = WithProperties(os.path.join('%(builddir)s', cxx_path))
-
-    # Add --liblto-path if necessary. We assume it will be in a lib directory
-    # adjacent to cc_path.
-    #
-    # FIXME: This is currently only going to work on Darwin.
-    if '-flto' in nt_flags:
-        base_directory = os.path.dirname(os.path.dirname(cc_path))
-        nt_flags.extend(['--liblto-path', WithProperties(
-                         os.path.join('%(builddir)s', base_directory, 'lib',
-                                      'libLTO.dylib'))])
-
-    # Get the LNT sources.
-    f.addStep(SVN(name='pull.lnt', mode='incremental', method='fresh',
-                  baseURL='http://llvm.org/svn/llvm-project/lnt/',
-                  defaultBranch='trunk', workdir='lnt', alwaysUseLatest=True))
-
-    # Get the LLVM test-suite sources.
-    f.addStep(SVN(name='pull.test-suite', mode='incremental', method='fresh',
-                  baseURL='http://llvm.org/svn/llvm-project/test-suite/',
-                  defaultBranch='trunk', workdir='test-suite',
-                  alwaysUseLatest=False))
-
-    # Create the LNT virtual env.
-    f.addStep(buildbot.steps.shell.ShellCommand(
-            name='venv.lnt.clean', command=['rm', '-rf', 'lnt.venv'],
-            haltOnFailure=True, description=['clean', 'LNT', 'venv'],
-            workdir=WithProperties('%(builddir)s')))
-    f.addStep(buildbot.steps.shell.ShellCommand(
-            name='venv.lnt.create', command=['virtualenv', 'lnt.venv'],
-            haltOnFailure=True, description=['create', 'LNT', 'venv'],
-            workdir=WithProperties('%(builddir)s'), 
-            env={'PATH' : '${PATH}:/usr/local/bin'}))
-    f.addStep(buildbot.steps.shell.ShellCommand(
-            name='venv.lnt.install', haltOnFailure=True,
-            command=[WithProperties('%(builddir)s/lnt.venv/bin/pip'), 'install',
-                     '--no-index',
-                     '--find-links', package_cache,
-                     '-e', '.'],
-            description=['install', 'LNT'], workdir='lnt',
-            env={'ARCHFLAGS' : '-arch i386 -arch x86_64'}))
-
-    # Clean up the sandbox dir.
-    f.addStep(buildbot.steps.shell.ShellCommand(
-            name='lnt.nightly-test.clean', command=['rm', '-rf', 'nt'],
-            haltOnFailure=True, description=['clean', 'LNT', 'sandbox'],
-            workdir='tests'))
-
-    if reportBuildslave:
-        reportName = '%(slavename)s'
-
-        if testerName:
-            reportName += '__' + testerName
-        reportName = WithProperties(reportName)
-    else:
-        reportName = testerName
-
-    # Run the nightly test.
-    args = [WithProperties('%(builddir)s/lnt.venv/bin/python'),
-            WithProperties('%(builddir)s/lnt.venv/bin/lnt'),
-            'runtest', 'nt', '--verbose']
-
-
-    # Only submit if a URL has been specified
-    if submitURL is not None:
-      if type(submitURL) != type([]):
-        submitURL = [submitURL]
-
-      for url in submitURL:
-        args.extend(['--submit', url])
-
-    args.extend(['--commit', '1',
-             '--sandbox', 'nt',
-             '--no-timestamp',
-             '--cc', cc_path, '--cxx', cxx_path,
-             '--without-llvm',
-             '--test-suite', WithProperties('%(builddir)s/test-suite'), 
-             '--no-machdep-info', reportName])
-    if parallel:
-        args.extend(['-j', WithProperties(jobs)])
-    args.extend(nt_flags)
-    f.addStep(zorg.buildbot.commands.LitTestCommand.LitTestCommand(
-            name='lnt.nightly-test', command=args, haltOnFailure=True,
-            description=['nightly test'], workdir='tests',
-            logfiles={'configure.log' : 'nt/build/configure.log',
-                      'build-tools.log' : 'nt/build/build-tools.log',
-                      'test.log' : 'nt/build/test.log',
-                      'report.json' : 'nt/build/report.json'},
-            env=env))
-    return f
-
-def CreateLNTNightlyFactory(nt_flags, cc_path=None, cxx_path=None,
-                            parallel = False, jobs = '%(jobs)s',
-                            db_url=None, external_URL=None):
-    # Paramaters used by this method:
-    # nt_flags  : a list of flags passed to the lnt process
-    # cc_path   : explicit path to c compiler
-    # cxx_path  : explicit path to c++ compiler
-    # parallel  : set to True if using multiple cores for faster turnaround
-    #             set to False if measuring performance
-    # db_url    : set to the submission URL for an LNT database when measuring 
-    #             performance
-    # external_URL : Used to pull additional tests from a separate svn 
-    #                repository
-    # Properties set externally but used by this method:
-    # builddir  : This property is set below
-    # jobs      : This property is set by the slave, it indicates the number of
-    #             cores availble to use.
-    # revision  : This property should be set by an upstream builder.
-    # slavename : This property is set by the slave
-    # buildername : This property is set by the master
-
-    f = buildbot.process.factory.BuildFactory()
-    # Determine the build directory.
-    f = getBuildDir(f)
-    f = GetCompilerRoot(f)
-    if cc_path:
-       cc_command = ['echo', cc_path]
-    else:
-       cc_command = ['find', 'host-compiler', '-name', 'clang']
-    f.addStep(buildbot.steps.shell.SetProperty(
-              name='find.cc',
-              command=cc_command,
-              extract_fn=_get_cc,
-              workdir=WithProperties('%(builddir)s')))
-    if cxx_path:
-       cc_command = ['echo', cxx_path]
-    else:
-       cc_command = ['find', 'host-compiler', '-name', 'clang++']
-    f.addStep(buildbot.steps.shell.SetProperty(
-              name='find.cxx',
-              command=cc_command,
-              extract_fn=_get_cxx,
-              workdir=WithProperties('%(builddir)s')))
-    f.addStep(buildbot.steps.shell.ShellCommand(
-            name='sanity.test', haltOnFailure=True,
-            command=[WithProperties('%(builddir)s/%(cc_path)s'), '-v'],
-            description=['sanity test']))
-    args = [WithProperties('%(builddir)s/lnt.venv/bin/python'),
-            WithProperties('%(builddir)s/lnt.venv/bin/lnt'),
-            'runtest', '--verbose']
-    if db_url:
-        f.addStep(buildbot.steps.shell.SetProperty(
-                  name='db_url',
-                  command=['echo', db_url, ':', WithProperties('%(buildername)s')],
-                  extract_fn=_get_db_url,
-                  workdir=WithProperties('%(builddir)s')))
-        args.extend(['--submit', WithProperties('%(db_url)s')])
-    # Add --liblto-path if necessary.
-    if '-flto' in nt_flags:
-        f.addStep(buildbot.steps.shell.SetProperty(
-                  name='find.liblto',
-                  command=['find', 'host-compiler', '-name', 'libLTO.dylib'],
-                  extract_fn=_get_liblto,
-                  workdir=WithProperties('%(builddir)s')))
-        nt_flags.extend(['--liblto-path', WithProperties('%(builddir)s/%(lto_path)s')])
-    # Get the LNT sources.
-    f.addStep(buildbot.steps.shell.ShellCommand(
-            name = 'svn.clean.lnt',
-            command = ['svn', 'cleanup'],
-            haltOnFailure=False, flunkOnFailure=False,
-            description = ['svn clean lnt'],
-            workdir='lnt'))
-    f.addStep(SVN(name='pull.lnt', mode='full', method='fresh',
-                  repourl='http://llvm.org/svn/llvm-project/lnt/trunk',
-                  haltOnFailure=False, flunkOnFailure=False,
-                  workdir='lnt', alwaysUseLatest=True, retry = (60, 5),
-                  description='pull.lnt'))
-    # Get the LLVM test-suite sources.
-    f.addStep(buildbot.steps.shell.ShellCommand(
-            name = 'svn.clean.tests',
-            command = ['svn', 'cleanup'],
-            haltOnFailure=False, flunkOnFailure=False,
-            description = ['svn clean test-suite'],
-            workdir='test-suite'))
-    f.addStep(SVN(name='pull.test-suite', mode='full', method='fresh',
-                  repourl='http://llvm.org/svn/llvm-project/test-suite/trunk',
-                  haltOnFailure=False, flunkOnFailure=False,
-                  workdir='test-suite', retry = (60, 5),
-                  description='pull.test-suite'))
-    if external_URL:
-        external_dir = WithProperties('%(builddir)s/test-suite-externals')
-        f.addStep(buildbot.steps.shell.ShellCommand(
-                name = 'svn.clean.externals',
-                command = ['svn', 'cleanup'],
-                haltOnFailure=False, flunkOnFailure=False,
-                description = ['svn clean externals'],
-                workdir='test-suite-externals'))
-        f.addStep(SVN(name='pull.test-suite-externals', mode='full',
-                      repourl=external_URL, retry = (60, 5), method='fresh',
-                      workdir='test-suite-externals', alwaysUseLatest=True,
-                      haltOnFailure=False, flunkOnFailure=False,
-                      description='pull.test-suite-externals',
-                      timeout=300))
-        # Buildbot uses got_revision instead of revision to identify builds.
-        # The previous step will set it incorrectly
-        # We set it to the correct value in th following step
-        setProperty(f, 'got_revision', WithProperties('%(revision)s'))
-    # Create the LNT virtual env.
-    f.addStep(buildbot.steps.shell.ShellCommand(
-            name='venv.lnt.clean', command=['rm', '-rfv', 'lnt.venv'],
-            haltOnFailure=True, description=['clean', 'LNT', 'venv'],
-            workdir=WithProperties('%(builddir)s')))
-    f.addStep(buildbot.steps.shell.ShellCommand(
-            name='venv.lnt.create', command=['virtualenv', 'lnt.venv'],
-            haltOnFailure=True, description=['create', 'LNT', 'venv'],
-            workdir=WithProperties('%(builddir)s'),
-            env={'PATH' : '${PATH}:/usr/local/bin'}))
-    f.addStep(buildbot.steps.shell.ShellCommand(
-            name='venv.lnt.install', haltOnFailure=True,
-            command=[WithProperties('%(builddir)s/lnt.venv/bin/pip'), 'install',
-                     '--index-url', package_url,
-                     '-e', '.'],
-            description=['install', 'LNT'], workdir='lnt'))
-    # Clean up the sandbox dir.
-    f.addStep(buildbot.steps.shell.ShellCommand(
-            name='lnt.nightly-test.clean', command=['rm', '-rfv', 'nt'],
-            haltOnFailure=True, description=['clean', 'LNT', 'sandbox'],
-            workdir='tests'))
-    # Run the nightly test.
-    nick = '%(slavename)s-%(buildername)s'
-    args.extend(['nt', '--sandbox', 'nt', '--cc',
-            WithProperties('%(builddir)s/%(cc_path)s'), '--cxx',
-            WithProperties('%(builddir)s/%(cxx_path)s'), '--without-llvm',
-            '--test-suite', WithProperties('%(builddir)s/test-suite'),
-            '--no-timestamp', '--no-machdep-info', '--no-auto-name',
-            '--no-configure'])
-    if external_URL:
-        args.extend(['--test-externals', external_dir])
-    if parallel:
-        args.extend(['-j', WithProperties(jobs)])
-    args.extend(nt_flags)
-    f.addStep(zorg.buildbot.commands.LitTestCommand.LitTestCommand(
-            name='lnt.nightly-test', command=args, haltOnFailure=True,
-            description=['nightly test'], workdir='tests',
-            logfiles={'configure.log' : 'nt/build/configure.log',
-                      'build-tools.log' : 'nt/build/build-tools.log',
-                      'test.log' : 'nt/build/test.log',
-                      'report.json' : 'nt/build/report.json'}))
-
-    return f

zorg/buildbot/builders/LibCXXBuilder.py

-
-import os
-
-import buildbot
-import buildbot.process.factory
-import buildbot.steps.shell
-import buildbot.steps.source as source
-import buildbot.steps.source.svn as svn
-import buildbot.process.properties as properties
-
-import zorg.buildbot.commands.LitTestCommand as lit_test_command
-import zorg.buildbot.util.artifacts as artifacts
-import zorg.buildbot.util.phasedbuilderutils as phased_builder_utils
-
-reload(lit_test_command)
-reload(artifacts)
-reload(phased_builder_utils)
-
-def getLibCXXBuilder(f=None, source_path=None,
-                     lit_dir=None):
-    if f is None:
-        f = buildbot.process.factory.BuildFactory()
-        # Find the build directory. We assume if f is passed in that the build
-        # directory has already been found.
-        f = phased_builder_utils.getBuildDir(f)
-    
-    # Grab the sources if we are not passed in any.
-    if source_path is None:
-        source_path = 'sources'
-        src_url = 'http://llvm.org/svn/llvm-project/libcxx/trunk'
-        f = phased_builder_utils.SVNCleanupStep(f, source_path)
-        f.addStep(svn.SVN(name='pull.src', mode='full', repourl=src_url,
-                          workdir=source_path, method='fresh',
-                          alwaysUseLatest=False, retry = (60, 5),
-                          description='pull.src'))
-    
-    # Grab the artifacts for our build.
-    f = artifacts.GetCompilerArtifacts(f)
-    host_compiler_dir = properties.WithProperties('%(builddir)s/host-compiler')
-    f = artifacts.GetCCFromCompilerArtifacts(f, host_compiler_dir)
-    f = artifacts.GetCXXFromCompilerArtifacts(f, host_compiler_dir)
-    
-    # Build libcxx.
-    CC = properties.WithProperties('%(cc_path)s')
-    CXX = properties.WithProperties('%(cxx_path)s')
-    HEADER_INCLUDE = \
-        properties.WithProperties('-I %s' % os.path.join('%(builddir)s',
-                                                         source_path,
-                                                         'include'))
-    SOURCE_LIB = \
-        properties.WithProperties(os.path.join('%(builddir)s',
-                                               source_path, 'lib',
-                                               'libc++.1.dylib'))
-    
-    f.addStep(buildbot.steps.shell.ShellCommand(
-              name='build.libcxx', command=['./buildit'], haltOnFailure=True, 
-              workdir=os.path.join(source_path, 'lib'),
-              env={ 'CC' : CC, 'CXX' : CXX, 'TRIPLE' : '-apple-'}))
-
-    # Get the 'lit' sources if we need to.
-    if lit_dir is None:
-        lit_dir = 'lit.src'
-        f.addStep(svn.SVN(
-            name='pull.lit', mode='incremental', method='fresh',
-            repourl='http://llvm.org/svn/llvm-project/llvm/trunk/utils/lit',
-            workdir=lit_dir, alwaysUseLatest=False))
-
-    # Install a copy of 'lit' in a virtualenv.
-    f.addStep(buildbot.steps.shell.ShellCommand(
-            name='venv.lit.clean',
-            command=['rm', '-rf', 'lit.venv'],
-            workdir='.', haltOnFailure=True))
-    f.addStep(buildbot.steps.shell.ShellCommand(
-            name='venv.lit.make',
-            command=['/usr/local/bin/virtualenv', 'lit.venv'],
-            workdir='.', haltOnFailure=True))
-    f.addStep(buildbot.steps.shell.ShellCommand(
-            name='venv.lit.install',
-            command=[
-                properties.WithProperties('%(builddir)s/lit.venv/bin/python'),
-                'setup.py', 'install'],
-            workdir=lit_dir, haltOnFailure=True))
-
-    # Run the tests with the system's dylib
-    f.addStep(lit_test_command.LitTestCommand(
-            name='test.libcxx.system',
-            command=[
-                properties.WithProperties('%(builddir)s/lit.venv/bin/lit'),
-                '-v', '--show-xfail', '--show-unsupported',
-                properties.WithProperties(
-                    '--param=cxx_under_test=%(cxx_path)s'),
-                '--param=use_system_lib=true',
-                'sources/test'],
-            workdir='.'))
-    # Run the tests with the newly built dylib
-    f.addStep(lit_test_command.LitTestCommand(
-            name='test.libcxx.new',
-            command=[
-                properties.WithProperties('%(builddir)s/lit.venv/bin/lit'),
-                '-v', '--show-xfail', '--show-unsupported',
-                properties.WithProperties(
-                    '--param=cxx_under_test=%(cxx_path)s'),
-                '--param=use_system_lib=false',
-                'sources/test'],
-            workdir='.'))
-
-    return f

zorg/buildbot/builders/LibcxxAndAbiBuilder.py

 import buildbot.steps.shell
 import buildbot.process.properties as properties
 
-import zorg.buildbot.commands.LitTestCommand as lit_test_command
-import zorg.buildbot.util.artifacts as artifacts
-import zorg.buildbot.util.phasedbuilderutils as phased_builder_utils
-
 from zorg.buildbot.commands.LitTestCommand import LitTestCommand
 from zorg.buildbot.commands.CmakeCommand import CmakeCommand
 from zorg.buildbot.process.factory import LLVMBuildFactory
 from zorg.buildbot.builders import UnifiedTreeBuilder
 
-reload(lit_test_command)
-reload(artifacts)
-reload(phased_builder_utils)
-
 def getLibcxxAndAbiBuilder(f=None, env=None,
                            cmake_extra_opts=None, lit_extra_opts=None,
                            lit_extra_args=None, check_libcxx_abilist=False,
                            check_libcxx_benchmarks=None,
                            depends_on_projects=None,
                            use_cache=None,
                            **kwargs):
 

zorg/buildbot/builders/NightlytestBuilder.py

-from buildbot.steps.shell import Configure, ShellCommand
-from buildbot.process.properties import WithProperties
-from buildbot.steps.source import SVN
-
-from zorg.buildbot.commands.NightlyTestCommand import NightlyTestCommand
-
-import LLVMGCCBuilder
-import ClangBuilder
-
-def getNightlytestBuildFactory(submitAux=None, *args, **kwargs):
-  f = LLVMGCCBuilder.getLLVMGCCBuildFactory(*args, **kwargs)
-
-  # Prepare environmental variables. Set here all env we want everywhere.
-  merged_env = {
-                   'TERM' : 'dumb'     # Make sure Clang doesn't use color escape sequences.
-               }
-
-  env = kwargs.pop('env', None)
-  if env is not None:
-      merged_env.update(env)  # Overwrite pre-set items with the given ones, so user can set anything.
-
-  # Copy NT script.
-  f.addStep(ShellCommand(name="cp test script", 
-                         command=["cp", 
-                                  WithProperties("%(builddir)s/llvm.src/utils/NewNightlyTest.pl"),
-                                  "."],
-                         haltOnFailure = True,
-                         description   = "cp test script",
-                         workdir       = "llvm.nt",
-                         env           = merged_env))
-
-  submitCommand = []
-  if submitAux is not None:
-      submitCommand = ['-submit-aux',
-                       submitAux]
-
-  f.addStep(ShellCommand(name="nightlytest", 
-                         command=["./NewNightlyTest.pl",
-                                  "-parallel-jobs", WithProperties("%(jobs)s"), 
-                                  "-parallel",
-                                  "-noremoveatend",
-                                  "-noremoveresults",
-                                  "-release",
-                                  "-enable-llcbeta",
-                                  "-verbose",
-                                  "-nickname", WithProperties("%(slavename)s"),
-                                  "-test-cxxflags", "-I/usr/include/c++/4.2.1/i686-apple-darwin10 -I/usr/include/c++/4.2.1",
-                                  "-nosubmit",
-                                  "-teelogs"] + submitCommand,
-                         haltOnFailure = True,
-                         description   = "nightlytest",
-                         workdir       = "llvm.nt",
-                         env           = {
-                                          'LLVMGCCDIR' : WithProperties("%(builddir)s/llvm-gcc.install"),
-                                          'BUILDDIR'   : WithProperties("%(builddir)s/llvm.nt/build"),
-                                          'WEBDIR'     : WithProperties("%(builddir)s/llvm.nt/testresults"),
-                                         }.update(merged_env)))
-  return f
-
-def getFastNightlyTestBuildFactory(triple, xfails=[], clean=True, test=False, make='make', **kwargs):
-  # Build compiler to test.  
-  f = ClangBuilder.getClangBuildFactory(
-    triple, clean=clean, test=test,
-    make=make, **kwargs)
-
-  # Prepare environmental variables. Set here all env we want everywhere.
-  merged_env = {
-                   'TERM' : 'dumb'     # Make sure Clang doesn't use color escape sequences.
-               }
-  env = kwargs.pop('env', None)
-  if env is not None:
-      merged_env.update(env)  # Overwrite pre-set items with the given ones, so user can set anything.
-
-  # Get the test-suite sources.
-  f.addStep(SVN(name          = 'svn-test-suite',
-                mode          = 'update',
-                baseURL       = 'http://llvm.org/svn/llvm-project/test-suite/',
-                defaultBranch = 'trunk',
-                workdir       = 'test-suite.src'))
-
-  # Clean up.
-  if clean:
-      f.addStep(ShellCommand(name="rm.test-suite",
-                             command=["rm", "-rf", "test-suite.obj"],
-                             haltOnFailure = True,
-                             description   = "rm test-suite build dir",
-                             workdir       = ".",
-                             env           = merged_env))
-
-  # Configure.
-  f.addStep(Configure(name="configure.test-suite",
-                      command=['../test-suite.src/configure',
-                               WithProperties("--with-llvmsrc=%(builddir)s/llvm.src"),
-                               WithProperties("--with-llvmobj=%(builddir)s/llvm.obj"),
-                               WithProperties("--with-built-clang")],
-                      haltOnFailure   = True,
-                      description     = ["configuring", "test-suite"],
-                      descriptionDone = ["configure",   "test-suite"],
-                      workdir         = 'test-suite.obj',
-                      env             = merged_env))
-
-  # Build and test.
-  f.addStep(ShellCommand(name="rm.test-suite.report",
-                         command=["rm", "-rf",
-                                  "test-suite.obj/report.nightly.raw.out",
-                                  "test-suite.obj/report.nightly.txt"],
-                         haltOnFailure = True,
-                         description   = "rm test-suite report",
-                         workdir       = ".",
-                         env           = merged_env))
-  f.addStep(NightlyTestCommand(name="make.test-suite",
-                               command=[make, WithProperties("-j%(jobs)s"),
-                                        "ENABLE_PARALLEL_REPORT=1",
-                                        "DISABLE_CBE=1", "DISABLE_JIT=1",
-                                        "TEST=nightly", "report"],
-                               haltOnFailure   = True,
-                               logfiles        = { 'report' : 'report.nightly.txt' },
-                               xfails          = xfails,
-                               description     = ["running", "test-suite"],
-                               descriptionDone = ["run",     "test-suite"],
-                               workdir         = 'test-suite.obj',
-                               env             = merged_env))
-
-  return f

zorg/buildbot/builders/SanitizerBuilderII.py

-from buildbot.process.factory import BuildFactory
-from buildbot.process.properties import WithProperties 
-from buildbot.steps.source import SVN
-from buildbot.steps.shell import ShellCommand, SetProperty
-from buildbot.steps.shell import WarningCountingShellCommand 
-from buildbot.steps.slave import RemoveDirectory
-from zorg.buildbot.commands.AnnotatedCommand import AnnotatedCommand
-from zorg.buildbot.commands.NinjaCommand import NinjaCommand
-from zorg.buildbot.conditions.FileConditions import FileDoesNotExist
-
-def getSanitizerBuildFactoryII(
-           clean=False,
-           sanity_check=True,
-           sanitizers=['sanitizer','asan','lsan','msan','tsan','ubsan','dfsan'],
-           common_cmake_options=None, # FIXME: For backward compatibility. Will be removed.
-           extra_configure_args=[],
-           prefixCommand=["nice", "-n", "10"], # For backward compatibility.
-           env=None,
-           jobs="%(jobs)s",
-           timeout=1200):
-
-    llvm_srcdir   = "llvm.src"
-    llvm_objdir   = "llvm.obj"
-
-    # Prepare environmental variables. Set here all env we want everywhere.
-    merged_env = {
-        'TERM' : 'dumb', # Make sure Clang doesn't use color escape sequences.
-                 }
-    if env:
-        # Overwrite pre-set items with the given ones, so user can set anything.
-        merged_env.update(env)
-
-    f = BuildFactory()
-
-    # Clean directory, if requested.
-    cleanBuildRequested = lambda step: step.build.getProperty("clean") or clean
-    f.addStep(RemoveDirectory(name='clean '+llvm_objdir,
-                dir=llvm_objdir,
-                haltOnFailure=False,
-                flunkOnFailure=False,
-                doStepIf=cleanBuildRequested
-                ))
-
-    # Get llvm, clang, ompiler-rt, libcxx, libcxxabi, libunwind
-    f.addStep(SVN(name='svn-llvm',
-                  mode='update',
-                  description='svn-llvm',
-                  descriptionDone='svn-llvm',
-                  baseURL='http://llvm.org/svn/llvm-project/llvm/',
-                  defaultBranch='trunk',
-                  workdir=llvm_srcdir))
-
-    f.addStep(SVN(name='svn-clang',
-                  mode='update',
-                  description='svn-clang',
-                  descriptionDone='svn-clang',
-                  baseURL='http://llvm.org/svn/llvm-project/cfe/',
-                  defaultBranch='trunk',
-                  workdir='%s/tools/clang' % llvm_srcdir))
-
-    f.addStep(SVN(name='svn-compiler-rt',
-                  mode='update',
-                  description='svn-compiler-rt',
-                  descriptionDone='svn--compiler-rt',
-                  baseURL='http://llvm.org/svn/llvm-project/compiler-rt/',
-                  defaultBranch='trunk',
-                  workdir='%s/projects/compiler-rt' % llvm_srcdir))
-
-    f.addStep(SVN(name='svn-libcxx',
-                  mode='update',
-                  description='svn-libcxx',
-                  descriptionDone='svn-libcxx',
-                  baseURL='http://llvm.org/svn/llvm-project/libcxx/',
-                  defaultBranch='trunk',
-                  workdir='%s/projects/libcxx' % llvm_srcdir))
-
-    f.addStep(SVN(name='svn-libcxxabi',
-                  mode='update',
-                  description='svn-libcxxabi',
-                  descriptionDone='svn-libcxxabi',
-                  baseURL='http://llvm.org/svn/llvm-project/libcxxabi/',
-                  defaultBranch='trunk',
-                  workdir='%s/projects/libcxxabi' % llvm_srcdir))
-
-    f.addStep(SVN(name='svn-libunwind',
-                  mode='update',
-                  description='svn-libunwind',
-                  descriptionDone='svn-libunwind',
-                  baseURL='http://llvm.org/svn/llvm-project/libunwind/',
-                  defaultBranch='trunk',
-                  workdir='%s/projects/libunwind' % llvm_srcdir))
-
-    # Run annotated command for sanitizer.
-    if sanity_check:
-        f.addStep(
-            AnnotatedCommand(
-                name="lint check",
-                description="lint check",
-                timeout=timeout,
-                haltOnFailure=False, #True,
-                warnOnWarnings=True,
-                command=["./check_lint.sh"],
-                workdir="%s/projects/compiler-rt/lib/sanitizer_common/scripts" % llvm_srcdir,
-                env=merged_env))
-
-    # Always build with ninja.
-    cmakeCommand = ["cmake", "-G", "Ninja"]
-
-    # Reconsile configure args with the defaults we want.
-    if not any(a.startswith('-DCMAKE_BUILD_TYPE=')   for a in extra_configure_args):
-       cmakeCommand.append('-DCMAKE_BUILD_TYPE=Release')
-    if not any(a.startswith('-DLLVM_ENABLE_WERROR=') for a in extra_configure_args):
-       cmakeCommand.append('-DLLVM_ENABLE_WERROR=OFF')
-    if not any(a.startswith('-DLLVM_ENABLE_ASSERTIONS=') for a in extra_configure_args):
-       cmakeCommand.append('-DLLVM_ENABLE_ASSERTIONS=ON')
-    if not any(a.startswith('-DCMAKE_C_COMPILER') for a in extra_configure_args):
-       cmakeCommand.append('-DCMAKE_C_COMPILER=clang')
-    if not any(a.startswith('-DCMAKE_CXX_COMPILER') for a in extra_configure_args):
-       cmakeCommand.append('-DCMAKE_CXX_COMPILER=clang++')
-    if not any(a.startswith('-DCMAKE_CXX_FLAGS') for a in extra_configure_args):
-       cmakeCommand.append('-DCMAKE_CXX_FLAGS=\"-std=c++11 -stdlib=libc++\"')
-    if not any(a.startswith('-DCMAKE_EXE_LINKER_FLAGS') for a in extra_configure_args):
-       cmakeCommand.append('-DCMAKE_EXE_LINKER_FLAGS=-lcxxrt')
-    if not any(a.startswith('-DLIBCXXABI_USE_LLVM_UNWINDER=') for a in extra_configure_args):
-       cmakeCommand.append('-DLIBCXXABI_USE_LLVM_UNWINDER=ON')
-    if not any(a.startswith('-DLLVM_LIT_ARGS=') for a in extra_configure_args):
-       cmakeCommand.append('-DLLVM_LIT_ARGS=\"-v\"')
-
-    cmakeCommand += extra_configure_args + ["../%s" % llvm_srcdir]
-
-    # Note: ShellCommand does not pass the params with special symbols right.
-    # The " ".join is a workaround for this bug.
-    f.addStep(ShellCommand(
-        name="cmake-configure",
-        description=["cmake configure"],
-        haltOnFailure=False, #True,
-        warnOnWarnings=True,
-        command=WithProperties(" ".join(cmakeCommand)),
-        env=merged_env,
-        workdir=llvm_objdir,
-        doStepIf=FileDoesNotExist("./%s/CMakeCache.txt" % llvm_objdir)))
-
-    # Build everything.
-    f.addStep(NinjaCommand(name='build',
-                           haltOnFailure=False, #True,
-                           warnOnWarnings=True,
-                           description=['building', 'with', 'ninja'],
-                           descriptionDone=['built', 'with', 'ninja'],
-                           workdir=llvm_objdir,
-                           env=merged_env))
-
-    # Run tests for each of the requested sanitizers.
-    if sanitizers:
-        for s in sanitizers:
-            f.addStep(
-                NinjaCommand(name='test %s' % s,
-                             targets=['check-%s' % s],
-                             haltOnFailure=False, #True,
-                             description=['testing', '%s' % s],
-                             descriptionDone=['test', '%s' % s],
-                             workdir=llvm_objdir,
-                             env=merged_env))
-
-    return f

zorg/buildbot/builders/Util.py

if vs.lower() == "autodetect":
         Otherwise, use the vs argument to construct a path to the expected location of vcvarsall.bat
 
         This code is following the guidelines from strategy 1 in this blog post:
         https://blogs.msdn.microsoft.com/vcblog/2017/03/06/finding-the-visual-c-compiler-tools-in-visual-studio-2017/
 
         It doesn't work when VS is not installed at the default location.
         """
 
-        # TODO: Implement autodetect for VS versions other than 2017
+        # LLVM code base requires VS 2017 or later.
+        # This means vswhere.exe later than 1.0.40, so we add `-products *` to include Build Tools in the search
+        # to support build-tools only installations.
         vcvars_command = "for /f \"tokens=* USEBACKQ\" %%F in " \
-                            "(`\"%ProgramFiles(x86)%\\Microsoft Visual Studio\\Installer\\vswhere.exe\" -latest -property installationPath`) DO " \
+                            "(`\"%ProgramFiles(x86)%\\Microsoft Visual Studio\\Installer\\vswhere.exe\" -products * -latest -property installationPath`) DO " \
                             "\"%%F\"\\VC\\Auxiliary\\Build\\vcvarsall.bat"
     else:
-        # Older versions of VS
+        # Note: Support for older versions of VS is deprecated and will be removed.
         vcvars_command = "\"" + "\\".join((vs, '..','..','VC', 'vcvarsall.bat')) + "\""
 
     vcvars_command = "%s %s && set" % (vcvars_command, arch_arg)
     return vcvars_command
 
 def extractSlaveEnvironment(exit_status, stdout, stderr):
     '''Helper function for SetPropertyCommand. Loads Slave Environment
     into a dictionary, and returns slave_env property for ShellCommands.'''

zorg/buildbot/process/factory.py

 from collections import OrderedDict
 
 from buildbot.process.factory import BuildFactory
-from buildbot.steps.source import SVN, Git
+from buildbot.steps.source import Git
 from buildbot.steps.shell import WithProperties
 
-# NOTE: svn_repos is deprecated and will be removed.
-svn_repos = OrderedDict([
-  ('llvm'             , ("%(llvm_srcdir)s",                         '%(vcs_protocol:-http)s://llvm.org/svn/llvm-project/llvm/')),
-  ('clang'            , ("%(llvm_srcdir)s/tools/clang",             '%(vcs_protocol:-http)s://llvm.org/svn/llvm-project/cfe/')),
-  ('clang-tools-extra', ("%(llvm_srcdir)s/tools/clang/tools/extra", '%(vcs_protocol:-http)s://llvm.org/svn/llvm-project/clang-tools-extra/')),
-  ('compiler-rt'      , ("%(llvm_srcdir)s/projects/compiler-rt",    '%(vcs_protocol:-http)s://llvm.org/svn/llvm-project/compiler-rt/')),
-  ('libcxx'           , ("%(llvm_srcdir)s/projects/libcxx",         '%(vcs_protocol:-http)s://llvm.org/svn/llvm-project/libcxx/')),
-  ('libcxxabi'        , ("%(llvm_srcdir)s/projects/libcxxabi",      '%(vcs_protocol:-http)s://llvm.org/svn/llvm-project/libcxxabi/')),
-  ('libunwind'        , ("%(llvm_srcdir)s/projects/libunwind",      '%(vcs_protocol:-http)s://llvm.org/svn/llvm-project/libunwind/')),
-  ('lld'              , ("%(llvm_srcdir)s/tools/lld",               '%(vcs_protocol:-http)s://llvm.org/svn/llvm-project/lld/')),
-  ('lnt'              , ("test/lnt",                                '%(vcs_protocol:-http)s://llvm.org/svn/llvm-project/lnt/')),
-  ('test-suite'       , ("test/test-suite",                         '%(vcs_protocol:-http)s://llvm.org/svn/llvm-project/test-suite/')),
-  ('lldb'             , ("%(llvm_srcdir)s/tools/lldb",              '%(vcs_protocol:-http)s://llvm.org/svn/llvm-project/lldb/')),
-  ('llgo'             , ("%(llvm_srcdir)s/tools/llgo",              '%(vcs_protocol:-http)s://llvm.org/svn/llvm-project/llgo/')),
-  ('polly'            , ("%(llvm_srcdir)s/tools/polly",             '%(vcs_protocol:-http)s://llvm.org/svn/llvm-project/polly/')),
-  ('openmp'           , ("%(llvm_srcdir)s/tools/openmp",            '%(vcs_protocol:-http)s://llvm.org/svn/llvm-project/openmp/')),
-  ('zorg'             , ("zorg",                                    '%(vcs_protocol:-http)s://llvm.org/svn/llvm-project/zorg/')),
-  ])
-
 class LLVMBuildFactory(BuildFactory):
     """
     TODO: Document
     """
 
     def __init__(self, steps=None, depends_on_projects=None, **kwargs):
         # Cannot use "super" here as BuildFactory is an old style class.
         BuildFactory.__init__(self, steps)
 
         if depends_on_projects is None:
             self.depends_on_projects = frozenset(['llvm'])
         else:
             self.depends_on_projects = frozenset(depends_on_projects)
 
-        # By default LLVMBuildFactory works in the legacy mode.
+        # FIXME: legacy mode is deprecated and will be removed.
         self.is_legacy_mode = kwargs.pop('is_legacy_mode', False)
 
         # Directories.
         self.llvm_srcdir = kwargs.pop('llvm_srcdir', None)
         self.obj_dir = kwargs.pop('obj_dir', None)
         self.install_dir = kwargs.pop('install_dir', None)
 
         # Preserve the rest of the given extra attributes if any, so we could
         # expand the factory later.
         for k,v in kwargs.items():
             setattr(self, k, v)
 
-        if self.is_legacy_mode:
-            self.llvm_srcdir = self.llvm_srcdir or "llvm"
-            self.obj_dir = self.obj_dir or "build"
-        else:
-            self.monorepo_dir = self.llvm_srcdir or "llvm-project"
-            self.llvm_srcdir = \
                 "%(monorepo_dir)s/llvm" % {'monorepo_dir' : self.monorepo_dir}
-            self.obj_dir = \
                 self.obj_dir or "build"
 
-            # Repourl_prefix could be specified per builder. Otherwise we use github.
-            self.repourl_prefix = kwargs.pop('repourl_prefix', 'https://github.com/llvm/')
 
 
     @staticmethod
     def pathRelativeToBuild(path, buildPath):
         if path.startswith('/'):
             # The path is absolute. Don't touch it.
             return path
         else:
def pathRelativeToBuild(path, buildPath):
                     break
             else:
                 # Everything matches.
                 rel_path = '.'
 
             return rel_path
 
 
-    # llvm_srcdir - Path to the root of the unified source tree.
-    # mode - SVN checkout mode.
-    # defaultBranch - the default branch to checkout.
-    # and so on, see the list of the SVN params. 
-    # NOTE: addSVNSteps is deprecated and will be removed. Please use addGetSourcecodeSteps instead.
-    def addSVNSteps(self, llvm_srcdir=None, **kwargs):
-        if llvm_srcdir is None:
-            llvm_srcdir = self.llvm_srcdir
-        if not kwargs.get('mode', None):
-            kwargs['mode'] = 'update'
-        if not kwargs.get('defaultBranch', None):
-            kwargs['defaultBranch'] = 'trunk'
-
-        # Add a SVM step for each project this builder depends on.
-        # We want the projects be always checked out in a certain order.
-        for project in svn_repos.keys():
-            if project in self.depends_on_projects:
-                workdir, baseURL = svn_repos[project]
-                self.addStep(
-                    SVN(name='svn-%s' % project,
-                        workdir=workdir % {'llvm_srcdir' : llvm_srcdir},
-                        baseURL=WithProperties(baseURL),
-                        **kwargs))
-
-
     def addGetSourcecodeSteps(self, **kwargs):
         # Remove 'is_legacy_mode' if it leaked in to kwargs.
         kwargs.pop('is_legacy_mode', None)
 
-        # Bail out if we are in the legacy mode and SVN checkout is required.
-        if self.is_legacy_mode:
-            self.addSVNSteps(**kwargs)
-            return
-
         # Checkout the monorepo.
         self.addStep(
             Git(name='Checkout the source code',
                 repourl=self.repourl_prefix + "llvm-project.git",
                 progress=True,
                 workdir=WithProperties(self.monorepo_dir),
                 **kwargs))
 
 
     # Checkout a given LLVM project to the given directory.
     # TODO: Handle clean property and self.clean attribute.
     def addGetSourcecodeForProject(self, project, name=None, src_dir=None, **kwargs):
         # Remove 'is_legacy_mode' if it leaked in to kwargs.
         kwargs.pop('is_legacy_mode', None)
 
-        # Bail out if we are in the legacy mode and SVN checkout is required.
-        if self.is_legacy_mode:
-            workdir, baseURL = svn_repos[project]
-
-            if not name:
-                name = 'svn-%s' % project
-
-            # Check out to the given directory if any.
-            # Otherwise this is a part of the unified source tree.
-            if src_dir is None:
-                src_dir = workdir % {'llvm_srcdir' : self.llvm_srcdir}
-
-            if not kwargs.get('mode', None):
-                kwargs['mode'] = 'update'
-            if not kwargs.get('defaultBranch', None):
-                kwargs['defaultBranch'] = 'trunk'
-
-            self.addStep(
-                SVN(name=name,
-                    workdir=src_dir,
-                    baseURL=WithProperties(baseURL),
-                    **kwargs))
-        else:
-            # project contains a repo name which is not a part of the monorepo.
-            #  We do not enforce it here, though.
-            _repourl = kwargs.pop('repourl', None)
-            if not _repourl:
-                _repourl = self.repourl_prefix + "llvm-%s.git" % project
 
-            if not name:
-                name = 'Checkout %s' % project
 
-            # Check out to the given directory if any.
-            # Otherwise this is a part of the unified source tree.
-            if src_dir is None:
-                src_dir = 'llvm-%s' % project
 
-            # Ignore workdir if given. We check out to src_dir.
-            kwargs.pop('workdir', None)
 
-            self.addStep(
-                Git(name=name,
-                    repourl=_repourl,
-                    progress=True,
-                    workdir=WithProperties(src_dir),
-                    **kwargs))