This is an archive of the discontinued LLVM Phabricator instance.

[buildbot] Moved MLIR buildbot config under buildbot/google/mlir
AcceptedPublic

Authored by tra on Jul 21 2020, 11:24 AM.

Details

Summary

buildbot/google itself will also hold configs for other bots owned by Google.

Event Timeline

tra created this revision.Jul 21 2020, 11:24 AM

Sorry I did not understand why do you want to move these files.
Why not keep the new docker files for the existing machines right next to the existing ones?

So my thought on the directory structure was:

  • /buildbot/google/
    • docker/
      • mlir-nvidia/
      • win32/
      • your-docker-dir/
      • some-other-bot/
      • <common scripting>
    • terraform/
      • <one terraform config for all machines>
tra added a comment.Jul 22 2020, 10:58 AM

Sorry I did not understand why do you want to move these files.
Why not keep the new docker files for the existing machines right next to the existing ones?

So my thought on the directory structure was:

  • /buildbot/google/
    • docker/
      • mlir-nvidia/
      • win32/
      • your-docker-dir/
      • some-other-bot/
      • <common scripting>

I'm fine with this part.

  • terraform/
    • <one terraform config for all machines>

That's the part I'm not quite sure about. Terraform is an all-or-nothing kind of thing. I.e. terraform apply will try to set thing up for *everything* described in main.tf. I'm not convinced that we should mix all our bot setups here. I don't want MLIR folks and myself stepping on each other's toes and affecting each others' cluster setups. We may eventually consolidate them, but I don't think we're there yet.

Hence, I have my own terraform setup. We could structure it similarly to your suggestion for the docker files -- move current files under terraform/mlir and mine under terraform/cudabot, but with mlir and cudabot setups being split everywhere, we may as well maintain mlir and cudabot separately.

What is strange to me right now is the asymmetry:

google/docker/...
google/mlir/docker/...

google/terraform/
google/mlir/terraform/

tra added a comment.Jul 22 2020, 1:27 PM

What is strange to me right now is the asymmetry:

google/docker/...
google/mlir/docker/...

google/terraform/
google/mlir/terraform/

I'm not sure what exactly you have in mind. Could you elaborate?
The stuff that's currently in google/ will move under google/mlir and there will be a new set of configs under google/cuda in the patch up the stack.

In D84256#2167804, @tra wrote:

I'm not sure what exactly you have in mind. Could you elaborate?
The stuff that's currently in google/ will move under google/mlir and there will be a new set of configs under google/cuda in the patch up the stack.

I don't have this context so I was just seeing this revision alone which setup google/mlir/docker/ and still has google/docker.
It makes sense if after your stack of revisions if there is no google/docker left.

tra added a comment.Jul 22 2020, 3:40 PM

I don't have this context so I was just seeing this revision alone which setup google/mlir/docker/ and still has google/docker.
It makes sense if after your stack of revisions if there is no google/docker left.

File list is indeed somewhat hard to parse. After all the changes directory structure looks like this:

$ find buildbot/google -type d
buildbot/google
buildbot/google/cuda
buildbot/google/cuda/terraform
buildbot/google/cuda/docker
buildbot/google/cuda/docker/cudabot
buildbot/google/mlir
buildbot/google/mlir/terraform
buildbot/google/mlir/docker
buildbot/google/mlir/docker/buildbot-mlir-nvidia

Well, then let's go with your proposal of merging the docker directory and separate the terraform directory.
At least for now. If we later figure out a way to work in the same terraform config, we can always invest some time to merge that later on.

tra updated this revision to Diff 280535.Jul 24 2020, 11:53 AM
tra retitled this revision from [buildbot] Moved MLIR buildbot config under buildbot/google/mlir to [buildbot] Moved MLIR buildbot config under buildbot/google/mlir.

Updated directory structure.

tra updated this revision to Diff 280536.Jul 24 2020, 11:54 AM

Really updated directory structure.

tra added a comment.Jul 24 2020, 12:02 PM

Phabricator seems to be confused by the renames. The change just moved terraform -> terraform/buildbot-mlir-nvidia
The directory structure looks like this now:

buildbot/google/terraform
buildbot/google/terraform/buildbot-mlir-nvidia
buildbot/google/docker
buildbot/google/docker/windows-base-vscode2019
buildbot/google/docker/buildbot-windows10-vs2019
buildbot/google/docker/buildbot-mlir-nvidia
kuhnel accepted this revision.Jul 27 2020, 7:24 AM
This revision is now accepted and ready to land.Jul 27 2020, 7:24 AM
buildbot/google/mlir/terraform/deployment-mlir-nvidia.yaml