[buildbot] Added builders and slaves for the new CUDA build/test bots.

[tra]

Changed bot/slave structure

[tra]

Fixed notification list.

[gkistanova]

Hello Artem.

I have commented the getCUDAAnnotatedBuildFactory, but it doesn't seem you need a special build factory for your builders.
Just use the existing AnnotatedBuilder.getAnnotatedBuildFactory in the builders.py for your build configurations.

zorg/buildbot/builders/CUDATestsuiteBuilder.py

224

You don't need to handle is_legacy_mode, so, better skip it here and let the default machinery to take place.

227

is_legacy_mode used twice.

254

bot_dir is not defined.

257

You can skip it and let the LLVMBuildFactory handle it by default.

258

Should it be clean=clean here instead?

263

You do not use the builddir property. Even if you would need an absolute path into your build directory, you do not need a build step. There is a workdir property, just use that instead. For example, something like

WithProperties("%(workdir)s/" + f.obj_dir + "/bin/clang++")

would give you a fully qualified path to the installed clang++.

[tra]

Addressed review comments.

[tra]

Use AnnotatedBuilder with an external build script.

[tra]

In D81118#2078854, @gkistanova wrote:

Hello Artem.

I have commented the getCUDAAnnotatedBuildFactory, but it doesn't seem you need a special build factory for your builders.
Just use the existing AnnotatedBuilder.getAnnotatedBuildFactory in the builders.py for your build configurations.

Thank you for pointing this out. I've missed the fact that AnnotatedBuildFactory already includes LLVMBuildFactory.
I've changed the patch to use AnnotatedBuildFactory + a script to launch external command.

[gkistanova]

Thanks for updating the patch, Artem.

Could you elaborate why you need the script launcher (zorg/buildbot/builders/annotated/external.py), please? You can use your cuda-related scripts directly with the annotated builder without having an extra layer.

[tra]

In D81118#2081147, @gkistanova wrote:

Thanks for updating the patch, Artem.

Could you elaborate why you need the script launcher (zorg/buildbot/builders/annotated/external.py), please? You can use your cuda-related scripts directly with the annotated builder without having an extra layer.

The main reason at the moment is that I want to decouple LLVM buildbot from the details of the slave implementation. It will give me the ability to tinker with what the slaves are doing without having to make the round-trip to the zorg repo. I.e. I'll land this patch and stop bothering you with reviews of every little WIP change until I have something working.

Second reason is that the details of the slave implementation may be of no use to anyone else. I'm likely to use some things that are google-specific. If the final version of the script ends up being fairly generic, I'll send a follow-up patch to add it to zorg repo and switch the bots to use it.

[tra]

ping! Will the current version of the patch do?

[gkistanova]

I can temporarily set you up like this in the staging. You would do all the experiments you are after, tinker, and such. Once you are happy you could prepare a final patch for the review.
For how long do you expect tinkering?

[tra]

In D81118#2090901, @gkistanova wrote:

I can temporarily set you up like this in the staging. You would do all the experiments you are after, tinker, and such. Once you are happy you could prepare a final patch for the review.
For how long do you expect tinkering?

There are two goals -- get CUDA testing back online ASAP, and improve bold/test across all the bots we have (i.e. deduplicate builds, optimize cloud use costs, etc.). Former I can do relatively quickly (weeks?), the latter will likely take much longer.

My rough plan is :

• set up these annotated build slaves in staging
• set up a bot version to roughly match basic CUDA functionality testing that used to be done by the old CUDA bot
• replace old/dead CUDA bot with the replica of the new one, but also keep the copy of the bots in staging, so I can continue tinkering with them.

Does it make sense?

[gkistanova]

Staging is ready for your experiments.

[tra]

In D81118#2092283, @gkistanova wrote:

Staging is ready for your experiments.

Thank you.

It appears I 'll need to land external.py in zorg -- AnnotatedBuild factory always checks out a fresh copy of llvm-zorg.
I've attempted to manually copy the script where the bot is looking for it, but it's a rather fragile arrengement as the buildmaster apparently ends up cleaning the repo directories now and then and removes it.

Are you OK with me landing the external.py alone for now?

[gkistanova]

You could put that script somewhere local and give me the fully qualified path to that script.I'll set the staging accordingly.

Thanks

Galina

[tra]

In D81118#2112774, @gkistanova wrote:

You could put that script somewhere local and give me the fully qualified path to that script.I'll set the staging accordingly.

The local copy lives in /buildbot/external.py

[tra]

Here's an example of the failure to find external.py: http://lab.llvm.org:8014/builders/clang-cuda-gce-build/builds/303/steps/annotate/logs/stdio

[gkistanova]

I have changed the staging to invoke /buildbot/external.py.
http://lab.llvm.org:8014/builders/clang-cuda-gce-build/builds/313/steps/annotate/logs/stdio

[tra]

In D81118#2113147, @gkistanova wrote:

I have changed the staging to invoke /buildbot/external.py.
http://lab.llvm.org:8014/builders/clang-cuda-gce-build/builds/313/steps/annotate/logs/stdio

Now the script runs before LLVM source tree has been checked out, so there's no sources for me to work with. AFAICT, revision hash is not available in the environment, so there's no way for me to check out the correct revision.

As of this patch in general, I think the best thing to do here is to teach AnnotatedBuilder to accept external script to launch if they are specified with an absolute path. This way we'll not need external.py at all and the builders could be just set up with getAnnotatedBuildFactory(script="/buildbot/cuda-build"),.

[tra]

In D81118#2115258, @tra wrote:

AFAICT, revision hash is not available in the environment, so there's no way for me to check out the correct revision.

BUILDBOT_REVISION= appears to be set for some builds, but not others.
E.g. http://lab.llvm.org:8014/builders/clang-cuda-gce-build/builds/313/steps/annotate/logs/stdio does not have it
but http://lab.llvm.org:8014/builders/clang-cuda-gce-build/builds/376/steps/annotate/logs/stdio does.

I guess I can live with that, but it would be great to let the external script run after the source code has been checked out.

[gkistanova]

Now the script runs before LLVM source tree has been checked out, so there's no sources for me to work with. AFAICT, revision hash is not available in the environment, so there's no way for me to check out the correct revision.

That was my error. Sorry. Now the source code should be checked out for you by the bot.

[tra]

Updated CUDA bot setup.

[tra]

Removed external.py

[tra]

@gkistanova : I think the bots are in a reasonable shape now and are ready to move to the normal build master.
This patch simplifies things a bit based on the changes I've added in D83503 which lets annotated builder use external scripts (sounds like that's what you may have done on the staging bot already) and added an option to control whether the source repo is checked out.

[gkistanova]

Hello Artem,

Good. Now, since you are done with the experiments, please make sure all your scripts are in llvm-zorg, and update this patch accordingly.

[tra]

In D81118#2149139, @gkistanova wrote:

Hello Artem,

Good. Now, since you are done with the experiments, please make sure all your scripts are in llvm-zorg, and update this patch accordingly.

Done. Please see the patches in the stack.

[gkistanova]

LGTM

[tra]

Updated status.py with the new builder names.

buildbot/osuosl/master/config/builders.py

return [
                      gpu_devices=["GPU-af66efa4",  # K40c(sm_35),
                                   "GPU-44fe2444"   # GTX1070(sm_61)
                      ],
                      cuda_jobs = 4, # Speeds up test execution time by ~2x.
                      extra_ts_cmake_args=[],
                      enable_thrust_tests=False,
          ),
          'category' : 'clang'},
-
+        {'name': "clang-cuda-gce-build",
+         'slavenames' :["cuda-gce-build-0"],
+         'builddir': "clang-cuda-gce-build",
+         'factory': CUDATestsuiteBuilder.getCUDAAnnotatedBuildFactory(),
+         'category' : 'clang'},
+        {'name': "clang-cuda-gce-test-k80",
+         'slavenames' :["cuda-gce-test-k80-0"],
+         'builddir': "clang-cuda-gce-test-k80",
+         'factory': CUDATestsuiteBuilder.getCUDAAnnotatedBuildFactory(),
+         'category' : 'clang'},
+        {'name': "clang-cuda-gce-test-p4",
+         'slavenames' :["cuda-gce-test-p4-0"],
+         'builddir': "clang-cuda-gce-test-p4",
+         'factory': CUDATestsuiteBuilder.getCUDAAnnotatedBuildFactory(),
+         'category' : 'clang'},
+        {'name': "clang-cuda-gce-test-t4",
+         'slavenames' :["cuda-gce-test-t4-0"],
+         'builddir': "clang-cuda-gce-test-t4",
+         'factory': CUDATestsuiteBuilder.getCUDAAnnotatedBuildFactory(),
+         'category' : 'clang'},
         {'name': "clang-ve-ninja",
          'slavenames':["nec-arrproto41"],
          'builddir':"clang-ve-ninja",
          'factory': UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
                         clean=True,
                         depends_on_projects=['llvm','clang','openmp'],
                         extra_configure_args=[
                             "-DLLVM_TARGETS_TO_BUILD=X86",

buildbot/osuosl/master/config/slaves.py

return [
         # Ubuntu 16.04.3 LTS x86_64-scei-ps4, 2 x Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
         create_slave("ps4-buildslave4"),
 
         # NetBSD amd64
         create_slave("netbsd-amd64", properties={'jobs': 8}, max_builds=1),
 
         # Ubuntu 16.04 x86_64, 2 x Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz, 64GB of RAM
         create_slave("cuda-build-test-01", properties={'jobs': 72}, max_builds=1),
+        # WIP migration of the CUDA buildbot to GCE.
+        create_slave("cuda-gce-build-0", max_builds=1),
+        create_slave("cuda-gce-test-k80-0", max_builds=1),
+        create_slave("cuda-gce-test-p4-0", max_builds=1),
+        create_slave("cuda-gce-test-t4-0", max_builds=1),
 
         # Ubuntu 14.04 x86_64, AMD Athlon(tm) 5150 APU with Radeon(tm) R3, 8GiB RAM
         create_slave("am1i-slv1", properties={'jobs': 8}),
 
         # Ubuntu 14.04 x86_64, AMD Athlon(tm) 5150 APU with Radeon(tm) R3, 8GiB RAM
         create_slave("am1i-slv3", properties={'jobs': 8}),
 
         # Ubuntu 14.04 x86_64, AMD Athlon(tm) 5150 APU with Radeon(tm) R3, 8GiB RAM

buildbot/osuosl/master/config/status.py

return [
             addLogs=False,
             num_lines = 15),
         InformativeMailNotifier(
             fromaddr = "[e-mail]",
             sendToInterestedUsers= False,
             extraRecipients = ["[e-mail]"],
             subject="Build %(builder)s Failure",
             mode = "failing",
-            builders = ["clang-cuda-build"],
+            builders = ["clang-cuda-build", "clang-cuda-gce-build", "clang-cuda-gce-test"],
             addLogs=False,
             num_lines = 15),
         InformativeMailNotifier(
             fromaddr = "[e-mail]",
             sendToInterestedUsers= False,
             extraRecipients = ["[e-mail]", "[e-mail]"],
             subject="Build %(builder)s Failure",
             mode = "failing",

zorg/buildbot/builders/CUDATestsuiteBuilder.py

if (enable_thrust_tests):
                 jobs=1, # lit will parallelize the jobs.
                 haltOnFailure=True,
                 description=split("running all CUDA tests."),
                 descriptionDone=split("all cuda tests done."),
                 workdir=ts_build_dir,
                 env=merged_env))
 
     return f
+
+# Simplified CUDA builder which uses external script to perform build & test.
+def getCUDAAnnotatedBuildFactory(
+        script=None, # build/test script that will generate annotated logs.
+        clean=False,
+        is_legacy_mode=False,
+        timeout=1200,
+        env=None,  # Environmental variables for all steps.
+        is_legacy_mode=False):
+
+    # Prepare environmental variables. Set here all env we want for all steps.
+    merged_env = {
+        'TERM': 'dumb'  # Make sure Clang doesn't use color escape sequences.
+    }
+
+    if env is not None:
+        # Overwrite pre-set items with the given ones, so user can set
+        # anything.
+        merged_env.update(env)
+
+    depends_on_projects = [
+        "llvm",
+        "clang",
+        "compiler-rt",
+        "libcxx",
+        "libcxxabi",
+        "libunwind",
+        "lld"]
+
+    # Explicitly use '/' as separator, because it works on *nix and Windows.
+    # The bot script is expected to be provided
+    cuda_bot_dir = "cuda_buildbot"
+    if script:
+      cuda_bot_script = script
+    else:
+      cuda_bot_script = bot_dir + "/run_cuda_bot"
+
+    f = LLVMBuildFactory(
+        is_legacy_mode=is_legacy_mode,
+        clean=always_clean,
+        depends_on_projects=depends_on_projects,
+        llvm_srcdir=cuda_bot_dir)
+
+    # Determine the build directory.
+    f.addStep(SetProperty(name="get_builddir",
+                          command=["pwd"],
+                          property="builddir",
+                          description="set build dir",
+                          workdir=".",
+                          env=merged_env))
+
+    # Run annotated command for cuda bot.
+    f.addStep(AnnotatedCommand(name="annotate",
+                               description="annotate",
+                               timeout=timeout,
+                               haltOnFailure=True,
+                               command=cuda_bot_script,
+                               env=merged_env))
+    return f