[zorg] Add support for uploading artifacts to the 'llvmlab bisect' bucket and enable this for clang-cmake-mips.

[dsanders]

Summary

Adds a new keyword argument to getClangCMakeBuildFactory():

stage1_upload_directory - This is the bucket subdirectory to upload to and should match the builder name.

The buildslave must have gsutil installed to use this argument.

[dsanders]

Once our little-endian builder+buildslave is added to the buildbot I'll enable this there too.
Also, once I've configured the necessary machine I'll follow up with a further patch to add an lnt builder factory that takes pre-built toolchains from this bucket.

[gkistanova]

Could you consider subclassing and creating a separate build factory for this kind of builders instead, since you do not expect all the existing builders upload to the Google Cloud Storage, please?

The current getClangCMakeBuildFactory is consistent with the build performance data collection and analyses. With the uploading steps it wouldn't be any more. I prefer having a separate build factory for this case.
Thanks for working on this!

Thanks

Galina

zorg/buildbot/builders/ClangBuilder.py

689

Please define this as a builder env from the beginning, without introducing a new builder param.

[dsanders]

In D14866#295170, @gkistanova wrote:

Could you consider subclassing and creating a separate build factory for this kind of builders instead

I'm not sure what you mean by subclassing in this context since it's a plain function. I can duplicate the function or split it into separate stage1/stage2 parts and add the upload between stage1+stage2 if that helps.

since you do not expect all the existing builders upload to the Google Cloud Storage, please?

When Chris announced the bisection tool (http://lists.llvm.org/pipermail/llvm-dev/2015-October/091140.html) he invited other bot owners to upload to this area so it may turn out that others are interested too. I haven't asked around though.

The current getClangCMakeBuildFactory is consistent with the build performance data collection and analyses. With the uploading steps it wouldn't be any more. I prefer having a separate build factory for this case.

I'm not sure I understand this. Adding an upload step doesn't prevent the collection of data in the (optional) lnt step as far as I can tell. Am I missing something?

Thanks for working on this!

Thanks

Galina

zorg/buildbot/builders/ClangBuilder.py

689

Sure

[dsanders]

Dropped boto_config_file in favour of 'env'.

[• chatur01]

Hi Daniel,

FWIW, I had a near identical patch locally (making the same decision to add an extra keyword argument, etc...) so it LGTM. My only interpretation of Galina's suggestion was to create a new function called getClangGCSBuildFactory(...) that calls getClangBuildFactory(...) and then adds the extra upload step. I don't see what benefit that would bring TBH. Hopefully Galina can clarify.

Have you acquired the relevant credentials to use gsutil like this? I asked a while back, and was told I need to make sure everything works OK with my own GCS account before pushing to the official GCS. Unfortunately I'm blocked getting a GCS internally, I'd be interested in hearing what policy you followed to get this commit-ready. If you've demonstrated this works, then that should allow the rest of us Buildbot users to follow your lead and get our builds into the cloud as well.

[dsanders]

In D14866#297725, @chatur01 wrote:

Hi Daniel,

FWIW, I had a near identical patch locally (making the same decision to add an extra keyword argument, etc...) so it LGTM.
My only interpretation of Galina's suggestion was to create a new function called getClangGCSBuildFactory(...) that calls
getClangBuildFactory(...) and then adds the extra upload step. I don't see what benefit that would bring TBH. Hopefully Galina can clarify.

As far as I know it's only possible to append steps to a factory so I think I'd have to duplicate or split the function.

Have you acquired the relevant credentials to use gsutil like this? I asked a while back, and was told I need to make sure
everything works OK with my own GCS account before pushing to the official GCS. Unfortunately I'm blocked getting a GCS
internally, I'd be interested in hearing what policy you followed to get this commit-ready.
If you've demonstrated this works, then that should allow the rest of us Buildbot users to follow your lead and get our builds into the cloud as well.

I've got the config and I've checked it works by uploading a file and deleting it again.

To get it ready, I tested it with a private buildbot using the zorg scripts and modifications to disable mail and most of the builders/slaves. I'm using a newer version of buildbot but I don't think I've used anything new. On the GCS side, it turned out I didn't need to go through an approval process so long as I didn't spend any money :-). I therefore used the free trial. I've tested phased builders and some of the 'llvmlab bisect' commands. I haven't tried an actual bisect, but I have confirmed the filenames and tarball contents are the same as the two builders that are already uploading.

[gkistanova]

Sorry for not being clear from the beginning.
Anyway, your interpretation is correct. I meant having a separate get<something>BuildFactory(...) function for the builders which would upload blobs.

My only interpretation of Galina's suggestion was to create a new function called getClangGCSBuildFactory(...) that calls
getClangBuildFactory(...) and then adds the extra upload step.

Unfortunately, with the current getClangBuildFactory design this is not that simple. To do that right would require a relatively large refactoring of the getClangBuildFactory, which I don't want to force Daniel do just to get this patch in.

So, adding a new get<something>BuildFactory(...) which would simply call the changed getClangBuildFactory or copying and changing the actually needed code from the getClangBuildFactory would do for now.

I don't see what benefit that would bring TBH. Hopefully Galina can clarify.

There are 2 issues here:

1. Having the getClangBuildFactory changed this way, we could all of a sudden get a lot of builders uploading or trying to upload blobs by just changing the default value of one of the many optional parameters. This might be something the buildslave owners didn't consider and have setup for in the first place.

2. The getClangBuildFactory already due for refactoring, and adding more parameters there is not a good idea in general. Though, this is acceptable in this particular case as a "till-the-next-refactoring" solution, but I'd keep the dependencies as small and clear as possible.

Hope this make sense.

[dsanders]

• Renamed getClangCMakeBuilder() to _getClangCMakeBuilder()
• Added getClangCMakeBuildFactory() with the old set of arguments and getClangCMakeGCSBuildFactory() with the additional one.
• Split additions to _getClangCMakeBuilder() into a separate addGCSUploadSteps() to be more in line with the likely result of refactoring getClangCMakeBuilder(). Also generalized it enough to be reusable for stage 2 uploads.
• Small cosmetic tweaks (e.g. 'upload' -> 'uploading' for the in-progress description) and tidy up.

[dsanders]

Thanks, I think I understand now. Is the updated diff what you have in mind? I've taken the easy option in this patch but it should be easy to extract some of the duplication using Bicycle Repair Man or a similar tool.

Am I right in thinking that long term you want to end up with something like:

getClangCMakeBuildFactory(...) \
    .addStage1Steps(clean=..., test=..., install=...) \
    .addStage2Steps(clean=..., test=..., install=...) \
    .addGCSUploadSteps(...) \
    .addTestsuiteSteps(...)

[gkistanova]

Thanks, Daniel!

Is the updated diff what you have in mind?

This is exactly what I have in mind.

Am I right in thinking that long term you want to end up with something like:

Yes, I'm thinking along those lines. Maybe having more universal "bricks" which could be used for both stages, and such. Grouping steps and re-using the groups would help adding custom actions in the middle of the build flow, and keep specific factories separated, rather than having one large factory heavily parametric.

[dsanders]

Thanks. I've committed it but it looks like the buildmaster hasn't picked up the changes automatically. It might need a restart.

buildbot/osuosl/master/config/builders.py

def _get_sanitizer_builders():
           {'name': "clang-cmake-mips",
            'slavenames':["mips-kl-m001"],
            'builddir':"clang-cmake-mips",
            'factory' : ClangBuilder.getClangCMakeBuildFactory(
                            clean=False,
                            checkout_compiler_rt=True,
                            extra_cmake_args=["-DLLVM_HOST_TRIPLE=mips-unknown-linux-gnu",
                                              "-DLLVM_DEFAULT_TARGET_TRIPLE=mips-unknown-linux-gnu",
-                                             "-DLLVM_TARGET_ARCH=Mips"])},
+                                             "-DLLVM_TARGET_ARCH=Mips"],
+                           stage1_upload_directory='clang-cmake-mips',
+                           env = {'BOTO_CONFIG': '/var/buildbot/llvmlab-build-artifacts.boto'},
           ]
 
 def _get_openmp_builders():
     return [
         {'name': "libomp-gcc-x86_64-linux-debian",
          'slavenames':["gribozavr4"],
          'builddir':"libomp-gcc-x86_64-linux-debian",
          'factory' : Libiomp5Builder.getLibompCMakeBuildFactory(

zorg/buildbot/builders/ClangBuilder.py

 import buildbot
 import buildbot.process.factory
 import copy
 import os
+from datetime import datetime
 
 from buildbot.process.properties import WithProperties, Property
 from buildbot.steps.shell import Configure, ShellCommand, SetProperty
 from buildbot.steps.shell import WarningCountingShellCommand
 from buildbot.steps.source import SVN
 from buildbot.steps.transfer import FileDownload
 
 import zorg.buildbot.util.artifacts as artifacts
if checkout_test_suite:
                       mode='update', baseURL='http://llvm.org/svn/llvm-project/lnt/',
                       defaultBranch='trunk',
                       workdir='test/lnt'))
         f.addStep(SVN(name='svn-test-suite',
                       mode='update', baseURL='http://llvm.org/svn/llvm-project/test-suite/',
                       defaultBranch='trunk',
                       workdir='test/test-suite'))
 
+def addGCSUploadSteps(f, package_name, install_prefix, gcs_directory, env,
+                      gcs_url_property=None):
+    """
+    Add steps to upload to the Google Cloud Storage bucket.
+
+    f - The BuildFactory to modify.
+    package_name - The name of this package for the descriptions (e.g.
+                   'stage 1')
+    install_prefix - The directory the build has been installed to.
+    gcs_directory - The subdirectory of the bucket root to upload to. This
+                    should match the builder name.
+    env - The environment to use. Set BOTO_CONFIG to use a configuration file
+          in a non-standard location, and BUCKET to use a different GCS bucket.
+    gcs_url_property - Property to assign the GCS url to.
+    """
+
+    gcs_url_fmt = ('gs://%(gcs_bucket)s/%(gcs_directory)s/'
+                   'clang-r%(got_revision)s-t%(now)s-b%(buildnumber)s.tar.xz')
+    time_fmt = '%Y-%m-%d_%H-%M-%S'
+    gcs_url = \
+        WithProperties(
+            gcs_url_fmt,
+            gcs_bucket=lambda _: env.get('BUCKET', 'llvm-build-artifacts'),
+            gcs_directory=lambda _: gcs_directory,
+            now=lambda _: datetime.utcnow().strftime(time_fmt))
+
+    if gcs_url_property:
+        f.addStep(SetProperty(
+                      name="record GCS url for " + package_name,
+                      command=['echo', gcs_url],
+                      property=gcs_url_property))
+
+    f.addStep(ShellCommand(name='package ' + package_name,
+                           command=['tar', 'cvfJ', '../install.tar.xz', '.'],
+                           description='packaging ' + package_name + '...',
+                           workdir=install_prefix,
+                           env=env))
+
+    f.addStep(ShellCommand(
+                  name='upload ' + package_name + ' to storage bucket',
+                  command=['gsutil', 'cp', '../install.tar.xz', gcs_url],
+                  description=('uploading ' + package_name +
+                               'to storage bucket ...'),
+                  workdir=install_prefix,
+                  env=env))
+
+def getClangCMakeGCSBuildFactory(
+            clean=True,
+            test=True,
+            cmake='cmake',
+            jobs=None,
+
+            # VS tools environment variable if using MSVC. For example,
+            # %VS120COMNTOOLS% selects the 2013 toolchain.
+            vs=None,
+            vs_target_arch='x86',
+
+            # Multi-stage compilation
+            useTwoStage=False,
+            testStage1=True,
+            stage1_config='Release',
+            stage2_config='Release',
+
+            # Test-suite
+            runTestSuite=False,
+            nt_flags=[],
+            submitURL=None,
+            testerName=None,
+
+            # Environmental variables for all steps.
+            env={},
+            extra_cmake_args=[],
+
+            # Extra repositories
+            checkout_clang_tools_extra=True,
+            checkout_compiler_rt=True,
+
+            # Upload artifacts to Google Cloud Storage (for the llvmbisect tool)
+            stage1_upload_directory=None,
+
+            # Triggers
+            trigger_after_stage1=[]):
+    return _getClangCMakeBuildFactory(
+               clean=clean, test=test, cmake=cmake, jobs=jobs, vs=vs,
+               vs_target_arch=vs_target_arch, useTwoStage=useTwoStage,
+               testStage1=testStage1, stage1_config=stage1_config,
+               stage2_config=stage2_config, runTestSuite=runTestSuite,
+               nt_flags=nt_flags, submitURL=submitURL, testerName=testerName,
+               env=env, extra_cmake_args=extra_cmake_args,
+               checkout_clang_tools_extra=checkout_clang_tools_extra,
+               checkout_compiler_rt=checkout_compiler_rt,
+               stage1_upload_directory=stage1_upload_directory,
+               trigger_after_stage1=trigger_after_stage1)
 
 def getClangCMakeBuildFactory(
             clean=True,
             test=True,
             cmake='cmake',
             jobs=None,
 
             # VS tools environment variable if using MSVC. For example,
def getClangCMakeBuildFactory(
 
             # Environmental variables for all steps.
             env={},
             extra_cmake_args=[],
 
             # Extra repositories
             checkout_clang_tools_extra=True,
             checkout_compiler_rt=True):
+    return _getClangCMakeBuildFactory(
+               clean=clean, test=test, cmake=cmake, jobs=jobs, vs=vs,
+               vs_target_arch=vs_target_arch, useTwoStage=useTwoStage,
+               testStage1=testStage1, stage1_config=stage1_config,
+               stage2_config=stage2_config, runTestSuite=runTestSuite,
+               nt_flags=nt_flags, submitURL=submitURL, testerName=testerName,
+               env=env, extra_cmake_args=extra_cmake_args,
+               checkout_clang_tools_extra=checkout_clang_tools_extra,
+               checkout_compiler_rt=checkout_compiler_rt)
+
+def _getClangCMakeBuildFactory(
+            clean=True,
+            test=True,
+            cmake='cmake',
+            jobs=None,
+
+            # VS tools environment variable if using MSVC. For example,
+            # %VS120COMNTOOLS% selects the 2013 toolchain.
+            vs=None,
+            vs_target_arch='x86',
+
+            # Multi-stage compilation
+            useTwoStage=False,
+            testStage1=True,
+            stage1_config='Release',
+            stage2_config='Release',
+
+            # Test-suite
+            runTestSuite=False,
+            nt_flags=[],
+            submitURL=None,
+            testerName=None,
+
+            # Environmental variables for all steps.
+            env={},
+            extra_cmake_args=[],
+
+            # Extra repositories
+            checkout_clang_tools_extra=True,
+            checkout_compiler_rt=True,
+
+            # Upload artifacts to Google Cloud Storage (for the llvmbisect tool)
+            stage1_upload_directory=None,
+
+            # Triggers
+            trigger_after_stage1=[]):
 
     ############# PREPARING
     f = buildbot.process.factory.BuildFactory()
 
     addSVNUpdateSteps(f,
                       checkout_clang_tools_extra=checkout_clang_tools_extra,
                       checkout_compiler_rt=checkout_compiler_rt,
                       checkout_test_suite=runTestSuite)
def _getClangCMakeBuildFactory(
     # Global configurations
     stage1_build = 'stage1'
     stage1_install = 'stage1.install'
     stage2_build = 'stage2'
     stage2_install = 'stage2.install'
 
     # Set up VS environment, if appropriate.
     if vs:
         f.addStep(SetProperty(
             command=builders_util.getVisualStudioEnvironment(vs, vs_target_arch),
             extract_fn=builders_util.extractSlaveEnvironment))
         assert not env, "Can't have custom builder env vars with VS"
         env = Property('slave_env')
 
 
     ############# CLEANING
     if clean:
if test and testStage1:
         f.addStep(lit_test_command.LitTestCommand(name='ninja check 1',
                                    command=ninja_check_cmd,
                                    haltOnFailure=haltOnStage1Check,
                                    description=["checking stage 1"],
                                    descriptionDone=["stage 1 checked"],
                                    workdir=stage1_build,
                                    env=env))
 
-    if useTwoStage or runTestSuite:
+    if useTwoStage or runTestSuite or stage1_upload_directory:
         f.addStep(ShellCommand(name='install stage 1',
                                command=ninja_install_cmd,
                                description='ninja install',
                                workdir=stage1_build,
                                env=env))
 
+    if stage1_upload_directory:
+        addGCSUploadSteps(f, 'stage 1', stage1_install, stage1_upload_directory,
+                          env, gcs_url_property='stage1_package_gcs_url')
+
     # Compute the cmake define flag to set the C and C++ compiler to clang. Use
     # clang-cl if we used MSVC for stage1.
     if not vs:
         cc = 'clang'
         cxx = 'clang++'
     else:
         cc = 'clang-cl.exe'
         cxx = 'clang-cl.exe'