[zorg] Enable GCS uploads on clang-cmake-mipsel and add an LNT runner that uses them.

[dsanders]

Summary

At the moment, we have a choice between systems with sufficient memory to build
LLVM and those that have the FPU needed to run the test-suite in a reasonable
amount of time. This patch uses the GCS bucket to transfer the just-built clang
to a machine more suited to running the test-suite.

[dsanders]

Refresh patch and ping

[cmatthews]

You could just use curl/urllib/requests to download the file into the build. No need to have gsutil as a dependency where it is not needed. In fact, the code to do this is already in zorg in the llvmlab tool.

[dsanders]

I mainly used gsutil for symmetry but it was also more convenient since it allowed me to re-use the gsutil:// URL by sending it to the triggered builder in a property. I don't mind changing it to use construct an HTTP GET request (or adding a dependency on llvmlab and having that do it) if that's what we prefer. The change will require additional SetProperty steps and an extra forwarded property to forward the information needed to build the request but that's not difficult.

Would you like me to switch this over?

[cmatthews]

Which ever you think is best. I used an HTTP get for my jobs to avoid the dependency, but either way is fine. I was able to use requests, so it was super simple to do it. I think for a few jobs, it won’t matter either way.

[dsanders]

Ok, I think I'd prefer to stick to gsutil for now and change it later if others start to use this code but don't like the dependency.

Is it ok to commit it?

[cmatthews]

It LGTM. Galina should approve though.

[dsanders]

It seems that this patch fell through the cracks so: Refresh and ping.

Looking back at the comments, I had a LGTM from @cmatthews that was conditonal
on @gkistanova also approving it.

buildbot/osuosl/master/config/builders.py

 from zorg.buildbot.builders import ClangBuilder
 from zorg.buildbot.builders import LLVMBuilder
 from zorg.buildbot.builders import PollyBuilder
 from zorg.buildbot.builders import LLDBBuilder
 from zorg.buildbot.builders import LLDBuilder
 from zorg.buildbot.builders import LLGoBuilder
 from zorg.buildbot.builders import ClangAndLLDBuilder
 from zorg.buildbot.builders import SanitizerBuilder
 from zorg.buildbot.builders import SanitizerBuilderII
 from zorg.buildbot.builders import SanitizerBuilderWindows
 from zorg.buildbot.builders import Libiomp5Builder
 from zorg.buildbot.builders import LibcxxAndAbiBuilder
 from zorg.buildbot.builders import SphinxDocsBuilder
 from zorg.buildbot.builders import ABITestsuitBuilder
 from zorg.buildbot.builders import ClangLTOBuilder3Stage
 
+from buildbot.process.properties import Property
+
 # Plain LLVM builders.
 def _get_llvm_builders():
     return [
         # We currently have to force LLVM_HOST_TRIPLE and
         # LLVM_DEFAULT_TARGET_TRIPLE on this system. CMake gets the value
         # correct for the processor but it's currently not possible to emit O32
         # code using a mips64-* triple. This is a bug and should be fixed soon.
         # We must also force LLVM_TARGET_ARCH so that the ExecutionEngine tests
def _get_sanitizer_builders():
            'builddir':"clang-cmake-mipsel",
            'factory' : ClangBuilder.getClangCMakeGCSBuildFactory(
                            clean=False,
                            checkout_compiler_rt=True,
                            extra_cmake_args=["-DLLVM_HOST_TRIPLE=mipsel-unknown-linux-gnu",
                                              "-DLLVM_DEFAULT_TARGET_TRIPLE=mipsel-unknown-linux-gnu",
                                              "-DLLVM_TARGET_ARCH=Mips"],
                            stage1_upload_directory='clang-cmake-mipsel',
-                           env = {'BOTO_CONFIG': '/var/buildbot/llvmlab-build-artifacts.boto'})},
+                           env = {'BOTO_CONFIG': '/var/buildbot/llvmlab-build-artifacts.boto'},
+                           trigger_after_stage1_upload=['lnt-mipsel'])},
+          ]
+
+def _get_triggerable_builders():
+      return [
+          {'name': "lnt-mipsel",
+           'slavenames': ["mips-kl-ci20_001"],
+           'builddir': "lnt-mipsel",
+           'factory': LNTBuilder.getGCSAndLNTFactory(
+                          nt_flags=[],
+                          env={
+                              'BOTO_CONFIG':
+                                  '/var/buildbot/llvmlab-build-artifacts.boto'},
+                          tester_name='lnt-mipsel')}
           ]
 
 def _get_openmp_builders():
     return [
         {'name': "libomp-gcc-x86_64-linux-debian",
          'slavenames':["gribozavr4"],
          'builddir':"libomp-gcc-x86_64-linux-debian",
          'factory' : Libiomp5Builder.getLibompCMakeBuildFactory(
def get_builders():
     for b in _get_llgo_builders():
         b['category'] = 'llgo'
         yield b
 
     for b in _get_sanitizer_builders():
         b['category'] = 'sanitizer'
         yield b
 
+    for b in _get_triggerable_builders():
+        b['category'] = 'triggerable'
+        yield b
+
     for b in _get_openmp_builders():
         b['category'] = 'openmp'
         yield b
 
     for b in _get_libcxx_builders():
         b['category'] = 'libcxx'
         yield b
 

buildbot/osuosl/master/config/slaves.py

return [
         create_slave("mips-kl-m001", properties={'jobs' : 10}, max_builds=1),
 
         # Cavium Octeon II V0.2, MIPS64r2 big endian, Debian Jessie
         create_slave("mips-kl-m002", properties={'jobs' : 10}, max_builds=1),
 
         # Cavium Octeon II V0.1, MIPS64r2 little endian, Debian Jessie
         create_slave("mips-kl-erpro001", properties={'jobs' : 2}, max_builds=1),
 
+        # Ingenic XBurst V4.15, FPU V0.0, MIPS32r2 little endian, Debian Jessie
+        create_slave("mips-kl-ci20_001", properties={'jobs' : 1}, max_builds=1),
+
         # Debian Testing x86-64, Intel(R) Core(TM) i7-2700K CPU @ 3.50GHz
         #create_slave("gribozavr1", properties={'jobs': 8}, max_builds=1),
 
         # Ubuntu 13.04 x86-64, Intel(R) Xeon(R) CPU 5160 @ 3.00GHz
         create_slave("gribozavr2", properties={'jobs': 4}, max_builds=1),
 
         # CentOS 6.5 x86-64, VM on Intel(R) Core(TM) i5 CPU 750 @ 2.67GHz
         create_slave("gribozavr3", properties={'jobs': 4}, max_builds=1),

buildbot/osuosl/master/master.cfg

 
 c['schedulers'].append(SingleBranchScheduler(name="llgo_scheduler",
                             treeStableTimer=2*60,
                             builderNames=get_all_for("llgo"),
                             change_filter=depends_on([
                                 "llvm",
                                 "cfe",
                                 "llgo"])))
+c['schedulers'].append(
+    triggerable.Triggerable(name="lnt-mipsel", builderNames=["lnt-mipsel"]))
 
 c['schedulers'].extend(
     config.schedulers.getSingleBranchSchedulers(c['builders'], c['schedulers'])
     )
 
 ####### PROJECT IDENTITY
 
 c['title'] = "LLVM"
 c['titleURL'] = "http://llvm.org/"
 c['buildbotURL'] = "http://lab.llvm.org:8011/"
 
 ####### DB URL
 
 c['db_url'] = "postgresql://localhost/buildmasterdb"

zorg/buildbot/builders/ClangBuilder.py

 import buildbot
 import buildbot.process.factory
 import copy
 import os
 from datetime import datetime
 
 from buildbot.process.properties import WithProperties, Property
 from buildbot.steps.shell import Configure, ShellCommand, SetProperty
 from buildbot.steps.shell import WarningCountingShellCommand
 from buildbot.steps.source import SVN
 from buildbot.steps.transfer import FileDownload
+from buildbot.steps.trigger import Trigger
 
 import zorg.buildbot.util.artifacts as artifacts
 import zorg.buildbot.builders.Util as builders_util
 import zorg.buildbot.util.phasedbuilderutils as phasedbuilderutils
 import zorg.buildbot.commands as commands
 import zorg.buildbot.commands.BatchFileDownload as batch_file_download
 import zorg.buildbot.commands.LitTestCommand as lit_test_command
 
gcs_url = \
             gcs_directory=lambda _: gcs_directory,
             now=lambda _: datetime.utcnow().strftime(time_fmt))
 
     if gcs_url_property:
         f.addStep(SetProperty(
                       name="record GCS url for " + package_name,
                       command=['echo', gcs_url],
                       property=gcs_url_property))
+        # Prevent a second expansion since the timestamp will change.
+        gcs_url = Property(gcs_url_property)
 
     f.addStep(ShellCommand(name='package ' + package_name,
                            command=['tar', 'cvfJ', '../install.tar.xz', '.'],
                            description='packaging ' + package_name + '...',
                            workdir=install_prefix,
                            env=env))
 
     f.addStep(ShellCommand(
                   name='upload ' + package_name + ' to storage bucket',
                   command=['gsutil', 'cp', '../install.tar.xz', gcs_url],
                   description=('uploading ' + package_name +
                                'to storage bucket ...'),
                   workdir=install_prefix,
                   env=env))
 
+def addTriggerBuilderSteps(f, builder_names, gcs_url_property=None):
+    """
+    Add steps to trigger other builders.
+
+    builder_names - A sequence containing the builders to trigger.
+    gcs_url_property - The name of a property which containing the GCS URL. It
+                       will be forwarded to the triggered builder.
+    """
+
+    set_properties = {'triggeredBuilders': builder_names}
+    if gcs_url_property:
+        set_properties[gcs_url_property] = Property(gcs_url_property)
+
+    f.addStep(
+        phasedbuilderutils.NamedTrigger(
+            'trigger builders',
+            schedulerNames=builder_names,
+            waitForFinish=False,
+            set_properties=set_properties))
+
 def getClangCMakeGCSBuildFactory(
             clean=True,
             test=True,
             cmake='cmake',
             jobs=None,
 
             # VS tools environment variable if using MSVC. For example,
             # %VS120COMNTOOLS% selects the 2013 toolchain.
def getClangCMakeGCSBuildFactory(
             # Extra repositories
             checkout_clang_tools_extra=True,
             checkout_compiler_rt=True,
 
             # Upload artifacts to Google Cloud Storage (for the llvmbisect tool)
             stage1_upload_directory=None,
 
             # Triggers
-            trigger_after_stage1=[]):
+            trigger_after_stage1_upload=[]):
     return _getClangCMakeBuildFactory(
                clean=clean, test=test, cmake=cmake, jobs=jobs, vs=vs,
                vs_target_arch=vs_target_arch, useTwoStage=useTwoStage,
                testStage1=testStage1, stage1_config=stage1_config,
                stage2_config=stage2_config, runTestSuite=runTestSuite,
                nt_flags=nt_flags, submitURL=submitURL, testerName=testerName,
                env=env, extra_cmake_args=extra_cmake_args,
                checkout_clang_tools_extra=checkout_clang_tools_extra,
                checkout_compiler_rt=checkout_compiler_rt,
                stage1_upload_directory=stage1_upload_directory,
-               trigger_after_stage1=trigger_after_stage1)
+               trigger_after_stage1_upload=trigger_after_stage1_upload)
 
 def getClangCMakeBuildFactory(
             clean=True,
             test=True,
             cmake='cmake',
             jobs=None,
 
             # VS tools environment variable if using MSVC. For example,
def _getClangCMakeBuildFactory(
             # Extra repositories
             checkout_clang_tools_extra=True,
             checkout_compiler_rt=True,
 
             # Upload artifacts to Google Cloud Storage (for the llvmbisect tool)
             stage1_upload_directory=None,
 
             # Triggers
-            trigger_after_stage1=[]):
+            trigger_after_stage1_upload=[]):
 
     ############# PREPARING
     f = buildbot.process.factory.BuildFactory()
 
     addSVNUpdateSteps(f,
                       checkout_clang_tools_extra=checkout_clang_tools_extra,
                       checkout_compiler_rt=checkout_compiler_rt,
                       checkout_test_suite=runTestSuite)
if useTwoStage or runTestSuite or stage1_upload_directory:
                                description='ninja install',
                                workdir=stage1_build,
                                env=env))
 
     if stage1_upload_directory:
         addGCSUploadSteps(f, 'stage 1', stage1_install, stage1_upload_directory,
                           env, gcs_url_property='stage1_package_gcs_url')
 
+        if trigger_after_stage1_upload:
+            addTriggerBuilderSteps(f, trigger_after_stage1_upload,
+                                   gcs_url_property='stage1_package_gcs_url')
+
     # Compute the cmake define flag to set the C and C++ compiler to clang. Use
     # clang-cl if we used MSVC for stage1.
     if not vs:
         cc = 'clang'
         cxx = 'clang++'
     else:
         cc = 'clang-cl.exe'
         cxx = 'clang-cl.exe'

zorg/buildbot/builders/LNTBuilder.py

 """
 Builders for using LNT to test LLVM/Clang.
 """
 
 import os
 
 import buildbot
 from buildbot.steps.source.svn import SVN
-from buildbot.process.properties import WithProperties
+from buildbot.process.properties import WithProperties, Property
+from buildbot.steps.shell import ShellCommand
 
 import zorg
 from zorg.buildbot.builders import ClangBuilder
 from zorg.buildbot.util.phasedbuilderutils import getBuildDir, setProperty
 from zorg.buildbot.util.artifacts import GetCompilerRoot, package_url
 
 def _get_cc(status, stdin, stdout):
     lines = filter(bool, stdin.split('\n'))
def getLNTFactory(triple, nt_flags, xfails=[], clean=True, test=False,
     # Add an LNT test runner.
     AddLNTTestsToFactory(f, nt_flags,
                          cc_path="llvm.install.1/bin/clang",
                          cxx_path="llvm.install.1/bin/clang++",
                          **lnt_args);
 
     return f
 
+def addGCSDownloadSteps(f, gcs_url, env):
+    """
+    Add steps to download from the Google Cloud Storage bucket.
+
+    f - The BuildFactory to modify.
+    gcs_url - The GCS URL of the package to download.
+    env - The environment to use. Set BOTO_CONFIG to use a configuration file
+          in a non-standard location, and BUCKET to use a different GCS bucket.
+    """
+    f.addStep(ShellCommand(name='download from storage bucket',
+                           command=['gsutil', 'cp', gcs_url,
+                                    'toolchain.tar.xz'],
+                           description='downloading from storage bucket...',
+                           haltOnFailure=True,
+                           workdir='llvm.install',
+                           env=env))
+
+    f.addStep(ShellCommand(name="remove old toolchain",
+                           command=["rm", "-rf", 'install'],
+                           haltOnFailure=True,
+                           description=["removing old toolchain..."],
+                           workdir="."))
+
+    f.addStep(ShellCommand(name="unpack toolchain",
+                           command=['tar', 'xvfJ', 'toolchain.tar.xz'],
+                           haltOnFailure=True,
+                           description=["unpacking toolchain..."],
+                           workdir="llvm.install"))
+
+def getGCSAndLNTFactory(nt_flags, env={}, tester_name=None):
+    f = buildbot.process.factory.BuildFactory()
+    addGCSDownloadSteps(f, Property('stage1_package_gcs_url'), env=env)
+    cc_path = 'llvm.install/bin/clang'
+    cxx_path = 'llvm.install/bin/clang++'
+    AddLNTTestsToFactory(f, nt_flags, cc_path=cc_path,
+                         cxx_path=cxx_path, testerName=tester_name);
+    return f
+
 def AddLNTTestsToFactory(f, nt_flags, cc_path, cxx_path, **kwargs):
     """
     Add the buildbot steps necessary to run an LNT driven test of a compiler.
 
     This assumes at a minimum that the factory has already been set up to
     contain a builddir property which points at the full path to the build
     directory.
     """