[zorg] Add lldb build factory that launches scripts on slave machine

[chying]

Summary

• Add new function getLLDBScriptCommandsFactory, it will launch script file for each step
• Add new builders lldb Windows, and lldb buildserver
• Create triggerable scheduler to trigger android builds
• Add new builders to rotation email monitor list

[chaoren]

zorg/buildbot/builders/LLDBBuilder.py

923

Is this for both Linux/OS X and Windows? On Linux/OS X, *.sh might be more appropriate. You can add another argument called 'scriptExtension' or something.

926

We don't need this until we start testing. Could you move this between build and test?

943

We can probably condense cleanup, checkout, and configure into one configure script. I don't think the first two steps would be useful to see separately.

[chying]

zorg/buildbot/builders/LLDBBuilder.py

923

Yes, this factory should work for all host platforms.
Will use different extensions.

926

I was thinking if this revision of binaries doesn't exist, then the build will terminate from here.
This might happen when the build is not triggered by buildserver, like when user click Force Build from webpage.

943

We had some build failures with source checkout due to SVN server issue.
How about combine cleanup and checkout?

[chaoren]

zorg/buildbot/builders/LLDBBuilder.py

926

like when user click Force Build from webpage

In that case shouldn't we just fetch the most recent lldb-server?

943

Sounds good.

[chying]

zorg/buildbot/builders/LLDBBuilder.py

926

Yes, when user click Force Build, it should fetch most recent revision.
But it depends on buildserver to create lldb-server binaries.
It is possible bulidserver hasn't finished build with the most recent revision.

[chaoren]

zorg/buildbot/builders/LLDBBuilder.py

926

The Force Build button seems useless then, unless you add some sort of circular trigger. It doesn't make sense for the bot to just fail either. Can you disable the button just for these bots?

[chying]

Address code review feedback

• Combine clewqanup step with source checkout
• Move download binary step after local build
• Use different extension based on host platform
• Add new argument to indicate whether download binary or not
• Remove triggerBuild argument as it always has the same value with bulidAndroid

[chying]

zorg/buildbot/builders/LLDBBuilder.py

926

Had offline discussion with Chaoren, Force Build will take the latest available lldb-server binaries for android. It will be handled in downloadBinary script.

[chaoren]

LGTM.

buildbot/osuosl/master/config/builders.py

#         'category' : 'clang'},
                         '--enable-cxx11',
                         '--enable-optimized',
                         '--enable-assertions'],
                     env={'SHELL':"/bin/bash"})},
         {'name': "lldb-x86_64-ubuntu-14.04-cmake",
          'slavenames': ["lldb-build1-ubuntu-1404"],
          'builddir': "buildWorkingDir",
          'category' : 'lldb',
-         'factory': LLDBBuilder.getLLDBUbuntuCMakeBuildFactory(
-                    build_compiler="clang",
-                    build_type="Release",
-                    local_test_archs=['x86_64', 'i386'],
-                    local_test_compilers=['clang', 'totclang', 'gcc4.8.2', 'gcc4.9.2'],
-                    #remote_configs=[RemoteConfig('linux','x86_64',['clang'],['x86_64'])],
-                    env={'SHELL':"/bin/bash"})},
+         'factory': LLDBBuilder.getLLDBScriptCommandsFactory(
+                    downloadBinary=False,
+                    buildAndroid=False,
+                    runTest=True)},
         {'name': "lldb-x86_64-darwin-13.4",
          'slavenames': ["lldb-x86_64-darwin-13.4"],
          'builddir': "buildDir",
          'category' : 'lldb',
-         'factory': LLDBBuilder.getLLDBxcodebuildFactory(
-                    build_type='Release',
-                    remote_configs=[RemoteConfig('android', 'arm', ['arm-linux-androideabi-gcc'], ['arm'])],
-                    env={'SHELL':"/bin/bash"})},
+         'factory': LLDBBuilder.getLLDBScriptCommandsFactory(
+                    downloadBinary=True,
+                    buildAndroid=False,
+                    runTest=True)},
         {'name': "lldb-x86_64-ubuntu-14.04-android",
          'slavenames': ["lldb-lab-linux01"],
          'builddir': "lldb-cross-compile",
          'category' : 'lldb',
-         'factory': LLDBBuilder.getLLDBUbuntuCMakeBuildFactory(
-                    build_compiler="clang",
-                    build_type="Release",
-                    remote_configs= [
-                                     RemoteConfig('android', 'i386', ['i686-linux-android-gcc'], ['i386']),
-                                     RemoteConfig('android', 'arm', ['arm-linux-androideabi-gcc'], ['arm']),
-                                     RemoteConfig('android', 'aarch64', ['aarch64-linux-android-gcc'], ['aarch64'])
-                                    ],
-                    env={'SHELL':"/bin/bash"})},
+         'factory': LLDBBuilder.getLLDBScriptCommandsFactory(
+                    downloadBinary=True,
+                    buildAndroid=False,
+                    runTest=True)},
+        {'name': "lldb-windows7-android",
+         'slavenames': ["lldb-lab-win01"],
+         'builddir': "lldb-win7-android",
+         'category': "lldb",
+         'factory': LLDBBuilder.getLLDBScriptCommandsFactory(
+                    downloadBinary=True,
+                    buildAndroid=False,
+                    runTest=True,
+                    scriptExt='.bat')},
+        {'name': "lldb-x86_64-ubuntu-14.04-buildserver",
+         'slavenames': ["lldb-linux-android-buildserver"],
+         'builddir': "lldb-android-buildserver",
+         'category' : 'lldb',
+         'factory': LLDBBuilder.getLLDBScriptCommandsFactory(
+                    downloadBinary=False,
+                    buildAndroid=True,
+                    runTest=False)}
         ]
 
 # Builders responsible building Sphinix documentation
 def _get_documentation_builders():
     return [
              {
                'name':"llvm-sphinx-docs",
                'slavenames':["gribozavr4"],

buildbot/osuosl/master/config/slaves.py

32},
         create_slave("lldb-build1-ubuntu-1404", properties={'jobs': 32, 'loadaverage':32}, max_builds=1),
         # Darwin 13.4.0, x86_64
         create_slave("lldb-x86_64-darwin-13.4", properties={'jobs': 16, 'loadaverage':32}, max_builds=1),
         # Ubuntu 14.04 x86_64, Intel(R) Xeon(R) CPU @ 2.30GHz
         create_slave("llgo-builder", properties={'jobs': 2}, max_builds=1),
         # Ubuntu 14.04 x86_64, Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
         create_slave("lldb-lab-linux01", properties={'jobs': 40}, max_builds=1),
 
+        # Windows 7 x86_64
+        create_slave("lldb-lab-win01", properties={'jobs': 40}, max_builds=1),
+        # Ubuntu 14.04 x86_64
+        create_slave("lldb-linux-android-buildserver", properties={'jobs': 40}, max_builds=1),
         # Ubuntu 14.04 x86_64, Intel(R) Xeon(R) CPU L5520 @ 2.27GHz
         create_slave("bpf-build-slave01", properties={'jobs': 16}, max_builds=1),
 
         # Defunct.
         # Intel(R) Pentium(R) CPU G620 @ 2.60GHz, Ubuntu i686
         #create_slave("botether", properties={'jobs': 2}, max_builds=1),
 
         # Intel(R) Core(TM)2 CPU 6420  @ 2.13GHz, Ubuntu Oneiric x86_64

buildbot/osuosl/master/config/status.py

return [
         InformativeMailNotifier(
             fromaddr = "[e-mail]",
             sendToInterestedUsers= False,
             extraRecipients = ["[e-mail]",
                                "[e-mail]"],
             subject="Build %(builder)s Failure",
             mode = "failing",
             builders = ["lldb-x86_64-ubuntu-14.04-cmake",
+                        "lldb-x86_64-ubuntu-14.04-buildserver",
+                        "lldb-windows7-android",
                         "lldb-x86_64-darwin-13.4",
                         "lldb-x86_64-ubuntu-14.04-android"],
             addLogs=False,
             num_lines = 15),
         InformativeMailNotifier(
             fromaddr = "[e-mail]",
             sendToInterestedUsers= False,
             extraRecipients = ["[e-mail]",
                                "[e-mail]"],
             subject="Build %(builder)s Failure",
             mode = "failing",
             builders = ["clang-x86_64-linux-abi-test"],
             addLogs=False,
             num_lines = 15),
 
         ]

buildbot/osuosl/master/master.cfg

standard_categories = [b['category'] for b in builders
                           not b['name'].startswith('perf-x86_64')]
 
 c['status'] = config.status.get_status_targets(standard_builders,
                                                standard_categories)
 
 ####### SCHEDULERS
 
 from buildbot.scheduler import Scheduler
+from buildbot.schedulers import triggerable
 from buildbot.schedulers.filter import ChangeFilter
 from buildbot.schedulers.basic import SingleBranchScheduler
 
 def depends_on(projects):
     """
     Returns a ChangeFilter for the given projects.
     """
     return ChangeFilter(project=projects)

 
 c['schedulers'].append(SingleBranchScheduler(name="lld_scheduler",
                             treeStableTimer=2*60,
                             builderNames=get_all_for("lld"),
                             change_filter=depends_on([
                                 "llvm",
                                 "lld"])))
 
+# lldb android builders will be triggered by completion of buildserver instead of by source change
+lldb_android_builders=["lldb-x86_64-ubuntu-14.04-android",
+                       "lldb-windows7-android",
+                       "lldb-x86_64-darwin-13.4"]
+c['schedulers'].append((triggerable.Triggerable(name="lldb_android_scheduler",
+                                                builderNames=lldb_android_builders)))
+
 c['schedulers'].append(SingleBranchScheduler(name="lldb_scheduler",
                             treeStableTimer=2*60,
-                            builderNames=get_all_for("lldb"),
+                            builderNames=list(set(get_all_for("lldb"))-set(lldb_android_builders)),
                             change_filter=depends_on([
                                 "llvm",
                                 "cfe",
                                 "lldb"])))
 
 c['schedulers'].append(SingleBranchScheduler(name="sanitizer_scheduler",
                             treeStableTimer=2*60,
                             builderNames=get_all_for("sanitizer"),

zorg/buildbot/builders/LLDBBuilder.py

 import os
 import json
 import collections
 import buildbot
 import buildbot.process.factory
 from buildbot.steps.source import SVN
 from buildbot.steps.shell import Configure, SetProperty
 from buildbot.steps.shell import ShellCommand, WarningCountingShellCommand
 from buildbot.steps.slave import RemoveDirectory
 from buildbot.process.properties import WithProperties, Property
+from buildbot.steps import trigger
 import zorg.buildbot.commands.BatchFileDownload as batch_file_download
 from zorg.buildbot.commands.LitTestCommand import LitTestCommand
 from zorg.buildbot.builders.Util import getVisualStudioEnvironment
 from zorg.buildbot.builders.Util import extractSlaveEnvironment
 
 # We *must* checkout at least Clang, LLVM, and LLDB.  Once we add a step to run
 # tests (e.g. ninja check-lldb), we will also need to add a step for LLD, since
 # MSVC LD.EXE cannot link executables with DWARF debug info.
# Compress and upload test log
     f = archiveLLDBTestTraces(f, "build/lldb-test-traces*")
 
 # Results go in a directory coded named according to the date and time of the test run, e.g.:
 #
 # 2012-10-16-11_26_48/Failure-x86_64-_Applications_Xcode.app_Contents_Developer_Toolchains_XcodeDefault.xctoolchain_usr_bin_clang-TestLogging.LogTestCase.test_with_dsym.log
 #
 # Possible results are ExpectedFailure, Failure, SkippedTest, UnexpectedSuccess, and Error.    return f
     return f
+
+def getShellCommandStep(f,
+                        name,
+                        command,
+                        description="",
+                        flunkOnFailure=True,
+                        haltOnFailure=True,
+                        workdir='scripts',
+                        env=None):
+    if env is None:
+        env = {}
+    f.addStep(ShellCommand(name=name,
+                           command=command,
+                           description=description,
+                           env=env,
+                           flunkOnFailure=flunkOnFailure,
+                           haltOnFailure=haltOnFailure,
+                           workdir=workdir))
+
+# get configuration of tests
+# config file should be placed under builddir on builder machine
+# file name: test_cfg.json
+# content: json format with keys test[num]:target-compiler-architecture
+# example: {"test1":"android-gcc4.9-i386",
+#           "test2":"local-clang-x86",
+#           "test3":"local-clang-i386"}
+def getTestConfig(f):
+    def getRemoteCfg(rc, stdout, stderr):
+        return json.loads(stdout)
+    f.addStep(SetProperty(name="get test config",
+                          command="cat test_cfg.json",
+                          extract_fn=getRemoteCfg,
+                          description="get remote target",
+                          workdir="scripts"))
+    return f
+def getTestSteps(f, scriptExt):
+    # buildbot doesn't support dynamic step creation, so create 8 test steps as place holder
+    # then each builder will define available tests in test_cfg.json
+    # if there're less than 8 tests defined on certain builder, extra steps will be skipped and hidden from test details view
+    # **hide step is not supported by buildbot 0.8.5
+    # flunkOnFailure only takes boolean value, and cannot take configurable property.
+    # workaround: don't flunk the last two tests
+    # put non flunkable tests as the last two, test7 and test8
+    getTestConfig(f)
+    for x in range(1, 9):
+        test='test'+str(x)
+        f.addStep(LitTestCommand(name=test,
+                                 command=['test' + scriptExt,
+                                          Property(test)],
+                                 description="",
+                                 doStepIf=lambda step: step.build.hasProperty(step.name),
+                                 flunkOnFailure=(x<7),
+                                 warnOnFailure=(x<7),
+                                 workdir='scripts'))
+
+def getLLDBScriptCommandsFactory(
+                       downloadBinary=True,
+                       buildAndroid=False,
+                       runTest=True,
+                       scriptExt='.sh',
+                       ):
+    f = buildbot.process.factory.BuildFactory()
+
+    # Checkout source code
+    getShellCommandStep(f, name='checkout source code',
+                        command=['checkoutSource' + scriptExt,
+                                 WithProperties('%(revision)s')])
+
+    # Set source revision
+    f.addStep(SetProperty(name="set revision",
+              command=['svnversion'],
+              property="got_revision",
+              workdir="llvm"))
+
+    # Configure
+    getShellCommandStep(f, name='cmake local',
+                        command=['cmake' + scriptExt])
+
+    # Build
+    getShellCommandStep(f, name='ninja build local',
+                        command=['buildLocal' + scriptExt])
+    if buildAndroid:
+        getShellCommandStep(f, name='build android',
+                            command=['buildAndroid' + scriptExt])
+
+    # Get lldb-server binaries
+    if downloadBinary:
+        getShellCommandStep(f, name='get lldb-server binaries',
+                            command=['downloadBinaries' + scriptExt,
+                                      WithProperties('%(revision)s')])
+
+    # Test
+    if runTest:
+        getTestSteps(f, scriptExt)
+        # upload test traces
+        getShellCommandStep(f, name='upload test traces',
+                            command=['uploadTestTrace' + scriptExt],
+                            flunkOnFailure=False)
+
+    # Upload lldb-server binaries and trigger android builders
+    if buildAndroid:
+        getShellCommandStep(f, name='upload lldb-server binaries',
+                            command=['uploadBinaries' + scriptExt])
+        f.addStep(trigger.Trigger(schedulerNames=['lldb_android_scheduler'],
+                                  waitForFinish=False))
+    return f