[zorg] add llgo builder, osuosl config

[axw]

Summary

Add LLGoBuilder to zorg, and add base
changes to osuosl buildbot config. There are
no slaves set up yet.

Builds fail currently, pending fixes to llgo:
D7851, D7852.

[pcc]

check-libgo will also fail because the tests for regexp and runtime/pprof are currently failing. Maybe we can find some way to run the tests for every package except those two to make sure we don't regress?

[axw]

In D7989#132285, @pcc wrote:

check-libgo will also fail because the tests for regexp and runtime/pprof are currently failing. Maybe we can find some way to run the tests for every package except those two to make sure we don't regress?

Yes, I'll propose another change that removes those packages from TEST_PACKAGES in the libgo Makefile.

[axw]

In D7989#132432, @axw wrote:

In D7989#132285, @pcc wrote:

check-libgo will also fail because the tests for regexp and runtime/pprof are currently failing. Maybe we can find some way to run the tests for every package except those two to make sure we don't regress?

Yes, I'll propose another change that removes those packages from TEST_PACKAGES in the libgo Makefile.

check-libgo now passes on trunk.

[gkistanova]

Thanks for working on this, Andrew!

You may want to consider clean builds as well as incremental builds supported now.
In this case, you would need to support at least a 'clean' property and add a step to remove the llvm_objdir with all the content if the property is set (by using the doStepIf param). This is not a show stopper for this patch, though.

zorg/buildbot/builders/LLGoBuilder.py

25

You import from buildbot.steps.shell, why not import SetProperty there as well?

66

Could you use NinjaCommand instead of direct invoking ShellCommand with "ninja" here and below, please?

[axw]

Address review comments on LLGoBuilder

• Tidy up imports
• Add clean build option, now default
• Use NinjaCommand

[axw]

In D7989#138562, @gkistanova wrote:

Thanks for working on this, Andrew!

You may want to consider clean builds as well as incremental builds supported now.
In this case, you would need to support at least a 'clean' property and add a step to remove the llvm_objdir with all the content if the property is set (by using the doStepIf param). This is not a show stopper for this patch, though.

Thanks for the review, and suggestions. I've added the clean param, and defaulted it to True as in LLVMBuilder and ClangBuilder. Also removed some unnecessary imports.

zorg/buildbot/builders/LLGoBuilder.py

25

Done. (I had copied another builder.)

66

Done.

[gkistanova]

Could you add a comment that this builder does not support Windows as is, please?

[axw]

In D7989#139371, @gkistanova wrote:

Could you add a comment that this builder does not support Windows as is, please?

Done, thanks!

buildbot/osuosl/master/config/builders.py

 from zorg.buildbot.builders import LLDBBuilder
 reload(LLDBBuilder)
 from zorg.buildbot.builders import LLDBBuilder
 
 from zorg.buildbot.builders import LLDBuilder
 reload(LLDBuilder)
 from zorg.buildbot.builders import LLDBuilder
 
+from zorg.buildbot.builders import LLGoBuilder
+reload(LLGoBuilder)
+from zorg.buildbot.builders import LLGoBuilder
+
 from zorg.buildbot.builders import ClangAndLLDBuilder
 reload(ClangAndLLDBuilder)
 from zorg.buildbot.builders import ClangAndLLDBuilder
 
 from zorg.buildbot.builders import SanitizerBuilder
 reload(SanitizerBuilder)
 from zorg.buildbot.builders import SanitizerBuilder
 
return [
                                                   extra_configure_args=[
                                                       '-DCMAKE_EXE_LINKER_FLAGS=-lcxxrt',
                                                       '-DLLVM_ENABLE_WERROR=OFF'],
                                                   env={'CXXFLAGS' : "-std=c++11 -stdlib=libc++"}),
          'category'   : 'lld'},
 
          ]
 
+
+# llgo builders.
+def _get_llgo_builders():
+    # No build slaves set up for llgo yet.
+    return []
+
+
 # Sanitizer builders.
 def _get_sanitizer_builders():
       return [
           {'name': "sanitizer-x86_64-linux",
            'slavenames' :["sanitizer-buildbot1"],
            'builddir': "sanitizer-x86_64-linux",
            'factory': SanitizerBuilder.getSanitizerBuildFactory()},
 
def get_builders():
     for b in _get_lld_builders():
         b['category'] = 'lld'
         yield b
 
     for b in _get_lldb_builders():
         b['category'] = 'lldb'
         yield b
 
+    for b in _get_llgo_builders():
+        b['category'] = 'llgo'
+        yield b
+
     for b in _get_sanitizer_builders():
         b['category'] = 'sanitizer'
         yield b
 
     for b in _get_openmp_builders():
         b['category'] = 'openmp'
         yield b
 

buildbot/osuosl/master/master.cfg

c['change_source'].append(LLVMPoller(projects=[
     "clang-tests-external",
     "clang-tools-extra",
     "polly",
     "compiler-rt",
     "libcxx",
     "libcxxabi",
     "lld",
     "lldb",
+    "llgo",
     "openmp"]))
 
 #  c['change_source'].append(LLVMPoller("test-suite", "trunk"))
 
 ####### RESOURCE USAGE
 
 # Number of build records to maintain per builder.
 c['buildHorizon'] = 200

 
 c['schedulers'].append(SingleBranchScheduler(name="libcxx_scheduler",
                             treeStableTimer=2*60,
                             builderNames=get_all_for("libcxx"),
                             change_filter=depends_on([
                                 "libcxx",
                                 "libcxxabi"])))
 
+c['schedulers'].append(SingleBranchScheduler(name="llgo_scheduler",
+                            treeStableTimer=2*60,
+                            builderNames=get_all_for("llgo"),
+                            change_filter=depends_on([
+                                "llvm",
+                                "cfe",
+                                "llgo"])))
+
 ####### PROJECT IDENTITY
 
 c['title'] = "LLVM"
 c['titleURL'] = "http://llvm.org/"
 c['buildbotURL'] = "http://lab.llvm.org:8011/"
 
 ####### DB URL
 
 c['db_url'] = "postgresql://localhost/buildmasterdb"

zorg/buildbot/builders/LLGoBuilder.py

+import os
+
+import buildbot
+import buildbot.process.factory
+from buildbot.steps.source import SVN
+from buildbot.steps.shell import ShellCommand, SetProperty
+
+from zorg.buildbot.commands.NinjaCommand import NinjaCommand
+
+def getLLGoBuildFactory(
+            clean=True,
+            build_type='Release+Asserts',
+            test_libgo=True,             # run 'check-libgo' target if True
+    ):
+    llvm_srcdir = "llvm.src"
+    llvm_objdir = "llvm.obj"
+    llgo_srcdir = '%s/tools/llgo' % llvm_srcdir
+    clang_srcdir = '%s/tools/clang' % llvm_srcdir
+
+    f = buildbot.process.factory.BuildFactory()
+    # Determine the build directory.
+    f.addStep(SetProperty(name="get_builddir",
+                          command=["pwd"],
+                          property="builddir",
+                          description="set build dir",
+                          workdir="."))
+    # Get LLVM, clang and llgo
+    f.addStep(SVN(name='svn-llvm',
+                  mode='update',
+                  baseURL='http://llvm.org/svn/llvm-project/llvm/',
+                  defaultBranch='trunk',
+                  workdir=llvm_srcdir))
+    f.addStep(SVN(name='svn-clang',
+                  mode='update',
+                  baseURL='http://llvm.org/svn/llvm-project/cfe/',
+                  defaultBranch='trunk',
+                  workdir=clang_srcdir))
+    f.addStep(SVN(name='svn-llgo',
+                  mode='update',
+                  baseURL='http://llvm.org/svn/llvm-project/llgo/',
+                  defaultBranch='trunk',
+                  workdir=llgo_srcdir))
+
+    # Clean build directory, if requested.
+    f.addStep(ShellCommand(name="rm-llvm_objdir",
+                           command=["rm", "-rf", llvm_objdir],
+                           haltOnFailure=True,
+                           description=["rm build dir", "llvm"],
+                           workdir=".",
+                           doStepIf=clean))
+
+    # Create configuration files with cmake
+    f.addStep(ShellCommand(name="create-build-dir",
+                           command=["mkdir", "-p", llvm_objdir],
+                           haltOnFailure=False,
+                           description=["create build dir"],
+                           workdir="."))
+    cmakeCommand = [
+        "cmake", "-G", "Ninja",
+        "../%s" %llvm_srcdir,
+        "-DCMAKE_BUILD_TYPE=" + build_type,
+    ]
+    f.addStep(ShellCommand(name="cmake-configure",
+                           command=cmakeCommand,
+                           haltOnFailure=False,
+                           description=["cmake configure"],
+                           workdir=llvm_objdir))
+
+    # Build llgo
+    f.addStep(NinjaCommand(name="build_llgo",
+                           targets=["llgo"],
+                           haltOnFailure=True,
+                           description=["build llgo"],
+                           workdir=llvm_objdir))
+    # Test llgo
+    f.addStep(NinjaCommand(name="test_llgo",
+                           targets=["check-llgo"],
+                           haltOnFailure=True,
+                           description=["test llgo"],
+                           workdir=llvm_objdir))
+    # Test libgo
+    f.addStep(NinjaCommand(name="test_libgo",
+                           targets=["check-libgo"],
+                           haltOnFailure=True,
+                           description=["test libgo"],
+                           workdir=llvm_objdir))
+    return f
+

zorg/buildbot/changes/llvmpoller.py

def create_changes(self, new_logentries):
             'llvm'         : 'llvm',
             'cfe'          : 'clang',
             'polly'        : 'polly',
             'compiler-rt'  : 'compiler-rt',
             'libcxx'       : 'libcxx',
             'libcxxabi'    : 'libcxxabi',
             'lld'          : 'lld',
             'lldb'         : 'lldb',
+            'llgo'         : 'llgo',
             'openmp'       : 'openmp',
             }
 
         for el in new_logentries:
             revision = str(el.getAttribute("revision"))
 
             revlink = ''