[zorg] Add buildbot with reverse iteration enabled

[pzheng]

Summary

This reverse iteration buildbot builds LLVM, Clang and Polly with
LLVM_REVERSE_ITERATION enabled, and subsequently runs "make check-all". With
LLVM_REVERSE_ITERATION enabled, all supported unordered llvm containers would be
iterated in reverse order.

[mgrang]

Added Victor Leschuk as reviewer since Galina is on vacation.

[mgrang]

LGTM. However, I would wait for someone more aware of buildbots to +1 this.

[vleschuk]

buildbot/osuosl/master/config/builders.py

806

Why do you need to change this?

zorg/buildbot/builders/PollyBuilder.py

30

How are these changes related to adding reverse iterations builder?

[pzheng]

buildbot/osuosl/master/config/builders.py

806

This is actually a clean up for the PollyBuilder. The PollyBuilder has a "make polly-check-format" step which is no longer needed since "make check-polly" (or "make check-all" which we are adding) also includes it. Since we are reusing getPollyBuildFactory in the reverse iteration buildbot, it seems to be a good time to clean up the PollyBuiler at the same time.

zorg/buildbot/builders/PollyBuilder.py

30

We want to add a "make check-all" step in the reverse iterator buildbot, but only "make check-polly" is available in getPollyBuildFactory currently. Therefore we are introducing "checkAll" to control whether to run "make check-all" or "make check-polly". "checkAll" defaults to false which is the desired behavior for existing Polly builders. For the reverse iteration buildbot, "checkAll" is set to true to run "make check-all" instead of "make check-polly".

[grosser]

this lgtm from my side

buildbot/osuosl/master/config/builders.py

806

I am fine with this change!

[grosser]

@vleschuk is right that it is unclear why you change the Polly builder. I would suggest to motivate in the commit message that you modify the Polly builder to also include Polly into these builds!

[pzheng]

In D36203#829483, @grosser wrote:

@vleschuk is right that it is unclear why you change the Polly builder. I would suggest to motivate in the commit message that you modify the Polly builder to also include Polly into these builds!

Agree. I have updated the commit message to make it clear.

[grosser]

Cool, thanks!

[vleschuk]

buildbot/osuosl/master/config/builders.py

806

That's ok. But I'd prefer more granular commits. Could you please split changes into separate commits next time?

[grosser]

We could even split things in smaller commits this time, I assume. If there are concerns, I believe these should be addressed.

One could e.g.:

1. Remove formatting from the polly buildbot
2. Add then the new checkAll feature
3. Then add the new buildbot

And please excuse the "!", this slipped there by accident.

[grosser]

Very nice!

[pzheng]

@vleschuk, can you please update the buildbot master to make the changes live?

buildbot/osuosl/master/config/builders.py

return [
         {'name': "polly-arm-linux",
          'slavenames': ["hexagon-build-02", "hexagon-build-03"],
          'builddir': "polly-arm-linux",
          'factory': PollyBuilder.getPollyBuildFactory(
                 clean=True,
                 install=True,
                 make='ninja',
                 jobs=16,
-                checkFormat=False,
                 extraCmakeArgs=["-G", "Ninja",
                                 "-DLLVM_TARGETS_TO_BUILD='ARM;AArch64'",
                                 "-DLLVM_DEFAULT_TARGET_TRIPLE=arm-linux-gnueabi",
                                 "-DLLVM_TARGET_ARCH=arm-linux-gnueabi",
                                 "-DLLVM_ENABLE_ASSERTIONS=True",
                                 "-DCMAKE_C_COMPILER:FILEPATH=/local/clang+llvm-3.7.1-x86_64-linux-gnu-ubuntu-14.04/bin/clang",
                                 "-DCMAKE_CXX_COMPILER:FILEPATH=/local/clang+llvm-3.7.1-x86_64-linux-gnu-ubuntu-14.04/bin/clang++"])}
        ]
return [
                 jobs=16,
                 extra_make_args=None,
                 env={},
                 clean=False,
                 sync=False,
                 patch=None)}
     ]
 
+# Reverse iteration builders.
+def _get_rev_iter_builders():
+    return [
+        {'name': "reverse-iteration",
+         'slavenames': ["hexagon-build-03"],
+         'builddir': "reverse-iteration",
+         'factory': PollyBuilder.getPollyBuildFactory(
+                clean=True,
+                make='ninja',
+                jobs=16,
+                checkAll=True,
+                extraCmakeArgs=["-G", "Ninja",
+                                "-DLLVM_REVERSE_ITERATION:BOOL=ON",
+                                "-DLLVM_ENABLE_ASSERTIONS=True",
+                                "-DCMAKE_C_COMPILER:FILEPATH=/local/clang+llvm-3.7.1-x86_64-linux-gnu-ubuntu-14.04/bin/clang",
+                                "-DCMAKE_CXX_COMPILER:FILEPATH=/local/clang+llvm-3.7.1-x86_64-linux-gnu-ubuntu-14.04/bin/clang++"])}
+    ]
+
 # LLDB builders.
 def _get_lldb_builders():
     return [
         {'name': "lldb-x86-windows-msvc2015",
          'slavenames': ["zturner-win2008"],
          'builddir': "lldb-windows-x86",
          'factory': LLDBBuilder.getLLDBWindowsCMakeBuildFactory(test=False)},
         # Disable the builder till we fix the cmake configuration
def get_builders():
     for b in _get_polly_builders():
         b['category'] = 'polly'
         yield b
 
     for b in _get_aosp_builders():
         b['category'] = 'aosp'
         yield b
 
+    for b in _get_rev_iter_builders():
+        b['category'] = 'rev_iter'
+        yield b
+
     for b in _get_lld_builders():
         b['category'] = 'lld'
         yield b
 
     for b in _get_lldb_builders():
         b['category'] = 'lldb'
         yield b
 

buildbot/osuosl/master/config/status.py

return [
             mode = "failing",
             builders = ["polly-arm-linux",
                         "aosp-O3-polly-before-vectorizer-unprofitable"],
             addLogs=False,
             num_lines = 15),
         InformativeMailNotifier(
             fromaddr = "[e-mail]",
             sendToInterestedUsers= False,
+            extraRecipients = ["[e-mail]"],
+            subject="Build %(builder)s Failure",
+            mode = "failing",
+            builders = ["reverse-iteration"],
+            addLogs=False,
+            num_lines = 15),
+        InformativeMailNotifier(
+            fromaddr = "[e-mail]",
+            sendToInterestedUsers= False,
             extraRecipients = ["[e-mail]"],
             subject="Build %(builder)s Failure",
             mode = "failing",
             builders = ["clang-cuda-build"],
             addLogs=False,
             num_lines = 15),
         InformativeMailNotifier(
             fromaddr = "[e-mail]",

buildbot/osuosl/master/master.cfg

c['schedulers'].append(SingleBranchScheduler(name="llgo_scheduler",
                                 "llgo"])))
 
 c['schedulers'].append(Nightly(name="aosp_scheduler",
                                builderNames=get_all_for("aosp"),
                                hour=19,
                                minute=0,
                                branch=None))
 
+c['schedulers'].append(Nightly(name="rev_iter_scheduler",
+                               builderNames=get_all_for("rev_iter"),
+                               hour=18,
+                               minute=0,
+                               branch=None))
+
 c['schedulers'].extend(
     config.schedulers.getSingleBranchSchedulers(c['builders'], c['schedulers'])
     )
 
 ####### PROJECT IDENTITY
 
 c['title'] = "LLVM"
 c['titleURL'] = "http://llvm.org/"
 c['buildbotURL'] = "http://lab.llvm.org:8011/"
 
 ####### DB URL
 
 c['db_url'] = "postgresql://localhost/buildmasterdb"

zorg/buildbot/builders/AOSPBuilder.py

def getAOSPBuildFactory(
     clang_dir = target_clang
 
     # Build Clang for AOSP target build
     if build_clang:
         f = PollyBuilder.getPollyBuildFactory(clean=True,
                                               install=True,
                                               make='ninja',
                                               jobs=jobs,
-                                              checkFormat=False,
                                               extraCmakeArgs=extra_cmake_args)
         clang_dir = 'llvm.inst/bin'
 
     # Restore AOSP repo to a clean state
     if clean:
         f.addStep(ShellCommand(name="clean-repo",
                                command=['repo', 'forall', '-c',
                                         'git reset --hard; git clean -fdx'],

zorg/buildbot/builders/PollyBuilder.py

 import os
 
 import buildbot
 import buildbot.process.factory
 from buildbot.steps.source import SVN, Git
 from buildbot.steps.shell import Configure, ShellCommand
 from buildbot.process.properties import WithProperties
 
 from zorg.buildbot.builders import LNTBuilder
 from zorg.buildbot.builders import ClangBuilder
 
 def getPollyBuildFactory(
     clean=False,
     install=False,
     make='make',
     jobs=None,
-    checkFormat=True,
+    checkAll=False,
     extraCmakeArgs=[]):
     llvm_srcdir = "llvm.src"
     llvm_objdir = "llvm.obj"
     llvm_instdir = "llvm.inst"
     polly_srcdir = '%s/tools/polly' % llvm_srcdir
     clang_srcdir = '%s/tools/clang' % llvm_srcdir
     jobs_cmd = []
     if jobs is not None:
         jobs_cmd = ["-j"+str(jobs)]
     build_cmd = [make] + jobs_cmd
     install_cmd = [make, 'install'] + jobs_cmd
-    check_cmd = [make, 'check-polly'] + jobs_cmd
-    check_format_cmd = [make, 'polly-check-format'] + jobs_cmd
+    check_all_cmd = [make, 'check-all'] + jobs_cmd
+    check_polly_cmd = [make, 'check-polly'] + jobs_cmd
     cmake_install = []
     if install:
         cmake_install = ["-DCMAKE_INSTALL_PREFIX=../%s" % llvm_instdir]
 
     f = buildbot.process.factory.BuildFactory()
     # Determine the build directory.
     f.addStep(buildbot.steps.shell.SetProperty(name="get_builddir",
                                                command=["pwd"],
def getPollyBuildFactory(
     # Install
     if install:
         f.addStep(ShellCommand(name="install",
                                command=install_cmd,
                                haltOnFailure=False,
                                description=["install"],
                                workdir=llvm_objdir))
 
-    # Test Polly
-    f.addStep(ShellCommand(name="test_polly",
-                           command=check_cmd,
-                           haltOnFailure=True,
-                           description=["test polly"],
+    # Test
+    if checkAll:
+        f.addStep(ShellCommand(name="check_all",
+                               command=check_all_cmd,
+                               haltOnFailure=False,
+                               description=["check all"],
                                workdir=llvm_objdir))
-
-    # Check formatting
-    if checkFormat:
-        f.addStep(ShellCommand(name="test_polly_format",
-                               command=check_format_cmd,
+    else:
+        f.addStep(ShellCommand(name="check_polly",
+                               command=check_polly_cmd,
                                haltOnFailure=False,
-                               description=["check formatting"],
+                               description=["check polly"],
                                workdir=llvm_objdir))
+
     return f
 
 def AddExternalPollyBuildFactory(f, llvm_installdir, build_type = "Release"):
     polly_srcdir = 'polly.src'
     polly_objdir = 'polly.obj'
     polly_installdir = 'polly.install'
 
     # Determine the build directory.