[zorg] Add buildbot with reverse iteration enabled

[pzheng]

Summary

This reverse iteration buildbot builds LLVM, Clang and Polly with
LLVM_REVERSE_ITERATION enabled, and subsequently runs "make check-all". With
LLVM_REVERSE_ITERATION enabled, all supported unordered llvm containers would be
iterated in reverse order.

[mgrang]

Added Victor Leschuk as reviewer since Galina is on vacation.

[mgrang]

LGTM. However, I would wait for someone more aware of buildbots to +1 this.

[vleschuk]

buildbot/osuosl/master/config/builders.py

805–806

Why do you need to change this?

zorg/buildbot/builders/PollyBuilder.py

30 ↗

(On Diff #109256)

How are these changes related to adding reverse iterations builder?

[pzheng]

buildbot/osuosl/master/config/builders.py

805–806

This is actually a clean up for the PollyBuilder. The PollyBuilder has a "make polly-check-format" step which is no longer needed since "make check-polly" (or "make check-all" which we are adding) also includes it. Since we are reusing getPollyBuildFactory in the reverse iteration buildbot, it seems to be a good time to clean up the PollyBuiler at the same time.

zorg/buildbot/builders/PollyBuilder.py

30 ↗

(On Diff #109256)

We want to add a "make check-all" step in the reverse iterator buildbot, but only "make check-polly" is available in getPollyBuildFactory currently. Therefore we are introducing "checkAll" to control whether to run "make check-all" or "make check-polly". "checkAll" defaults to false which is the desired behavior for existing Polly builders. For the reverse iteration buildbot, "checkAll" is set to true to run "make check-all" instead of "make check-polly".

[grosser]

this lgtm from my side

buildbot/osuosl/master/config/builders.py

805–806

I am fine with this change!

[grosser]

@vleschuk is right that it is unclear why you change the Polly builder. I would suggest to motivate in the commit message that you modify the Polly builder to also include Polly into these builds!

[pzheng]

In D36203#829483, @grosser wrote:

@vleschuk is right that it is unclear why you change the Polly builder. I would suggest to motivate in the commit message that you modify the Polly builder to also include Polly into these builds!

Agree. I have updated the commit message to make it clear.

[grosser]

Cool, thanks!

[vleschuk]

buildbot/osuosl/master/config/builders.py

805–806

That's ok. But I'd prefer more granular commits. Could you please split changes into separate commits next time?

[grosser]

We could even split things in smaller commits this time, I assume. If there are concerns, I believe these should be addressed.

One could e.g.:

1. Remove formatting from the polly buildbot
2. Add then the new checkAll feature
3. Then add the new buildbot

And please excuse the "!", this slipped there by accident.

[grosser]

Very nice!

[pzheng]

@vleschuk, can you please update the buildbot master to make the changes live?

buildbot/osuosl/master/config/builders.py

return [
 
         {'name': "polly-arm-linux",
          'slavenames': ["hexagon-build-02", "hexagon-build-03"],
          'builddir': "polly-arm-linux",
          'factory': PollyBuilder.getPollyBuildFactory(
                 clean=True,
                 install=True,
                 make='ninja',
                 jobs=16,
                 extraCmakeArgs=["-G", "Ninja",
                                 "-DLLVM_TARGETS_TO_BUILD='ARM;AArch64'",
                                 "-DLLVM_DEFAULT_TARGET_TRIPLE=arm-linux-gnueabi",
                                 "-DLLVM_TARGET_ARCH=arm-linux-gnueabi",
                                 "-DLLVM_ENABLE_ASSERTIONS=True",
                                 "-DCMAKE_C_COMPILER:FILEPATH=/local/clang+llvm-3.7.1-x86_64-linux-gnu-ubuntu-14.04/bin/clang",
                                 "-DCMAKE_CXX_COMPILER:FILEPATH=/local/clang+llvm-3.7.1-x86_64-linux-gnu-ubuntu-14.04/bin/clang++"])}
        ]
 
return [
                 jobs=16,
                 extra_make_args=None,
                 env={},
                 clean=False,
                 sync=False,
                 patch=None)}
     ]
 
+# Reverse iteration builders.
+def _get_rev_iter_builders():
+    return [
+        {'name': "reverse-iteration",
+         'slavenames': ["hexagon-build-03"],
+         'builddir': "reverse-iteration",
+         'factory': PollyBuilder.getPollyBuildFactory(
+                clean=True,
+                make='ninja',
+                jobs=16,
+                checkAll=True,
+                extraCmakeArgs=["-G", "Ninja",
+                                "-DLLVM_REVERSE_ITERATION:BOOL=ON",
+                                "-DLLVM_ENABLE_ASSERTIONS=True",
+                                "-DCMAKE_C_COMPILER:FILEPATH=/local/clang+llvm-3.7.1-x86_64-linux-gnu-ubuntu-14.04/bin/clang",
+                                "-DCMAKE_CXX_COMPILER:FILEPATH=/local/clang+llvm-3.7.1-x86_64-linux-gnu-ubuntu-14.04/bin/clang++"])}
+    ]
+
 # LLDB builders.
 def _get_lldb_builders():
     return [
         {'name': "lldb-x86-windows-msvc2015",
          'slavenames': ["zturner-win2008"],
          'builddir': "lldb-windows-x86",
          'factory': LLDBBuilder.getLLDBWindowsCMakeBuildFactory(test=False)},
         # Disable the builder till we fix the cmake configuration
def get_builders():
     for b in _get_polly_builders():
         b['category'] = 'polly'
         yield b
 
     for b in _get_aosp_builders():
         b['category'] = 'aosp'
         yield b
 
+    for b in _get_rev_iter_builders():
+        b['category'] = 'rev_iter'
+        yield b
+
     for b in _get_lld_builders():
         b['category'] = 'lld'
         yield b
 
     for b in _get_lldb_builders():
         b['category'] = 'lldb'
         yield b
 

buildbot/osuosl/master/config/status.py

return [
             mode = "failing",
             builders = ["polly-arm-linux",
                         "aosp-O3-polly-before-vectorizer-unprofitable"],
             addLogs=False,
             num_lines = 15),
         InformativeMailNotifier(
             fromaddr = "[e-mail]",
             sendToInterestedUsers= False,
+            extraRecipients = ["[e-mail]"],
+            subject="Build %(builder)s Failure",
+            mode = "failing",
+            builders = ["reverse-iteration"],
+            addLogs=False,
+            num_lines = 15),
+        InformativeMailNotifier(
+            fromaddr = "[e-mail]",
+            sendToInterestedUsers= False,
             extraRecipients = ["[e-mail]"],
             subject="Build %(builder)s Failure",
             mode = "failing",
             builders = ["clang-cuda-build"],
             addLogs=False,
             num_lines = 15),
         InformativeMailNotifier(
             fromaddr = "[e-mail]",

buildbot/osuosl/master/master.cfg

c['schedulers'].append(SingleBranchScheduler(name="llgo_scheduler",
                                 "llgo"])))
 
 c['schedulers'].append(Nightly(name="aosp_scheduler",
                                builderNames=get_all_for("aosp"),
                                hour=19,
                                minute=0,
                                branch=None))
 
+c['schedulers'].append(Nightly(name="rev_iter_scheduler",
+                               builderNames=get_all_for("rev_iter"),
+                               hour=18,
+                               minute=0,
+                               branch=None))
+
 c['schedulers'].extend(
     config.schedulers.getSingleBranchSchedulers(c['builders'], c['schedulers'])
     )
 
 ####### PROJECT IDENTITY
 
 c['title'] = "LLVM"
 c['titleURL'] = "http://llvm.org/"
 c['buildbotURL'] = "http://lab.llvm.org:8011/"
 
 ####### DB URL
 
 c['db_url'] = "postgresql://localhost/buildmasterdb"