[zorg] Add Polly builder for ARM target

[grosser]

Very nice!

Some small questions from my side. What is different in the getPollyARMBuildFactory compared to getPollyBuildFactory? In the optimal case, we just have one build factory that is parametrized to the specific builder, no? Maybe some of the changes could be moved to the getPollyBuildFactory that all Polly builders can benefit. We could e.g. make the 'make-command' just a parameter, which is set to 'ninja' for the ARM builder. Also, while at it, I would not call the build command 'Build Polly', but rather 'Build or Build+LLVM'. The reason is that this will come up in the error messages when a build fails. In many cases such failures are due to errors in LLVM, but if the name implies this is Polly-only people might skip those mails by accident.

[rengolin]

Some comments on the implementation, but I share Tobias's concerns. Why not re-use the existing Poly builder?

buildbot/osuosl/master/config/status.py

174

You don't need to add this to new builders.

zorg/buildbot/builders/PollyBuilder.py

222

Paths should be configuration parameters (see the ARM buildbots)

257

Targets and triples should be configuration parameters (see the ARM buildbots)

273

Number of jobs should be a configuration parameter (see the ARM buildbots)

288

Number of jobs should be a configuration parameter (see the ARM buildbots)

295

Number of jobs should be a configuration parameter (see the ARM buildbots)

[pzheng]

Thanks for reviewing the patch!
I have refactored getPollyBuildFactory to take a few additional arguments and got rid of getPollyARMBuildFactory so that the builder for ARM also uses getPollyBuildFactory. Also addressed a few other comments from grosser and rengolin.

[pzheng]

buildbot/osuosl/master/config/status.py

174

We want to have someone monitor the builder, thus adding this extra stuff.

[grosser]

Thank you!

[rengolin]

buildbot/osuosl/master/config/status.py

174

This should really pick up from the bot owner's email, not this hard-coded address. But that's not for this patch.

zorg/buildbot/builders/PollyBuilder.py

100

Why do you need the install dir if you're not using it in the buildbot?

[pzheng]

zorg/buildbot/builders/PollyBuilder.py

100

The install dir will be used by another builder. More patches coming soon.

buildbot/osuosl/master/config/builders.py

return [
          'slavenames':["pollyperf15"],
          'builddir':"perf-x86_64-penryn-O3-polly-before-vectorizer-detect-only",
          'factory': PollyBuilder.getPollyLNTFactory(triple="x86_64-pc-linux-gnu",
                                                     #nt_flags=['--multisample=10', '--mllvm=-polly', '--rerun'],
                                                     nt_flags=['--multisample=10', '--mllvm=-polly', '--mllvm=-polly-position=before-vectorizer', '--mllvm=-polly-only-scop-detection'],
                                                     reportBuildslave=False,
                                                     package_cache="http://parkas1.inria.fr/packages",
                                                     submitURL=['http://gcc12.fsffrance.org:8808/submitRun','http://llvm.org/perf/submitRun'],
-                                                    testerName='x86_64-penryn-O3-polly-before-vectorizer-detect-only')}
+                                                    testerName='x86_64-penryn-O3-polly-before-vectorizer-detect-only')},
+
+        {'name': "polly-arm-linux",
+         'slavenames': ["hexagon-build-02", "hexagon-build-03"],
+         'builddir': "polly-arm-linux",
+         'factory': PollyBuilder.getPollyBuildFactory(
+                clean=True,
+                install=True,
+                make='ninja',
+                jobs=16,
+                checkFormat=False,
+                extraCmakeArgs=["-G", "Ninja",
+                                "-DLLVM_TARGETS_TO_BUILD='ARM;AArch64'",
+                                "-DLLVM_DEFAULT_TARGET_TRIPLE=arm-linux-gnueabi",
+                                "-DLLVM_TARGET_ARCH=arm-linux-gnueabi",
+                                "-DLLVM_ENABLE_ASSERTIONS=True",
+                                "-DCMAKE_C_COMPILER:FILEPATH=/local/clang+llvm-3.7.1-x86_64-linux-gnu-ubuntu-14.04/bin/clang",
+                                "-DCMAKE_CXX_COMPILER:FILEPATH=/local/clang+llvm-3.7.1-x86_64-linux-gnu-ubuntu-14.04/bin/clang++"])}
        ]
 
 # LLDB builders.
 def _get_lldb_builders():
     return [
         {'name': "lldb-x86-windows-msvc2015",
          'slavenames': ["zturner-win2008"],
          'builddir': "lldb-windows-x86",

buildbot/osuosl/master/config/status.py

return [
             mode = "failing",
             builders = ["lld-x86_64-win7", "clang-x86_64-linux-abi-test",
                         "lld-x86_64-freebsd", "sanitizer_x86_64-freebsd",
                         "lld-x86_64-darwin13", "clang-3stage-ubuntu",
                         "llvm-clang-lld-x86_64-scei-ps4-ubuntu-fast",
                         "llvm-clang-lld-x86_64-scei-ps4-windows10pro-fast",
                         "clang-with-lto-ubuntu"],
             addLogs=False),
+        InformativeMailNotifier(
+            fromaddr = "[e-mail]",
+            sendToInterestedUsers= False,
+            extraRecipients = ["[e-mail]"],
+            subject="Build %(builder)s Failure",
+            mode = "failing",
+            builders = ["polly-arm-linux"],
+            addLogs=False,
+            num_lines = 15),
         ]

zorg/buildbot/builders/PollyBuilder.py

 import os
 
 import buildbot
 import buildbot.process.factory
 from buildbot.steps.source import SVN, Git
 from buildbot.steps.shell import Configure, ShellCommand
 from buildbot.process.properties import WithProperties
 
 from zorg.buildbot.builders import LNTBuilder
 from zorg.buildbot.builders import ClangBuilder
 
-def getPollyBuildFactory():
+def getPollyBuildFactory(
+    clean=False,
+    install=False,
+    make='make',
+    jobs=None,
+    checkFormat=True,
+    extraCmakeArgs=[]):
     llvm_srcdir = "llvm.src"
     llvm_objdir = "llvm.obj"
+    llvm_instdir = "llvm.inst"
     polly_srcdir = '%s/tools/polly' % llvm_srcdir
     clang_srcdir = '%s/tools/clang' % llvm_srcdir
+    jobs_cmd = []
+    if jobs is not None:
+        jobs_cmd = ["-j"+str(jobs)]
+    build_cmd = [make] + jobs_cmd
+    install_cmd = [make, 'install'] + jobs_cmd
+    check_cmd = [make, 'check-polly'] + jobs_cmd
+    check_format_cmd = [make, 'polly-check-format'] + jobs_cmd
+    cmake_install = []
+    if install:
+        cmake_install = ["-DCMAKE_INSTALL_PREFIX=../%s" % llvm_instdir]
 
     f = buildbot.process.factory.BuildFactory()
     # Determine the build directory.
     f.addStep(buildbot.steps.shell.SetProperty(name="get_builddir",
                                                command=["pwd"],
                                                property="builddir",
                                                description="set build dir",
                                                workdir="."))
-    # Get LLVM, clang and Polly
+    # Get LLVM, Clang and Polly
     f.addStep(SVN(name='svn-llvm',
                   mode='update',
                   baseURL='http://llvm.org/svn/llvm-project/llvm/',
                   defaultBranch='trunk',
                   workdir=llvm_srcdir))
     f.addStep(SVN(name='svn-clang',
                   mode='update',
                   baseURL='http://llvm.org/svn/llvm-project/cfe/',
                   defaultBranch='trunk',
                   workdir=clang_srcdir))
     f.addStep(SVN(name='svn-polly',
                   mode='update',
                   baseURL='http://llvm.org/svn/llvm-project/polly/',
                   defaultBranch='trunk',
                   workdir=polly_srcdir))
 
+    # Clean build dir
+    if clean:
+        f.addStep(ShellCommand(name='clean-build-dir',
+                               command=['rm', '-rf', llvm_objdir],
+                               warnOnFailure=True,
+                               description=["clean build dir"],
+                               workdir='.'))
+
     # Create configuration files with cmake
     f.addStep(ShellCommand(name="create-build-dir",
-                               command=["mkdir", "-p", llvm_objdir],
-                               haltOnFailure=False,
-                               description=["create build dir"],
-                               workdir="."))
     cmakeCommand = ["cmake", "../%s" %llvm_srcdir,
-        "-DCMAKE_COLOR_MAKEFILE=OFF", "-DPOLLY_TEST_DISABLE_BAR=ON",
-        "-DCMAKE_BUILD_TYPE=Release"]
-
+                    "-DCMAKE_COLOR_MAKEFILE=OFF",
+                    "-DPOLLY_TEST_DISABLE_BAR=ON",
+                    "-DCMAKE_BUILD_TYPE=Release"] + cmake_install + extraCmakeArgs
     f.addStep(ShellCommand(name="cmake-configure",
-                               command=cmakeCommand,
-                               haltOnFailure=False,
-                               description=["cmake configure"],
-                               workdir=llvm_objdir))
-    # Build Polly
-    f.addStep(ShellCommand(name="build_polly",
-                               command=["make"],
-                               haltOnFailure=True,
-                               description=["build polly"],
-                               workdir=llvm_objdir))
+
+    # Build
+    f.addStep(ShellCommand(name="build",
+                           command=build_cmd,
+                           description=["build"],
+
+    # Clean install dir
+    if install and clean:
+        f.addStep(ShellCommand(name='clean-install-dir',
+                               command=['rm', '-rf', llvm_instdir],
+                               haltOnFailure=False,
+                               description=["clean install dir"],
+                               workdir='.'))
+
+    # Install
+    if install:
+        f.addStep(ShellCommand(name="install",
+                               command=install_cmd,
+                               haltOnFailure=False,
+                               description=["install"],
+                               workdir=llvm_objdir))
+
     # Test Polly
     f.addStep(ShellCommand(name="test_polly",
-                               command=["make", "polly-test"],
-                               haltOnFailure=True,
-                               description=["test polly"],
-                               workdir=llvm_objdir))
+                           command=check_cmd,
+
     # Check formatting
+    if checkFormat:
         f.addStep(ShellCommand(name="test_polly_format",
-                               command=["make", "polly-check-format"],
+                               command=check_format_cmd,
                                haltOnFailure=False,
-                               description=["Check formatting"],
+                               description=["check formatting"],
                                workdir=llvm_objdir))
     return f
 
 def AddExternalPollyBuildFactory(f, llvm_installdir, build_type = "Release"):
     polly_srcdir = 'polly.src'
     polly_objdir = 'polly.obj'
     polly_installdir = 'polly.install'
 
nt_flags.append(WithProperties("--cflag=%s/polly.install/lib/LLVMPolly.so",
                                    'builddir'))
 
     # Add an LNT test runner.
     LNTBuilder.AddLNTTestsToFactory(f, nt_flags,
                                     cc_path=(llvm_install_dir+'/bin/clang'),
                                     cxx_path=(llvm_install_dir+'/bin/clang++'),
                                     **lnt_args);
 
     return f