[zorg] Add polly test-suite builder.

[Meinersbur]

Summary

Add a polly build for compiling and running the test-suite. AFAIK, this will be the only builder running the test-suite on x86. I had to fix a few problems to make the test-suite run smoothly again.

An experimental version is current running at http://meinersbur.de:8011/builders/polly-x86_64-linux-test-suite .

[Meinersbur]

Note that I also changed some step in the current Polly builders:

• clean-dir: RemoveDirectory
• compile: WarningCountingShellCommand
• check: LitTestCommand

These more specific step commands parse the output to display on buildbot's build overview page.

[Meinersbur]

• Use zorg's method to checkout test-suite source

[Meinersbur]

ping

[gkistanova]

Please see inline comments.

zorg/buildbot/builders/PollyBuilder.py

17

Is the new line is missing between the arguments?

66

You do not need a step to get the builddir. This is workdir property, just use that. For example,

WithProperties("%(workdir)s/" + llvm_instdir + "/bin/clang++")

would give you a fully qualified path to the installed clang++.

106

This code will be running on the master. The path on master has nothing to do with the path on the bot which will be actually building. Here and in thew rest of similar places, please use forward slashes here as it would work everywhere.

176

Adding -vv lit flag might help troubleshooting test failures.

[Meinersbur]

• Address Galina's review

[Meinersbur]

zorg/buildbot/builders/PollyBuilder.py

66

I copied the get_builddir step from ClangBuilder. Thanks for the hint.

It is/was wrapped into WithProperties when the step was added.

106

llvm_objdir is set to "llvm.obj" at the beginning of this function, i.e. this is a path relative to the workdir on slave, as it was used already by all the other steps. I changed it to string concatenation as suggested.

Personally, I think os.path.join more idiomatic as it e.g. also works on Windows and if any argument is an absolute path.

176

-vv only adds additional output when execute_external is used, which seems only true for ABI-Testsuite.

[gkistanova]

LGTM

buildbot/osuosl/master/config/builders.py

return [
                                 "-DLLVM_TARGETS_TO_BUILD='ARM;AArch64'",
                                 "-DLLVM_DEFAULT_TARGET_TRIPLE=arm-linux-gnueabi",
                                 "-DLLVM_TARGET_ARCH=arm-linux-gnueabi",
                                 "-DLLVM_ENABLE_ASSERTIONS=True",
                                 "-DLLVM_ENABLE_LIBCXX:BOOL=ON",
                                 "-DCMAKE_C_COMPILER:FILEPATH=/local/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-14.04/bin/clang",
                                 "-DCMAKE_CXX_COMPILER:FILEPATH=/local/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-14.04/bin/clang++"])},
         {'name': "polly-x86_64-linux",
-         'slavenames': ["polly-x86_64-fdcserver", "polly-x86_64-gce1", "polly-x86_64-gce2"],
+         'slavenames': ["polly-x86_64-fdcserver", "polly-x86_64-gce1"],
          'builddir': "polly-x86_64-linux",
          'factory': PollyBuilder.getPollyBuildFactory(
                 clean=False,
                 install=False,
                 make='ninja',
                 extraCmakeArgs=["-G", "Ninja",
                                 "-DLLVM_ENABLE_ASSERTIONS=True",
                                 "-DLLVM_TARGETS_TO_BUILD='X86;NVPTX'",
                                 "-DCLANG_ENABLE_ARCMT=OFF",
                                 "-DCLANG_ENABLE_STATIC_ANALYZER=OFF",
-                                ])}
+                                ])},
+        {'name': "polly-x86_64-linux-test-suite",
+         'slavenames': ["polly-x86_64-fdcserver", "polly-x86_64-gce2"],
+         'builddir':"polly-x86_64-linux-test-suite",
+         'factory': PollyBuilder.getPollyBuildFactory(
+                clean=False,
+                install=False,
+                make='ninja',
+                extraCmakeArgs=["-G", "Ninja",
+                                "-DLLVM_ENABLE_ASSERTIONS=True",
+                                "-DLLVM_TARGETS_TO_BUILD='X86;NVPTX'",
+                                "-DCLANG_ENABLE_ARCMT=OFF",
+                                "-DCLANG_ENABLE_STATIC_ANALYZER=OFF",
+                                ],
+                testsuite=True,
+                extraTestsuiteCmakeArgs=["-G", "Ninja"])}
        ]
 
 # AOSP builders.
 def _get_aosp_builders():
     return [
         {'name': "aosp-O3-polly-before-vectorizer-unprofitable",
          'slavenames': ["hexagon-build-03"],
          'builddir': "aosp",

zorg/buildbot/builders/PollyBuilder.py

-from buildbot.steps.shell import Configure, ShellCommand
-from buildbot.process.properties import WithProperties
+import os
 
-from zorg.buildbot.builders import LNTBuilder
-from zorg.buildbot.builders import ClangBuilder
+from buildbot.process.properties import WithProperties
+from buildbot.steps.shell import Configure, ShellCommand, WarningCountingShellCommand, SetProperty
+from buildbot.steps.slave import RemoveDirectory
+from zorg.buildbot.commands.LitTestCommand import LitTestCommand
 from zorg.buildbot.process.factory import LLVMBuildFactory
 
 def getPollyBuildFactory(
     clean=False,
     install=False,
     make='make',
     jobs=None,
     checkAll=False,
     env=None,
     extraCmakeArgs=None,
+    testsuite=False,
+    extraTestsuiteCmakeArgs=None,
     **kwargs):
 
     if extraCmakeArgs is None:
-        extraCmakeArgs=[],
+        extraCmakeArgs=[]
+    if extraTestsuiteCmakeArgs is None:
+        extraTestsuiteCmakeArgs = []
+
     llvm_srcdir = "llvm.src"
     llvm_objdir = "llvm.obj"
     llvm_instdir = "llvm.inst"
+    testsuite_srcdir = "test-suite.src"
+    testsuite_builddir = "test-suite.build"
+
     jobs_cmd = []
     if jobs is not None:
         jobs_cmd = ["-j"+str(jobs)]
     build_cmd = [make] + jobs_cmd
     install_cmd = [make, 'install'] + jobs_cmd
     check_all_cmd = [make, 'check-all'] + jobs_cmd
     check_polly_cmd = [make, 'check-polly'] + jobs_cmd
     cmake_install = []
     if install:
         cmake_install = ["-DCMAKE_INSTALL_PREFIX=../%s" % llvm_instdir]
     # Prepare environmental variables. Set here all env we want everywhere.
     merged_env = {
                    'TERM' : 'dumb'     # Make sure Clang doesn't use color escape sequences.
                  }
     if env:
         merged_env.update(env)  # Overwrite pre-set items with the given ones, so user can set anything.
 
     depends_on_projects = ['llvm','clang','polly']
+    if testsuite:
+        # XRay tests in test-suite require compiler-rt.
+        depends_on_projects += ['compiler-rt']
 
     cleanBuildRequestedByProperty = lambda step: step.build.getProperty("clean", False)
     cleanBuildRequested = lambda step: clean or step.build.getProperty("clean", default=step.build.getProperty("clean_obj"))
 
     f = LLVMBuildFactory(
             depends_on_projects=depends_on_projects,
             llvm_srcdir=llvm_srcdir,
             obj_dir=llvm_objdir,
             install_dir=llvm_instdir,
             cleanBuildRequested=cleanBuildRequested,
             **kwargs) # Pass through all the extra arguments.
 
-    f.addStep(ShellCommand(name='clean-src-dir',
-                           command=['rm', '-rf', f.monorepo_dir],
+    f.addStep(RemoveDirectory(name='clean-src-dir',
+                           dir=f.monorepo_dir,
                            warnOnFailure=True,
-                           description=["clean src dir"],
-                           workdir='.',
-                           env=merged_env,
                            doStepIf=cleanBuildRequestedByProperty))
 
     # Get the source code.
     f.addGetSourcecodeSteps(**kwargs)
 
     # Clean build dir
     f.addStep(ShellCommand(name='clean-build-dir',
                            command=['rm', '-rf', llvm_objdir],
def getPollyBuildFactory(
     f.addStep(ShellCommand(name="cmake-configure",
                            command=cmakeCommand,
                            haltOnFailure=False,
                            description=["cmake configure"],
                            workdir=llvm_objdir,
                            env=merged_env))
 
     # Build
-    f.addStep(ShellCommand(name="build",
+    f.addStep(WarningCountingShellCommand(name="build",
                            command=build_cmd,
                            haltOnFailure=True,
                            description=["build"],
                            workdir=llvm_objdir,
                            env=merged_env))
 
+    clangexe   = os.path.join(llvm_objdir, 'bin', 'clang')
+    clangxxexe = os.path.join(llvm_objdir, 'bin', 'clang++')
+    litexe = os.path.join(llvm_objdir, 'bin', 'llvm-lit')
+    sizeexe = os.path.join(llvm_objdir, 'bin', 'llvm-size')
+
     # Clean install dir
     if install:
         f.addStep(ShellCommand(name='clean-install-dir',
                                command=['rm', '-rf', llvm_instdir],
                                haltOnFailure=False,
                                description=["clean install dir"],
                                workdir='.',
                                env=merged_env,
                                doStepIf=cleanBuildRequested))
 
         f.addStep(ShellCommand(name="install",
                                command=install_cmd,
-                               haltOnFailure=False,
+                               haltOnFailure=True,
                                description=["install"],
                                workdir=llvm_objdir,
                                env=merged_env))
 
+        # Use the installed version of clang
+        clangexe   = os.path.join(llvm_instdir, 'bin', 'clang')
+        clangxxexe = os.path.join(llvm_instdir, 'bin', 'clang++')       
+        sizeexe = os.path.join(llvm_instdir, 'bin', 'llvm-size')              
+
     # Test
     if checkAll:
-        f.addStep(ShellCommand(name="check_all",
+        f.addStep(LitTestCommand(name="check_all",
                                command=check_all_cmd,
                                haltOnFailure=False,
                                description=["check all"],
                                workdir=llvm_objdir,
                                env=merged_env))
     else:
-        f.addStep(ShellCommand(name="check_polly",
+        f.addStep(LitTestCommand(name="check_polly",
                                command=check_polly_cmd,
                                haltOnFailure=False,
                                description=["check polly"],
                                workdir=llvm_objdir,
                                env=merged_env))
 
+    if testsuite:
+        f.addStep(SetProperty(name="get_builddir",
+                command=["pwd"],
+                property="builddir",
+                description="set build dir",
+                workdir=".",
+                env=merged_env))
+
+        f.addStep(RemoveDirectory(name='test-suite_clean-src-dir',
+                           dir=testsuite_srcdir,
+                           haltOnFailure=False,
+                           warnOnFailure=True,
+                           doStepIf=cleanBuildRequestedByProperty))
+
+        # Does not work reliably: still assues that the monorepo and test-suite repo share that same revision identifiers
+        #f.addGetSourcecodeForProject(
+        #    project='test-suite',
+        #    src_dir=testsuite_srcdir,
+        #    alwaysUseLatest=True)
+
+        f.addStep(ShellCommand(name='test-suite_git-clone',
+                           description=["Test-Suite: git clone (will predictably fail if initialized in a previous build)"],
+                           command=['git', 'clone', '[e-mail]:llvm/llvm-test-suite.git', testsuite_srcdir],
+                           # Ignore failure of this step
+                           haltOnFailure=False,
+                           flunkOnFailure=False,
+                           warnOnFailure=False,
+                           workdir='.',
+                           env=merged_env))
+        f.addStep(ShellCommand(name='test-suite_git-pull',
+                           description=["Test-Suite: git pull"],
+                           descriptionDone=["Test-Suite: Getting latest test-suite revision"],
+                           command=['git', 'pull', 'origin', 'master'],
+                           workdir=testsuite_srcdir,
+                           haltOnFailure=True,
+                           env=merged_env))
+
+        f.addStep(RemoveDirectory(name='test-suite_clean-build-dir',
+                           dir=testsuite_builddir,
+                           haltOnFailure=False,
+                           warnOnFailure=True))
+
+        f.addStep(ShellCommand(name='test-suite_cmake-configure',
+                           description=["Test-Suite: cmake"],
+                           command=["cmake", '-B', testsuite_builddir, '-S', testsuite_srcdir, 
+                                    "-DCMAKE_BUILD_TYPE=Release",
+                                    "-DTEST_SUITE_COLLECT_STATS=ON",
+                                    "-DTEST_SUITE_EXTRA_C_FLAGS=-Wno-unused-command-line-argument -mllvm -polly",
+                                    "-DTEST_SUITE_EXTRA_CXX_FLAGS=-Wno-unused-command-line-argument -mllvm -polly",
+                                    WithProperties("-DCMAKE_C_COMPILER=%(builddir)s/" + clangexe),
+                                    WithProperties("-DCMAKE_CXX_COMPILER=%(builddir)s/" + clangxxexe),
+                                    WithProperties("-DTEST_SUITE_LIT=%(builddir)s/" + litexe),
+                                    WithProperties("-DTEST_SUITE_LIT_FLAGS=-sv;-o;report.json"),
+                                    WithProperties("-DTEST_SUITE_LLVM_SIZE=%(builddir)s/" + sizeexe)
+                                ] + extraTestsuiteCmakeArgs,
+                           haltOnFailure=True,
+                           workdir='.',
+                           env=merged_env))
+
+        f.addStep(WarningCountingShellCommand(name='test-suite_build',
+                           description=["Test-Suite: build"],
+                           command=[make, 'all', '-k0'] + jobs_cmd,
+                           # Continue executing, some tests may fail with NOEXE.
+                           haltOnFailure=False,
+                           flunkOnFailure=True,
+                           workdir=testsuite_builddir,
+                           env=merged_env))
+
+        f.addStep(LitTestCommand(name='test-suite_run',
+                            description=['Test-Suite: run'],
+                            command=['../' + litexe, '-sv', '-o', 'report.json', '.'],
+                            haltOnFailure=True,
+                            workdir=testsuite_builddir,
+                            logfiles={ 
+                                'test.log'   : 'test.log',
+                                'report.json': 'report.json'
+                                 },
+                            env=merged_env))
+
     return f