[buildbot] Added CUDATestsuiteBuilder.py

[tra]

Summary

getCUDATestsuiteBuildFactory() returns a factory that:

• checks out clang, libcxx(+deps), and test-suite.
• compiles & tests clang.
• compiles and runs CUDA tests.

Added checkout_libcxx and checkout_test_suite knobs to
getClangCmakeBuildFactory() so it can be reused for other
clang-based buildbots.

Tested on a local buildmaster+buildslave.

[jlebar]

zorg/buildbot/builders/CUDATestsuiteBuilder.py

86 ↗

(On Diff #81421)

Nit, this would be clearer to me if it was indented like cuda_test_env above.

135 ↗

(On Diff #81421)

Nit, no parens in if.

[tra]

fixed formatting nits.

[gkistanova]

Thanks for working on this, Artem.

zorg/buildbot/builders/CUDATestsuiteBuilder.py

16 ↗

(On Diff #81434)

Please do not use mutable default arguments.

52 ↗

(On Diff #81434)

Maybe adding an extra indent here? This is cosmetic, though.

101 ↗

(On Diff #81434)

Could you use CmakeCommand instead of ShellCommand here, please?

113 ↗

(On Diff #81434)

Could you use NinjaCommand instead of WarningCountingShellCommand here and in all the other similar places, please?

[tra]

Addressed Galina's comments.

[gkistanova]

It is almost there.

Few small things to address and the patch would be ready to land.

zorg/buildbot/builders/CUDATestsuiteBuilder.py

2 ↗

(On Diff #82004)

For the version of the buildbot we use, it should be
from buildbot.steps.slave import RemoveDirectory

and it used just as RemoveDirectory below.

18 ↗

(On Diff #82004)

Please do not use mutable default arguments.

42 ↗

(On Diff #82004)

You do not need this any more. Instead feel free to pass jobs straight to the NinjaCommand as is, and it will take care of the correct propagation of the jobs from a factory argument, or from a buildslave, a builder, or even a build parameters.

84 ↗

(On Diff #82004)

Just RemoveDirectory here. Please see my comment about the buildbot version above.

132 ↗

(On Diff #82004)

You do not need this any more, do you?

156 ↗

(On Diff #82004)

If it takes a long to build and there is no TTY output during that time, you may want to set a longer timeout for this step.

[tra]

Addressed Galina's comments.

[tra]

zorg/buildbot/builders/CUDATestsuiteBuilder.py

2 ↗

(On Diff #82004)

Buildbot seems to be moving RemoveDirectory all over the place in different versions.
I've changed the code to try loading it in a way that should work with both 0.8.5 and 0.8.12.

156 ↗

(On Diff #82004)

Individual compilations should be well within default timeout of 20 minutes.

[gkistanova]

LGTM.
Thanks for the changes, Artem.

zorg/trunk/zorg/buildbot/builders/CUDATestsuiteBuilder.py

+from buildbot.process.properties import WithProperties
+try:
+  # buildbot 0.8.5
+  from buildbot.steps.slave import RemoveDirectory
+except:
+  # buildbot 0.8.12
+  from buildbot.plugins import steps
+  RemoveDirectory = steps.RemoveDirectory
+
+from zorg.buildbot.builders import ClangBuilder
+from zorg.buildbot.commands.CmakeCommand import CmakeCommand
+from zorg.buildbot.commands.NinjaCommand import NinjaCommand
+
+
+def getCUDATestsuiteBuildFactory(
+        externals,  # Directory with CUDA, thrust and gcc versions for testing.
+        always_clean=True,
+        test=False,
+        useTwoStage=False,
+        cmake='cmake',
+        extra_cmake_args=None,  # Extra CMake args for all stages.
+        extra_ts_cmake_args=None,  # extra cmake args for testsuite.
+        jobs=None,
+        cuda_jobs=1,  # number of simultaneous CUDA apps to run
+        env=None,  # Environmental variables for all steps.
+        enable_thrust_tests=False,
+        split_thrust_tests=False,  # Each thrust test is a separate executable.
+        run_thrust_tests=False,
+        enable_libcxx=True,  # checkout/build libcxx to test with.
+        gpu_arch_list=None,
+        gpu_devices=None,  # List of devices to make visible to  CUDA
+        stage1_config='Release',
+        stage2_config='Release'):
+
+    if extra_cmake_args is None:
+        extra_cmake_args = []
+    if extra_ts_cmake_args is None:
+        extra_ts_cmake_args = []
+
+    # Prepare environmental variables. Set here all env we want for all steps.
+    merged_env = {
+        'TERM': 'dumb'  # Make sure Clang doesn't use color escape sequences.
+    }
+    if env is not None:
+        # Overwrite pre-set items with the given ones, so user can set
+        # anything.
+        merged_env.update(env)
+
+    source_dir = 'llvm'   # Should match the one used in getClangCMakeBuildFactory.
+    stage1_build_dir = 'stage1'  # Should match the one defined in getClangCMakeBuildFactory.
+    stage2_build_dir = 'stage2'  # Should match the one defined in getClangCMakeBuildFactory.
+
+    if useTwoStage:
+        clang_build_dir = stage2_build_dir
+    else:
+        clang_build_dir = stage1_build_dir
+
+    # Build clang.
+    f = ClangBuilder.getClangCMakeBuildFactory(
+            clean=always_clean,
+            test=test,
+            cmake=cmake,
+            extra_cmake_args=extra_cmake_args,
+            jobs=jobs,
+            env=merged_env,
+            useTwoStage=useTwoStage,
+            stage1_config=stage1_config,
+            stage2_config=stage2_config,
+            checkout_libcxx=enable_libcxx,
+            checkout_test_suite=True)
+
+    cuda_test_env = {
+        'PYTHONPATH': WithProperties("%(workdir)s/" + source_dir +
+                                     "/utils/lit:${PYTHONPATH}"),
+        'PATH': WithProperties("%(workdir)s/" + clang_build_dir +
+                               "/bin:${PATH}"),
+    }
+    # Limit GPUs visible to CUDA.
+    if gpu_devices:
+        cuda_test_env["CUDA_VISIBLE_DEVICES"] = ",".join(
+            str(x) for x in gpu_devices)
+    merged_env.update(cuda_test_env)
+
+    ts_build_dir = 'test-suite-build'
+
+    # Completely remove test suite build dir.
+    f.addStep(
+        RemoveDirectory(name="Remove old test-suite build directory",
+                        dir=ts_build_dir))
+
+    if extra_ts_cmake_args:
+        cmake_args = extra_ts_cmake_args[:]
+    else:
+        cmake_args = []
+
+    # Set proper defaults.
+    CmakeCommand.applyDefaultOptions(cmake_args, [
+        ('-DCMAKE_BUILD_TYPE=',        'Release'),
+    ])
+
+    # Some options are required for this stage no matter what.
+    CmakeCommand.applyRequiredOptions(cmake_args, [
+        ('-G',                      'Ninja'),
+    ])
+
+    cmake_args.append(
+        WithProperties(
+            "-DCMAKE_CXX_COMPILER=%(workdir)s/" +
+                                    clang_build_dir + "/bin/clang++"
+        ))
+    cmake_args.append(
+        WithProperties(
+            "-DCMAKE_C_COMPILER=%(workdir)s/" + clang_build_dir + "/bin/clang"
+        ))
+
+    cmake_args.append('-DTEST_SUITE_SUBDIRS=External'),
+                      # Limit to External tests only.
+
+    if externals:
+        cmake_args.append('-DTEST_SUITE_EXTERNALS_DIR=' + externals)
+    if split_thrust_tests:
+        cmake_args.append('-DTHRUST_SPLIT_TESTS=1')
+    if gpu_arch_list:
+        cmake_args.append('-DCUDA_GPU_ARCH=' + ';'.join(gpu_arch_list))
+    if cuda_jobs:
+        cmake_args.append('-DCUDA_JOBS=%s' % cuda_jobs)
+
+    # Then do fresh cmake configuration.
+    f.addStep(CmakeCommand(name='cmake test-suite',
+                           description='cmake test-suite',
+                           haltOnFailure=True,
+                           options=cmake_args,
+                           path="../test/test-suite",
+                           workdir=ts_build_dir,
+                           env=merged_env))
+
+    # Always build simple CUDA tests. They serve as compilation
+    # smoketests and will fail quickly if compiler has obvious issues
+    # compiling CUDA files.
+    f.addStep(NinjaCommand(
+        name='ninja build simple CUDA tests',
+        targets=["cuda-tests-simple"],
+        jobs=jobs,
+        haltOnFailure=True,
+        description=["building simple CUDA tests"],
+        descriptionDone=["simple CUDA tests built."],
+        workdir=ts_build_dir,
+        env=merged_env))
+
+    f.addStep(NinjaCommand(
+        name='run simple CUDA tests',
+        targets=["check-cuda-simple"],
+        jobs=1, # lit will parallelize the jobs
+        haltOnFailure=True,
+        description=["Running simple CUDA tests"],
+        descriptionDone=["simple CUDA tests done."],
+        workdir=ts_build_dir,
+        env=merged_env))
+
+    # If we've enabled thrust tests, build them now.
+    # WARNING: This takes a lot of time to build.
+    if (enable_thrust_tests):
+        f.addStep(NinjaCommand(
+            name='ninja build thrust',
+            targets=["cuda-tests-thrust"],
+            jobs=jobs,
+            haltOnFailure=True,
+            description=["building thrust tests"],
+            descriptionDone=["thrust tests built."],
+            workdir=ts_build_dir,
+            env=merged_env))
+        # Run them. That also takes a while.
+        # TODO(tra) we may want to run more than one instance so one
+        # can be compiling tests while another is running them on GPU.
+        if run_thrust_tests:
+            f.addStep(NinjaCommand(
+                name='run all CUDA tests',
+                targets=["check"],
+                jobs=1, # lit will parallelize the jobs.
+                haltOnFailure=True,
+                description=["running all CUDA tests."],
+                descriptionDone=["all cuda tests done."],
+                workdir=ts_build_dir,
+                env=merged_env))
+
+    return f

zorg/trunk/zorg/buildbot/builders/ClangBuilder.py

if package_dst:
                                env=merged_env))
 
     return f
 
 def addSVNUpdateSteps(f,
                       checkout_clang_tools_extra,
                       checkout_compiler_rt,
                       checkout_test_suite,
-                      checkout_lld):
+                      checkout_lld,
+                      checkout_libcxx):
     # We *must* checkout at least Clang+LLVM
     f.addStep(SVN(name='svn-llvm',
                   mode='update', baseURL='http://llvm.org/svn/llvm-project/llvm/',
                   defaultBranch='trunk',
                   workdir='llvm'))
     f.addStep(SVN(name='svn-clang',
                   mode='update', baseURL='http://llvm.org/svn/llvm-project/cfe/',
                   defaultBranch='trunk',
if checkout_test_suite:
                       mode='update', baseURL='http://llvm.org/svn/llvm-project/test-suite/',
                       defaultBranch='trunk',
                       workdir='test/test-suite'))
     if checkout_lld:
         f.addStep(SVN(name='svn-lld',
                       mode='update', baseURL='http://llvm.org/svn/llvm-project/lld/',
                       defaultBranch='trunk',
                       workdir='llvm/tools/lld'))
+    if checkout_libcxx:
+        f.addStep(SVN(name='svn-libcxx',
+                      mode='update',
+                      baseURL='http://llvm.org/svn/llvm-project/libcxx/',
+                      defaultBranch='trunk',
+                      workdir='llvm/projects/libcxx'))
+        f.addStep(SVN(name='svn-libcxxabi',
+                      mode='update',
+                      baseURL='http://llvm.org/svn/llvm-project/libcxxabi/',
+                      defaultBranch='trunk',
+                      workdir='llvm/projects/libcxxabi'))
+        f.addStep(SVN(name='svn-libunwind',
+                      mode='update',
+                      baseURL='http://llvm.org/svn/llvm-project/libunwind/',
+                      defaultBranch='trunk',
+                      workdir='llvm/projects/libunwind'))
 
 def addGCSUploadSteps(f, package_name, install_prefix, gcs_directory, env,
                       gcs_url_property=None):
     """
     Add steps to upload to the Google Cloud Storage bucket.
 
     f - The BuildFactory to modify.
     package_name - The name of this package for the descriptions (e.g.
def getClangCMakeBuildFactory(
 
             # Environmental variables for all steps.
             env={},
             extra_cmake_args=[],
 
             # Extra repositories
             checkout_clang_tools_extra=True,
             checkout_compiler_rt=True,
-            checkout_lld=True):
+            checkout_lld=True,
+            checkout_libcxx=False,
+            checkout_test_suite=False):
     return _getClangCMakeBuildFactory(
                clean=clean, test=test, cmake=cmake, jobs=jobs, vs=vs,
                vs_target_arch=vs_target_arch, useTwoStage=useTwoStage,
                testStage1=testStage1, stage1_config=stage1_config,
                stage2_config=stage2_config, runTestSuite=runTestSuite,
                nt_flags=nt_flags, submitURL=submitURL, testerName=testerName,
                env=env, extra_cmake_args=extra_cmake_args,
                checkout_clang_tools_extra=checkout_clang_tools_extra,
                checkout_lld=checkout_lld,
-               checkout_compiler_rt=checkout_compiler_rt)
+               checkout_compiler_rt=checkout_compiler_rt,
+               checkout_libcxx=checkout_libcxx,
+               checkout_test_suite=checkout_test_suite)
 
 def _getClangCMakeBuildFactory(
             clean=True,
             test=True,
             cmake='cmake',
             jobs=None,
 
             # VS tools environment variable if using MSVC. For example,
def _getClangCMakeBuildFactory(
             # Environmental variables for all steps.
             env={},
             extra_cmake_args=[],
 
             # Extra repositories
             checkout_clang_tools_extra=True,
             checkout_compiler_rt=True,
             checkout_lld=True,
+            checkout_libcxx=False,
+            checkout_test_suite=False,
 
             # Upload artifacts to Google Cloud Storage (for the llvmbisect tool)
             stage1_upload_directory=None,
 
             # Triggers
             trigger_after_stage1=[]):
 
     ############# PREPARING
     f = buildbot.process.factory.BuildFactory()
 
     addSVNUpdateSteps(f,
                       checkout_clang_tools_extra=checkout_clang_tools_extra,
                       checkout_compiler_rt=checkout_compiler_rt,
                       checkout_lld=checkout_lld,
-                      checkout_test_suite=runTestSuite)
+                      checkout_test_suite=runTestSuite or checkout_test_suite,
+                      checkout_libcxx=checkout_libcxx)
 
     # If jobs not defined, Ninja will choose a suitable value
     jobs_cmd = []
     lit_args = "'-v"
     if jobs is not None:
         jobs_cmd = ["-j"+str(jobs)]
         lit_args += " -j"+str(jobs)+"'"
     else:
ignores_path = os.path.join(path, 'gdb-1472-testsuite', 'expected_results',
                                 key)
     gdb_dg_ignores = (
         readList(os.path.join(ignores_path, 'FAIL.txt')) +
         readList(os.path.join(ignores_path, 'UNRESOLVED.txt')) +
         readList(os.path.join(ignores_path, 'XPASS.txt')))
     ignores['gdb-1472-testsuite' ] = gdb_dg_ignores
 
     return ignores
-