[buildbot] Extend libomp builder to test bot standalone and in-tree builds with and without clang bootstrapping.

[sfantao]

Summary

This patch extend the exiting libomp builder with a set of new option to control the creation of standalone and in-tree builds. With this patch the following configurations can be tested:

• in-tree build of libomp
• standalone build with host compiler
• standalone build with latest clang

The new functionalities are activated for the slave ppc64le-nvidia-K40. The maximum number of jobs was increased so that the clang in-tree compile runs a little faster. For the other libomp slave, gribozavr4`, the behavior is unchanged except for a slightly different naming of the steps.

A new build category was created to distinguish libomp builds that depend on llvm and clang.

[Hahnfeld]

I don't quite like Libiomp5Builder to know all that complex steps. Would it be possible to extend the existing logic for building Clang to include openmp? That way we could later enable the testing on more builders that already build clang on each commit - and finally get rid of rebuilding clang for a simple standalone build

[lijiansong]

zorg/buildbot/builders/Libiomp5Builder.py

74

how to offload with host ?

If I want to compile device-only device code, e.g. clang foo.mlu a.cpp -o bar,
I want to get the following action graph:
for foo.mlu:
input -> preprocess->compile->backend->assemble ,
Then i will get foo.o,
for a.cpp:
input -> preprocess->compile->backend->assemble
then I will get a.o
i want to link foo.o with a.o to get bar, a.cpp is the main, while foo.mlu has the device computation logic.

How can I manage it with offload?

[Hahnfeld]

Testing of openmp is currently very lacking, but I don't think this change will apply cleanly anymore...

buildbot/osuosl/master/config/builders.py

return [
                          env={'PATH':'/home/llvmbb/bin/clang-latest/bin:/home/llvmbb/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin'})},
 
         {'name': "libomp-clang-x86_64-linux-debian",
          'slavenames':["gribozavr4"],
          'builddir':"libomp-clang-x86_64-linux-debian",
          'factory' : Libiomp5Builder.getLibompCMakeBuildFactory(
                          c_compiler="clang",
                          cxx_compiler="clang++",
+                         build_standalone_host=False, 
+                         build_standalone_bootstrap=True,
                          env={'PATH':'/home/llvmbb/bin/clang-latest/bin:/home/llvmbb/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin'})},
 
         {'name': "libomp-ompt-gcc-x86_64-linux-debian",
          'slavenames':["gribozavr4"],
          'builddir':"libomp-ompt-gcc-x86_64-linux-debian",
          'factory' : Libiomp5Builder.getLibompCMakeBuildFactory(
                          c_compiler="gcc",
                          cxx_compiler="g++",
                          ompt=True,
                          env={'PATH':'/home/llvmbb/bin/clang-latest/bin:/home/llvmbb/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin'})},
 
         {'name': "libomp-ompt-clang-x86_64-linux-debian",
          'slavenames':["gribozavr4"],
          'builddir':"libomp-ompt-clang-x86_64-linux-debian",
          'factory' : Libiomp5Builder.getLibompCMakeBuildFactory(
                          c_compiler="clang",
                          cxx_compiler="clang++",
+                         build_standalone_host=False, 
+                         build_standalone_bootstrap=True,
                          ompt=True,
                          env={'PATH':'/home/llvmbb/bin/clang-latest/bin:/home/llvmbb/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin'})},
                                
+        ]
+
+def _get_openmp_complete_builders():
+    return [
         {'name': "libomp-clang-ppc64le-linux-debian",
          'slavenames':["ppc64le-nvidia-K40"],
          'builddir':"libomp-clang-ppc64le-linux-debian",
          'factory' : Libiomp5Builder.getLibompCMakeBuildFactory(
                          c_compiler="/home/bbot/opt/clang/bin/clang",
                          cxx_compiler="/home/bbot/opt/clang/bin/clang++",
+                         build_in_tree=True,
+                         build_standalone_host=True, 
+                         build_standalone_bootstrap=True,
                          env={'PATH':'/home/bbot/opt/cmake/bin:/home/bbot/opt/ninja/bin:/usr/local/bin:/usr/bin:/bin'})},
 
         ]
 
 def _get_libcxx_builders():
     return [
         # gribozavr's builders on gribozavr4 
         {'name': 'libcxx-libcxxabi-x86_64-linux-debian',
def get_builders():
 
     for b in _get_sanitizer_builders():
         b['category'] = 'sanitizer'
         yield b
 
     for b in _get_openmp_builders():
         b['category'] = 'openmp'
         yield b
         
+    for b in _get_openmp_complete_builders():
+        b['category'] = 'openmp_complete'
+        yield b
+
     for b in _get_libcxx_builders():
         b['category'] = 'libcxx'
         yield b
 
     for b in _get_documentation_builders():
         yield b
 
     for b in _get_experimental_scheduled_builders():
         if not b.get('category', '').endswith('.exp'):
            b['category'] = b.get('category', '') + '.exp'
         yield b
 
     for b in _get_on_demand_builders():
         if not b.get('category', '').endswith('.on-demand'):
            b['category'] = b.get('category', '') + '.on-demand'
         yield b

buildbot/osuosl/master/config/slaves.py

return [
 
         # POWER 8 PowerPC little endian (powerpc64le)
         create_slave("ppc64le-clang-test", properties={'jobs': 4}, max_builds=1),
         create_slave("ppc64le-clang-lnt-test", properties={'jobs': 6}, max_builds=1),
         create_slave("ppc64le-clang-multistage-test", properties={'jobs': 8}, max_builds=1),
         create_slave("ppc64le-sanitizer", properties={'jobs': 4}, max_builds=1),
         
         # POWER 8 PowerPC little endian (powerpc64le) with NVIDIA GPUs
-        create_slave("ppc64le-nvidia-K40", properties={'jobs': 4}, max_builds=1),
+        create_slave("ppc64le-nvidia-K40", properties={'jobs': 12}, max_builds=1),
 
         # Ubuntu x86-64, Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
         create_slave("hexagon-build-02", properties={'jobs': 12, 'loadaverage': 32},
             max_builds=1),
 
         # Ubuntu x86-64, Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
         create_slave("hexagon-build-03", properties={'jobs': 16, 'loadaverage':32},
             max_builds=1),

buildbot/osuosl/master/master.cfg

c['schedulers'].append(SingleBranchScheduler(name="sanitizer_scheduler",
                                 "libunwind",
                                 "lld"])))
 
 c['schedulers'].append(SingleBranchScheduler(name="openmp_scheduler",
                             treeStableTimer=2*60,
                             builderNames=get_all_for("openmp"),
                             change_filter=depends_on([
                                 "openmp"])))
                                 
+c['schedulers'].append(SingleBranchScheduler(name="openmp_complete_scheduler",
+                            treeStableTimer=2*60,
+                            builderNames=get_all_for("openmp_complete"),
+                            change_filter=depends_on([
+                                "openmp", 
+                                "llvm", 
+                                "cfe"])))
+
 c['schedulers'].append(SingleBranchScheduler(name="libcxx_scheduler",
                             treeStableTimer=2*60,
                             builderNames=get_all_for("libcxx"),
                             change_filter=depends_on([
                                 "libcxx",
                                 "libcxxabi",
                                 "libunwind"])))
 

zorg/buildbot/builders/Libiomp5Builder.py

 import os
 
 import buildbot
 import buildbot.process.factory
 from buildbot.steps.source import SVN
 from buildbot.steps.shell import ShellCommand
 from buildbot.steps.shell import WarningCountingShellCommand
 from buildbot.process.properties import WithProperties
 from zorg.buildbot.commands.LitTestCommand import LitTestCommand
 from zorg.buildbot.commands.NinjaCommand import NinjaCommand
 
 
-def getLibompCMakeBuildFactory(clean=True, env=None, ompt=False, test=True, c_compiler="gcc", cxx_compiler="g++"):
+def getLibompCMakeBuildFactory(clean=True, env=None, ompt=False, test=True, build_in_tree=False, build_standalone_host=True, build_standalone_bootstrap=False, c_compiler="gcc", cxx_compiler="g++"):
 
     # Prepare environmental variables. Set here all env we want everywhere.
     merged_env = {
         'TERM' : 'dumb' # Make sure Clang doesn't use color escape sequences.
                  }
     if env is not None:
         # Overwrite pre-set items with the given ones, so user can set anything.
         merged_env.update(env)
 
     openmp_srcdir = "openmp.src"
     openmp_builddir = "openmp.build"
+    openmp_builddir_bootstrap = "openmp.build.bootstrap"
     llvm_srcdir = "llvm.src"
     llvm_builddir = "llvm.build"
+    standalone = build_standalone_host or build_standalone_bootstrap;
 
     f = buildbot.process.factory.BuildFactory()
 
-    # Get libomp
-    f.addStep(SVN(name='svn-libomp',
+    # Get libomp in a separate folder if we are testing standalone builds.
+    if standalone:
+        f.addStep(SVN(name='svn-libomp standalone',
                       mode='update',
                       baseURL='http://llvm.org/svn/llvm-project/openmp/',
                       defaultBranch='trunk',
                       workdir=openmp_srcdir))
 
-    # Get llvm to build llvm-lit
+    # Get llvm to build llvm-lit and trigger an eventual in-tree build for the
+    # OpenMP library.
     f.addStep(SVN(name='svn-llvm',
                   mode='update',
                   baseURL='http://llvm.org/svn/llvm-project/llvm/',
                   defaultBranch='trunk',
                   workdir=llvm_srcdir))
 
-    # Get clang  if c/c++ compilers is clang/clang++
-    if c_compiler == "clang":
+    # Get clang if we are bootstrapping.
+    if build_standalone_bootstrap:
         f.addStep(SVN(name='svn-clang',
                       mode='update',
                       baseURL='http://llvm.org/svn/llvm-project/cfe/',
                       defaultBranch='trunk',
                       workdir='%s/tools/clang' % llvm_srcdir))
                       
+    # Checkout libomp in-tree if requested.                   
+    if build_in_tree:
+        f.addStep(SVN(name='svn-libomp in tree',
+                      mode='update',
+                      baseURL='http://llvm.org/svn/llvm-project/openmp/',
+                      defaultBranch='trunk',
+                      workdir='%s/projects/openmp' % llvm_srcdir))           
+
     # Clean directory, if requested.
     if clean:
         f.addStep(ShellCommand(name="clean",
-                               command=["rm", "-rf",openmp_builddir,llvm_builddir],
+                               command=["rm", "-rf", openmp_builddir, openmp_builddir_bootstrap, llvm_builddir],
                                warnOnFailure=True,
                                description=["clean"],
                                workdir='.',
                                env=merged_env))
 
-    # CMake llvm
-    f.addStep(ShellCommand(name='cmake llvm',
-                           command=["cmake", "../"+llvm_srcdir,
-                                    "-G", "Ninja",
-                                    "-DCMAKE_BUILD_TYPE=Release",
-                                    "-DLLVM_ENABLE_ASSERTIONS=ON",
-                                    "-DCMAKE_C_COMPILER="+c_compiler,
-                                    "-DCMAKE_CXX_COMPILER="+cxx_compiler],
+    # CMake llvm and everything else that was checkout in-tree.
+             "-DCMAKE_CXX_COMPILER="+cxx_compiler]
+             
+    # Add OMPT option to the in_tree cmake command if requested.
+    if ompt and build_in_tree:
+        command.append("-DLIBOMP_OMPT_SUPPORT=ON")
+        
+    f.addStep(ShellCommand(name='cmake llvm',
+                           command=command,
                            haltOnFailure=True,
                            description='cmake llvm',
                            workdir=llvm_builddir,
                            env=merged_env))
 
-    # Make clang build or just llvm utils build
-    if c_compiler == "clang": 
-        f.addStep(NinjaCommand(name="make clang build",
+    # Build everything in-tree. This will build clang if we are bootstrapping
+    # and build libomp if the caller requested so. Otherwise, just build the 
+    # target that gives us llvm-lit.
+    if build_standalone_bootstrap or build_in_tree: 
+        f.addStep(NinjaCommand(name="make tree build",
                                haltOnFailure=True,
-                               description=["make clang build"],
+                               description=["make tree build"],
                                workdir=llvm_builddir,
                                env=merged_env))
     else:
         f.addStep(NinjaCommand(name="make llvm utils build",
                                targets=["LLVMX86Utils"],
                                haltOnFailure=True,
                                description=["make llvm utils build"],
                                workdir=llvm_builddir,
                                env=merged_env))
 
     if ompt:
         f.addStep(NinjaCommand(name="make ompt test utils",
                                targets=["FileCheck"],
                                haltOnFailure=True,
                                description=["make ompt test utils"],
                                workdir=llvm_builddir,
                                env=merged_env))
 
     # Add clang/llvm-lit to PATH
     merged_env.update( { 'PATH' : WithProperties("%(workdir)s/" + llvm_builddir + "/bin:" + "${PATH}")} )
 
+    # Gather required information for standalone builds.
+    standalone_build_info = [];
+    if build_standalone_host:
+        standalone_build_info.append((c_compiler, cxx_compiler, openmp_builddir, "host"))    
+    if build_standalone_bootstrap:
+        standalone_build_info.append(("clang", "clang++", openmp_builddir_bootstrap, "bootstrap"))
+
+    # Create standalone build steps.
+    for cc, cxx, build_dir, name in standalone_build_info:
         # CMake libomp
         command=["cmake", "../"+openmp_srcdir,
-             "-DCMAKE_C_COMPILER="+c_compiler,
-             "-DCMAKE_CXX_COMPILER="+cxx_compiler]
+                 "-G", "Ninja",
+                 "-DCMAKE_C_COMPILER="+cc,
+                 "-DCMAKE_CXX_COMPILER="+cxx]
         if ompt:
             command.append("-DLIBOMP_OMPT_SUPPORT=ON")
-    f.addStep(ShellCommand(name='cmake libomp',
+        f.addStep(ShellCommand(name='cmake libomp ' + name,
                                command=command,
                                haltOnFailure=True,
-                           description='cmake libomp',
-                           workdir=openmp_builddir,
+                               description='cmake libomp ' + name,
+                               workdir=build_dir,
                                env=merged_env))
 
         # Make libomp
-    f.addStep(WarningCountingShellCommand(name='make libomp build',
-                                          command=['make'],
-                                          haltOnFailure=True,
-                                          description='make libomp build',
-                                          workdir=openmp_builddir,
-                                          env=merged_env))
+        f.addStep(NinjaCommand(name='make libomp build ' + name,
+                               description=['make libomp build ' + name],
+                               workdir=build_dir,
                                                
-    # Test, if requested
+    # Gather information for tests and run them, if required.
     if test:
-        f.addStep(LitTestCommand(name="make check-libomp",
-                                 command=["make", "check-libomp"],
-                                 haltOnFailure=True,
-                                 description=["make check-libomp"],
-                                 descriptionDone=["build checked"],
-                                 workdir=openmp_builddir,
+        tests_info = [];
+        if build_in_tree:
+          tests_info.append((llvm_builddir, "in-tree"))  
+        if build_standalone_host:
+            tests_info.append((openmp_builddir, "host"))    
+        if build_standalone_bootstrap:
+            tests_info.append((openmp_builddir_bootstrap, "bootstrap"))
+        for build_dir, name in tests_info:
+            f.addStep(LitTestCommand(name="make check-libomp " + name,
+                                     command=["ninja", "check-libomp"],
+                                     haltOnFailure=True,
+                                     description=["make check-libomp " + name],
+                                     descriptionDone=[name + " build checked"],
+                                     workdir=build_dir,
                                      env=merged_env))
 
     return f