[Zorg][OpenMP] Add CUDA offloading worker.

[Meinersbur]

Summary

This worker tests OpenMP offloading for the x86_64 and NVIDIA GPU. In addition to check-openmp it runs the SOLLVE Validation & Verification Suite via the LLVM test-suite External builder. The builder is configured to only warn if the SOLLVE suite fails, as it also tests features that have not been implemented in Clang yet.

CUDA is intentionally not installed in a default location (/opt/cuda-11.3) to resemble setups often found in computing clusters with multiple versions of CUDA to choose from.

The worker is currently connected to a private buildbot master where this patch has been applied. The builder are currently red because of the problems mentioned on the OpenMP mailing list post. At least D101265 and D101266 are required to build it correctly.

[gkistanova]

Thanks for the patch, Michael!

After the recent change in LLVMBuildFactory (https://reviews.llvm.org/rZORG2ed0c6ae25a4) and the corresponding change in UnifiedTreeBuilder (https://reviews.llvm.org/rZORG90c0b74c2ee5) this patch could be more straight forward.
You can just use f.enable_projects and f.enable_runtimes respectively, no need to explicitly specify t

Could you update the patch, please?

[Meinersbur]

• Rebase to use f.enable_projects / f.enable_runtimes
• Make TEST_SUITE_SOLLVEVV_OFFLOADING flags worker-dependent
• Use /opt/cuda instead of /opt/cuda-11.3 to allow future updated of CUDA without needing to update the worker.

[Meinersbur]

• Add missing import

[Meinersbur]

@gkistanova ping

[gkistanova]

LGTM

[gkistanova]

zorg/buildbot/builders/OpenMPBuilder.py

46

Somehow missed this during the review. Sorry.
runtime_projects is not defined. Most likely it has been left behind when you renamed the enable_runtimes argument. Anyway, you do not really need this, as LLVMBuildFactory does that for you.
The fix is committed as https://reviews.llvm.org/rZORG69e37a9fe8b7.

buildbot/osuosl/master/config/builders.py

# OpenMP builders.
     'workernames' : ["gribozavr4"],
     'builddir': "openmp-clang-x86_64-linux-debian",
     'factory' : OpenMPBuilder.getOpenMPCMakeBuildFactory(
                     env={
                         'PATH':'/home/llvmbb/bin/clang-latest/bin:/home/llvmbb/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin',
                         'CC': 'ccache clang', 'CXX': 'ccache clang++', 'CCACHE_CPP2': 'yes',
                     })},
 
+    {'name' : "openmp-offload-cuda-project",
+    'tags'  : ["openmp"],
+    'workernames' : ["minipc-1050ti-linux"],
+    'builddir': "openmp-offload-cuda-project",
+    'factory' : OpenMPBuilder.getOpenMPCMakeBuildFactory(
+                        clean=False,
+                        enable_runtimes=[],
+                        extraCmakeArgs=[
+                                "-DCUDA_TOOLKIT_ROOT_DIR=/opt/cuda",
+                                "-DLIBOMPTARGET_BUILD_NVPTX_BCLIB=ON",
+                                "-DCLANG_ENABLE_STATIC_ANALYZER=OFF",
+                                "-DCLANG_ENABLE_ARCMT=OFF",
+                                "-DCLANG_ENABLE_OBJC_REWRITER=OFF",
+                                "-DLLVM_TARGETS_TO_BUILD=X86;NVPTX",
+                                "-DLLVM_ENABLE_LLD=ON",
+                                '-DLLVM_PARALLEL_LINK_JOBS=2',
+                            ],
+                        install=True,
+                        testsuite=True,
+                        testsuite_sollvevv=True,
+                        extraTestsuiteCmakeArgs=[
+                            "-DTEST_SUITE_SOLLVEVV_OFFLOADING_CFLAGS=-fopenmp-targets=nvptx64-nvidia-cuda;--cuda-path=/opt/cuda",
+                            "-DTEST_SUITE_SOLLVEVV_OFFLOADING_LDFLAGS=-fopenmp-targets=nvptx64-nvidia-cuda;--cuda-path=/opt/cuda",
+                        ],
+                    )},
+
+    {'name' : "openmp-offload-cuda-runtime",
+    'tags'  : ["openmp"],
+    'workernames' : ["minipc-1050ti-linux"],
+    'builddir': "openmp-offload-cuda-runtime",
+    'factory' : OpenMPBuilder.getOpenMPCMakeBuildFactory(
+                        clean=False,
+                        enable_runtimes=['openmp'],
+                        extraCmakeArgs=[
+                                "-DCUDA_TOOLKIT_ROOT_DIR=/opt/cuda",
+                                "-DLIBOMPTARGET_BUILD_NVPTX_BCLIB=ON",
+                                "-DCLANG_ENABLE_STATIC_ANALYZER=OFF",
+                                "-DCLANG_ENABLE_ARCMT=OFF",
+                                "-DCLANG_ENABLE_OBJC_REWRITER=OFF",
+                                "-DLLVM_TARGETS_TO_BUILD=X86;NVPTX",
+                                "-DLLVM_ENABLE_LLD=ON",
+                                '-DLLVM_PARALLEL_LINK_JOBS=2',
+                            ],
+                        install=True,
+                        testsuite=True,
+                        testsuite_sollvevv=True,
+                        extraTestsuiteCmakeArgs=[
+                            "-DTEST_SUITE_SOLLVEVV_OFFLOADING_CFLAGS=-fopenmp-targets=nvptx64-nvidia-cuda;--cuda-path=/opt/cuda",
+                            "-DTEST_SUITE_SOLLVEVV_OFFLOADING_LDFLAGS=-fopenmp-targets=nvptx64-nvidia-cuda;--cuda-path=/opt/cuda",
+                        ],
+                    )},
+
 # Libc++ builders.
 
     {'name': 'libcxx-libcxxabi-x86_64-linux-debian',
     'tags'  : ["libcxx"],
     'workernames': ['gribozavr4'],
     'builddir': 'libcxx-libcxxabi-x86_64-linux-debian',
     'factory': LibcxxAndAbiBuilder.getLibcxxAndAbiBuilder(
                     env={'CC': 'clang', 'CXX': 'clang++'},

buildbot/osuosl/master/config/workers.py

return [
 
         # Ubuntu 18.04.LTS x86_64, GCE instance
         create_worker("polly-x86_64-gce1", properties={'jobs': 2}, max_builds=1),
         create_worker("polly-x86_64-gce2", properties={'jobs': 2}, max_builds=1),
 
         # Ubuntu 18.04.LTS x86_64, Intel(R) Xeon(R) CPU X3460 @ 2.80GHz, 32 GiB RAM
         create_worker("polly-x86_64-fdcserver", properties={'jobs': 8, 'loadaverage': 8}, max_builds=1),
 
+        # Ubuntu 20.04.LTS x86_64, Intel(R) Core(TM) i5-9400F CPU @ 2.90Ghz, 16 GiB RAM, NVIDIA GeForce GTX 1050 Ti (Pascal, sm_61, 4GiB)
+        create_worker("minipc-1050ti-linux", properties={'jobs': 6}, max_builds=1),
+
         create_worker("nersc-flang"),
         create_worker("alcf-theta-flang", properties={'jobs': 12}, max_builds=1),
 
         # ML-Driven Compiler Optimizations build workers (Ubuntu x86_64)
         create_worker("ml-opt-dev-x86-64-b1",
                      properties={'jobs': 64}, max_builds=1),
         create_worker("ml-opt-rel-x86-64-b1",
                      properties={'jobs': 64}, max_builds=1),

zorg/buildbot/builders/OpenMPBuilder.py

 from buildbot.steps.shell import ShellCommand
 from buildbot.process.properties import WithProperties
+from buildbot.plugins import steps
 
 from zorg.buildbot.commands.LitTestCommand import LitTestCommand
 from zorg.buildbot.commands.CmakeCommand import CmakeCommand
 from zorg.buildbot.commands.NinjaCommand import NinjaCommand
 from zorg.buildbot.builders import UnifiedTreeBuilder
 from zorg.buildbot.process.factory import LLVMBuildFactory
 
 def getOpenMPCMakeBuildFactory(
         jobs                = '%(jobs)s',   # Number of concurrent jobs.
         clean               = True,         # "clean" step is requested if true
         env                 = None,         # Environmental variables for all steps.
         ompt                = False,        # Whether to enable the OpenMP Tools Interface.
         test                = True,         # Test the built libraries.
         depends_on_projects = None,
+        enable_runtimes     = None,
+        extraCmakeArgs      = [],
+        install             = False,
+        testsuite           = False,
+        testsuite_sollvevv  = False,
+        extraTestsuiteCmakeArgs = [],
         **kwargs):
 
     # Prepare environmental variables. Set here all env we want everywhere.
     merged_env = {
         'TERM' : 'dumb' # Make sure Clang doesn't use color escape sequences.
                  }
     # Overwrite pre-set items with the given ones, so user can set anything.
     if env is not None:
         merged_env.update(env)
 
     llvm_srcdir = 'llvm.src'
     llvm_builddir = 'llvm.build'
+    llvm_instdir = 'llvm.inst'
+    testsuite_srcdir = "test-suite.src"
+    testsuite_builddir = "test-suite.build"
+    sollvevv_srcdir = "sollvevv.src"
 
     cleanBuildRequested = lambda step: clean or step.build.getProperty("clean", default=step.build.getProperty("clean_obj"))
 
     if depends_on_projects is None:
         # Monorepo configuration requires llvm and clang to get cmake work.
         depends_on_projects = ['llvm', 'clang', 'openmp']
+    depends_on_projects += [p for p in runtime_projects if p not in depends_on_projects]
 
     f = UnifiedTreeBuilder.getLLVMBuildFactoryAndSourcecodeSteps(
             depends_on_projects=depends_on_projects,
+            enable_runtimes=enable_runtimes,
             llvm_srcdir=llvm_srcdir,
             obj_dir=llvm_builddir,
             cleanBuildRequested=cleanBuildRequested,
             env=merged_env,
             **kwargs) # Pass through all the extra arguments.
 
     f.addStep(
         ShellCommand(
             name            = 'clean',
             command         = ['rm', '-rf', f.obj_dir],
             warnOnFailure   = True,
             description     = ['clean'],
             doStepIf        = cleanBuildRequested,
             workdir         = '.',
             env             = merged_env))
 
     # Configure LLVM and OpenMP (and Clang, if requested).
     cmake_args = ['-DCMAKE_BUILD_TYPE=Release', '-DLLVM_ENABLE_ASSERTIONS=ON']
     if ompt:
         cmake_args += ['-DLIBOMP_OMPT_SUPPORT=ON']
     if test:
         lit_args = '-vv --show-unsupported --show-xfail -j %s' % jobs
         cmake_args += [WithProperties('-DLLVM_LIT_ARGS=%s' % lit_args)]
+    if install:
+        cmake_args += [WithProperties('-DCMAKE_INSTALL_PREFIX=%(builddir)s/' + llvm_instdir)]
+    cmake_args += extraCmakeArgs
+
+    if f.enable_projects:
+        CmakeCommand.applyDefaultOptions(cmake_args, [
+            ('-DLLVM_ENABLE_PROJECTS=', ";".join(f.enable_projects)),
+            ])
 
-    CmakeCommand.applyRequiredOptions(cmake_args, [
-        ('-DLLVM_ENABLE_PROJECTS=', ";".join(f.depends_on_projects)),
+    if f.enable_runtimes:
+        CmakeCommand.applyDefaultOptions(cmake_args, [
+            ('-DLLVM_ENABLE_RUNTIMES=', ";".join(f.enable_runtimes)),
             ])
 
     # Add llvm-lit and clang (if built) to PATH
     merged_env.update({
         'PATH': WithProperties('%(builddir)s/' + llvm_builddir + '/bin:${PATH}')})
 
     src_dir = LLVMBuildFactory.pathRelativeTo(f.llvm_srcdir, f.obj_dir)
 
     f.addStep(CmakeCommand(name='configure-openmp',
if test:
         ninja_test_args = ['ninja', WithProperties('-j %s' % jobs)]
         f.addStep(
             LitTestCommand(
                 name        = 'test-openmp',
                 command     = ninja_test_args + ['check-openmp'],
                 description = 'test openmp',
                 workdir     = f.obj_dir,
                 env         = merged_env,
+                haltOnFailure=False,
+                flunkOnFailure=True))
+
+    clangexe = "%(builddir)s/" + llvm_builddir + "/bin/clang"
+    clangxxexe = "%(builddir)s/" + llvm_builddir + "/bin/clang++"
+    litexe = "%(builddir)s/" + llvm_builddir + "/bin/llvm-lit"
+    libdir = "%(builddir)s/" + llvm_builddir + "/lib"
+    if install:
+        f.addStep(steps.RemoveDirectory(name="LLVM: Clean Install Dir",
+                               dir=llvm_instdir,
+                               haltOnFailure=False))
+
+        f.addStep(NinjaCommand(name="LLVM: Install",
+                               description="installing",
+                               descriptionDone="install",
+                               descriptionSuffix="LLVM",
+                               workdir=f.obj_dir,
+                               targets=['install'],
+                               env=merged_env,
                                haltOnFailure=True))
+        # If installing, use the installed version of clang.
+        clangexe = "%(builddir)s/" + llvm_instdir + "/bin/clang"
+        clangxxexe = "%(builddir)s/" + llvm_instdir + "/bin/clang++"
+        libdir = "%(builddir)s/" + llvm_instdir + "/lib"
+
+
+    if testsuite:
+        f.addStep(steps.RemoveDirectory(name="Test-Suite: Clean Source",
+                           dir=testsuite_srcdir,
+                           haltOnFailure=False,
+                           warnOnFailure=True,
+                           doStepIf=cleanBuildRequested))
+
+        f.addGetSourcecodeForProject(name="Test-Suite: Checkout",
+            description="fetching",
+            descriptionDone="fetch",
+            descriptionSuffix="Test-Suite",
+            project='test-suite',
+            src_dir=testsuite_srcdir,
+            alwaysUseLatest=True)
+
+        if testsuite_sollvevv:
+            f.addStep(steps.RemoveDirectory(name="SOLLVE V&V: Clean Source",
+                           dir=sollvevv_srcdir,
+                           haltOnFailure=False,
+                           warnOnFailure=True,
+                           doStepIf=cleanBuildRequested))
+
+            f.addStep(steps.Git(name="SOLLVE V&V: Checkout",
+                    description="fetching",
+                    descriptionDone="fetch",
+                    descriptionSuffix="SOLLVE V&V",
+                    repourl='https://github.com/SOLLVE/sollve_vv.git',
+                    workdir=sollvevv_srcdir,
+                    alwaysUseLatest=True))
+
+        f.addStep(steps.RemoveDirectory(name="Test-Suite: Clean Build",
+                           dir=testsuite_builddir,
+                           haltOnFailure=False,
+                           warnOnFailure=True))
+
+        testsuite_options = [
+            "-DCMAKE_BUILD_TYPE=Release",
+            "-DTEST_SUITE_LIT_FLAGS=-vv;-s;-j6;-o;report.json",
+            "-DTEST_SUITE_EXTRA_C_FLAGS=-gline-tables-only",
+            "-DTEST_SUITE_EXTRA_CXX_FLAGS=-gline-tables-only",
+            WithProperties("-DCMAKE_C_COMPILER=" + clangexe),
+            WithProperties("-DCMAKE_CXX_COMPILER=" + clangxxexe),
+            WithProperties("-DTEST_SUITE_LIT=" + litexe),
+        ]
+        if testsuite_sollvevv:
+            testsuite_options += [
+                "-DTEST_SUITE_SUBDIRS=External/sollve_vv",
+                "-DTEST_SUITE_SOLLVEVV_ROOT=../" + sollvevv_srcdir,
+            ]
+        testsuite_options += extraTestsuiteCmakeArgs
+        f.addStep(CmakeCommand(name="Test-Suite: Configure",
+                            description="configuring",
+                            descriptionDone="configure",
+                            descriptionSuffix="Test-Suite",
+                            generator='Ninja',
+                            path='../' + testsuite_srcdir,
+                            workdir=testsuite_builddir,
+                            options=testsuite_options,
+                            haltOnFailure=True,
+                            logfiles={
+                                'CMakeCache.txt'   : 'CMakeCache.txt',
+                            }))
+
+        f.addStep(NinjaCommand(name="Test-Suite: Compile",
+                            description="compiling",
+                            descriptionDone="compile",
+                            descriptionSuffix="Test-Suite",
+                            options=['-k0'],      # Continue building; programs that don't compile will fail with NOEXE.
+                            haltOnFailure=False,
+                            flunkOnFailure=False, # SOLLVE V&V contains tests that clang have not been implemented yet.
+                            warnOnFailure=True,
+                            workdir=testsuite_builddir))
+
+        merged_env.update({
+            'LD_LIBRARY_PATH': WithProperties(libdir + ':${LD_LIBRARY_PATH}')
+            })
+        f.addStep(LitTestCommand(name="Test-Suite: Run",
+                            description="running",
+                            descriptionDone="run",
+                            descriptionSuffix="Test-Suite",
+                            command=[WithProperties(litexe), '-vv', '-s',  '-j6', '-o','report.json', '.'],
+                            haltOnFailure=False,
+                            flunkOnFailure=False, # SOLLVE V&V contains tests that clang have not been implemented yet.
+                            warnOnFailure=True,
+                            workdir=testsuite_builddir,
+                            env=merged_env,
+                            logfiles={
+                                'test.log'   : 'test.log',
+                                'report.json': 'report.json'
+                            }))
 
     return f