[Zorg] Add timeouts to OpenMP tests.

[Meinersbur]

Summary

This patch sets an individual test timeout of 10 seconds. Due to bug llvm.org/PR51235 some tests sometimes take 20 minutes to executed. The timeout is meant to reduce the overall build time and allow detection over tests that take abnormal time to execute.

The timeout of 10s was suggested by @tianshilei1992 which current causes 7 tests to fail (See http://meinersbur.de:8011/#/builders/1). I don't know whether all of them are due to llvm.org/PR51235 or whether some regularly take longer than 10 seconds.

[tianshilei1992]

My development machine has NVIDIA GTX2080 with 8GB GPU memory. What my extra arguments are -j 4 --timeout=10. I don't know if you already set the number of parallel jobs, because some test cases really need a large amount of GPU memory, and 4 in my case can make sure that no false failure because of OOM. If you didn't set that, I suggest to do it.
As for the timeout, 10 seconds are long enough to run any existing case in my machine.

[Meinersbur]

In D107195#2917948, @tianshilei1992 wrote:

My development machine has NVIDIA GTX2080 with 8GB GPU memory. What my extra arguments are -j 4 --timeout=10. I don't know if you already set the number of parallel jobs, because some test cases really need a large amount of GPU memory, and 4 in my case can make sure that no false failure because of OOM.

The system has "only" 6 cores and no SMT/hyperthreads. That is, llvm-lit by default launches 6 threads. I haven't observed OOM errors yet.

[tianshilei1992]

LG

[Meinersbur]

Changed timeout to 90s

[Meinersbur]

After llvm.org/PR51235 has been fixed, I measured the individual test execution in the worker using --time-tests -j1. These are all tests that run for longer than 5s:

63.08s: libomp :: tasking/omp_taskyield.c
60.10s: libomp :: tasking/omp_task_final.c
50.09s: libomp :: tasking/omp_taskwait.c
50.09s: libomp :: tasking/omp_task.c
50.07s: libomp :: api/omp_get_wtime.c
10.09s: libomp :: tasking/omp_task_if.c
10.06s: libomp :: flush/omp_flush.c
6.37s: libomp :: barrier/omp_barrier.c
5.10s: libomptarget :: nvptx64-nvidia-cuda :: offloading/bug49334.cpp
5.05s: libomp :: tasking/taskdep_if0_2.c

Time with -j6 was slightly longer (maybe 20%) for tests running < 10s. To account for this and jitter, I changed the timeout to 90s. (http://meinersbur.de:8011 currently running fine with a timeout of 70s)

Waiting for @gkistanova to approve D107193 before I can reconnect my worker.

buildbot/osuosl/master/config/builders.py

'factory' : OpenMPBuilder.getOpenMPCMakeBuildFactory(
                             ],
                         install=True,
                         testsuite=True,
                         testsuite_sollvevv=True,
                         extraTestsuiteCmakeArgs=[
                             "-DTEST_SUITE_SOLLVEVV_OFFLOADING_CFLAGS=-fopenmp-targets=nvptx64-nvidia-cuda;--cuda-path=/opt/cuda",
                             "-DTEST_SUITE_SOLLVEVV_OFFLOADING_LDFLAGS=-fopenmp-targets=nvptx64-nvidia-cuda;--cuda-path=/opt/cuda",
                         ],
+                        extraLitArgs=['--timeout=90'],
                     )},
 
     {'name' : "openmp-offload-cuda-runtime",
     'tags'  : ["openmp"],
     'workernames' : ["minipc-1050ti-linux"],
     'builddir': "openmp-offload-cuda-runtime",
     'factory' : OpenMPBuilder.getOpenMPCMakeBuildFactory(
                         clean=True,
'factory' : OpenMPBuilder.getOpenMPCMakeBuildFactory(
                             ],
                         install=True,
                         testsuite=True,
                         testsuite_sollvevv=True,
                         extraTestsuiteCmakeArgs=[
                             "-DTEST_SUITE_SOLLVEVV_OFFLOADING_CFLAGS=-fopenmp-targets=nvptx64-nvidia-cuda;--cuda-path=/opt/cuda",
                             "-DTEST_SUITE_SOLLVEVV_OFFLOADING_LDFLAGS=-fopenmp-targets=nvptx64-nvidia-cuda;--cuda-path=/opt/cuda",
                         ],
+                        extraLitArgs=['--timeout=90'],
                     )},
 
 # OpenMP AMDGPU Builders
     {'name' : "openmp-offload-amdgpu-project",
     'tags'  : ["openmp"],
     'workernames' : ["omp-vega20-0"],
     'builddir': "openmp-offload-amdgpu-project",
     'factory' : OpenMPBuilder.getOpenMPCMakeBuildFactory(

zorg/buildbot/builders/OpenMPBuilder.py

def getOpenMPCMakeBuildFactory(
         test                = True,         # Test the built libraries.
         depends_on_projects = None,
         enable_runtimes     = None,
         extraCmakeArgs      = [],
         install             = False,
         testsuite           = False,
         testsuite_sollvevv  = False,
         extraTestsuiteCmakeArgs = [],
+        extraLitArgs        = [],
         **kwargs):
 
     # Prepare environmental variables. Set here all env we want everywhere.
     merged_env = {
         'TERM' : 'dumb' # Make sure Clang doesn't use color escape sequences.
                  }
     # Overwrite pre-set items with the given ones, so user can set anything.
     if env is not None:
f.addStep(
             env             = merged_env))
 
     # Configure LLVM and OpenMP (and Clang, if requested).
     cmake_args = ['-DCMAKE_BUILD_TYPE=Release', '-DLLVM_ENABLE_ASSERTIONS=ON']
     if ompt:
         cmake_args += ['-DLIBOMP_OMPT_SUPPORT=ON']
     if test:
         lit_args = '-vv --show-unsupported --show-xfail -j %s' % jobs
+        lit_args = ' '.join([lit_args] + extraLitArgs)
         cmake_args += [WithProperties('-DLLVM_LIT_ARGS=%s' % lit_args)]
     if install:
         cmake_args += [WithProperties('-DCMAKE_INSTALL_PREFIX=%(builddir)s/' + llvm_instdir)]
     cmake_args += extraCmakeArgs
 
     if f.enable_projects:
         CmakeCommand.applyDefaultOptions(cmake_args, [
             ('-DLLVM_ENABLE_PROJECTS=', ";".join(f.enable_projects)),