[Zorg] Use ccache instead incremental build for openmp-offload-cuda-runtime.

[Meinersbur]

Summary

Building with LLVM_ENABLE_RUNTIMES=openmp has the disadvantage that if clang changes, the runtime that was built with the previous clang is not rebuilt. That is, if the runtime is miscompiling due to a change in clang, this will only be detected in the next clean build.

For instance, commit rG1100e4aafea233bc8bbc307c5758a7d287ad3bae caused the libomptarget device runtime to miscompile, but the openmp-offload-cuda-runtime builder shows it as green. Tests only started failing with the next clean build. In production, this would have blamed the wrong commit.

In contrast, the openmp-offload-cuda-project builder started failing with the expected commit.

Instead of building incrementally, use ccache to avoid this problem.

[tianshilei1992]

Given the fact it's not heavy to build OpenMP, I think a clean build every time should be fine. LGTM but better to ask BB owner to take a look.

Speaking of this, is it possible to integrate the result into Phabricator?

[ronlieb]

we will desire the same behavior for looming amdgpu buildbot, and we are closely following the cuda openmp buildbot.

[gkistanova]

LGTM

[Meinersbur]

In D106781#2905360, @tianshilei1992 wrote:

Given the fact it's not heavy to build OpenMP, I think a clean build every time should be fine.

ccache hashes the compiler executable. That is, if it turns out to be the same binary, it will re-use the cached result even for the deviceRTL.

LGTM but better to ask BB owner to take a look.

That would be me.

Speaking of this, is it possible to integrate the result into Phabricator?

The pre-merge buildkite already uses clean builds with ccache (or sccache). For small changes the incremental build typically is still by some factors faster than the ccache builds which is why I would prefer to keep incremental builds for openmp-offload-cuda-project. This also allows us to better identify problems that occur only in one of the builds (such as incorrect builds dependences) and still get frequent rebuilds.

This is a general problem buildbots using LLVM_ENABLE_RUNTIMES builds and might be fixed some day by e.g. clobbering the runtimes build dir after clang had to be recompiled.

[tianshilei1992]

In D106781#2905956, @Meinersbur wrote:

In D106781#2905360, @tianshilei1992 wrote:

Speaking of this, is it possible to integrate the result into Phabricator?

The pre-merge buildkite already uses clean builds with ccache (or sccache). For small changes the incremental build typically is still by some factors faster than the ccache builds which is why I would prefer to keep incremental builds for openmp-offload-cuda-project. This also allows us to better identify problems that occur only in one of the builds (such as incorrect builds dependences) and still get frequent rebuilds.

This is a general problem buildbots using LLVM_ENABLE_RUNTIMES builds and might be fixed some day by e.g. clobbering the runtimes build dir after clang had to be recompiled.

I mean, not about this patch. To integrate the execution result on GPUs into Phabricator. Currently we don't have test run on GPUs I think.

[Meinersbur]

In D106781#2906076, @tianshilei1992 wrote:

I mean, not about this patch. To integrate the execution result on GPUs into Phabricator. Currently we don't have test run on GPUs I think.

You can request it at https://github.com/google/llvm-premerge-checks/issues, but it will at most be possible with host offloading (I assume the BuildKite bots don't have GPUs) and only once they pass reliably.

buildbot/osuosl/master/config/builders.py

'factory' : OpenMPBuilder.getOpenMPCMakeBuildFactory(
                         ],
                     )},
 
     {'name' : "openmp-offload-cuda-runtime",
     'tags'  : ["openmp"],
     'workernames' : ["minipc-1050ti-linux"],
     'builddir': "openmp-offload-cuda-runtime",
     'factory' : OpenMPBuilder.getOpenMPCMakeBuildFactory(
-                        clean=False,
+                        clean=True,
                         enable_runtimes=['openmp'],
                         extraCmakeArgs=[
                                 "-DCUDA_TOOLKIT_ROOT_DIR=/opt/cuda",
                                 "-DLIBOMPTARGET_BUILD_NVPTX_BCLIB=ON",
                                 "-DCLANG_ENABLE_STATIC_ANALYZER=OFF",
                                 "-DCLANG_ENABLE_ARCMT=OFF",
                                 "-DCLANG_ENABLE_OBJC_REWRITER=OFF",
                                 "-DLLVM_TARGETS_TO_BUILD=X86;NVPTX",
                                 "-DLLVM_ENABLE_LLD=ON",
                                 '-DLLVM_PARALLEL_LINK_JOBS=2',
+                                "-DCMAKE_C_COMPILER_LAUNCHER=ccache",
+                                "-DCMAKE_CXX_COMPILER_LAUNCHER=ccache",
                             ],
                         install=True,
                         testsuite=True,
                         testsuite_sollvevv=True,
                         extraTestsuiteCmakeArgs=[
                             "-DTEST_SUITE_SOLLVEVV_OFFLOADING_CFLAGS=-fopenmp-targets=nvptx64-nvidia-cuda;--cuda-path=/opt/cuda",
                             "-DTEST_SUITE_SOLLVEVV_OFFLOADING_LDFLAGS=-fopenmp-targets=nvptx64-nvidia-cuda;--cuda-path=/opt/cuda",
                         ],