[OpenMP][LibC] Adds testing LibC GPU Mode for AMDGPU

[jplehr]

Summary

This adds testing LibC's GPU mode to the AMDGPU (experimental) buildbot.
The change requires the buildbot to also depend on the libc project and
pass-in a few more CMake flags.
Finally, it runs the additional check-libc target.

[jdoerfert]

buildbot/osuosl/master/config/builders.py

1888

can't we use native here (@jhuber6 ?)

[jhuber6]

buildbot/osuosl/master/config/builders.py

1888

Actually I never implemented that for libc, I probably should go ahead and do that. But I think for the buildbot it will help to be explicit, so if the GPU dies we don't silently start passing tests. theres a LIBC_GPU_TEST_ARCHITECTURE variable for that purpose.

[jplehr]

Rebase and review feedback

[jhuber6]

When do we want to land this in the buildbot?

[jplehr]

I wanted to do this today, my afternoon. This should give us some time to reasonably react if things don't go the way we want.

[gkistanova]

Could you elaborate on why did you specify LLVM_ENABLE_PROJECTS explicitly, and why it does not match what's in the depends_on_projects, please? The same question is for LLVM_ENABLE_RUNTIMES and enable_runtimes respectively.

With this patch the openmp-offload-amdgpu-runtime-experimental builder would not listen for compiler_rt changes, but would build that. Meaning false blame lists every time when a compiler_rt commit breaks a build. This does not look right. What am I missing?

[ronlieb]

I reverted the patch as it seems to be breaking our bot

[jplehr]

In D146722#4228797, @gkistanova wrote:

Could you elaborate on why did you specify LLVM_ENABLE_PROJECTS explicitly, and why it does not match what's in the depends_on_projects, please? The same question is for LLVM_ENABLE_RUNTIMES and enable_runtimes respectively.

With this patch the openmp-offload-amdgpu-runtime-experimental builder would not listen for compiler_rt changes, but would build that. Meaning false blame lists every time when a compiler_rt commit breaks a build. This does not look right. What am I missing?

Thanks for the comment and explanation. You're not missing anything, it was a mistake on my end. I'll fix it and update the patch.

[jplehr]

Addressed feedback: removed extra CMake flags to enable runtimes and projects manually. Instead rely on enable_runtimes and depends_on_projects lists from the buildbot factory.
The builtbot before specified the CMake flag to enable the openmp runtime manually (despite listing it in enable_runtimes), so I removed that completely.

If this is the preferred way to do it, we should change the AMDGPU-openmp buildbot (non-experimental) and also remove the manual CMake flag there.

[gkistanova]

buildbot/osuosl/master/config/builders.py

1877

Unless you need to handle a special case, you do not need to specify enable_runtimes. Just make sure you list all the dependencies in depends_on_projects (which seems you are), no matter if they are "projects" or "runtimes", and let the default machinery do the trick.

[jhuber6]

buildbot/osuosl/master/config/builders.py

1877

Will the default machinery put it in the right place? The libc project can go in both as far as I'm aware.

[gkistanova]

libc will be a runtime by default.

[jplehr]

buildbot/osuosl/master/config/builders.py

1877

I added these as enable_runtimes as the OpenMPBuilder will set the enable_runtimes=['openmp'] when no argument is given.

Should we then also change the OpenMPBuilder or is this part of the default machinery you mentioned?

[gkistanova]

buildbot/osuosl/master/config/builders.py

1877

As far as I can see OpenMPBuilder does the right thing. It passes through both depends_on_projects and enable_runtimes to LLVMBuildFactory and then uses the factory properties to set cmake params. Please see https://github.com/llvm/llvm-zorg/blob/91e9b7954ae867c3294b896e44fdc7ad498840b5/zorg/buildbot/builders/OpenMPBuilder.py#L75 and below for more details.

I added these as enable_runtimes the OpenMPBuilder will set the enable_runtimes=['openmp'] when no argument is given.

I do not think it would. Where do you see that in the code?

[jplehr]

Please let me know if this patch needs more revision after the changes I implemented, and thanks for all the help.

buildbot/osuosl/master/config/builders.py

1877

My bad, you are totally right. I must have confused the enable_projects parts in the OpenMPBuilder.
Thanks for your patience.

[jplehr]

Remvoed the enable_runtimes argument

[gkistanova]

Thanks for updating the patch, JP.

LGTM.

[jplehr]

It seems that the automation does not correctly enable libc or openmp as runtimes when only given as depend_on_projects.
The CMake invocation that actually happened was

cmake -G Ninja ../llvm.src/llvm -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON '-DLLVM_LIT_ARGS=-vv --show-unsupported --show-xfail -j 32' -DCMAKE_INSTALL_PREFIX=/home/ompworker/bbot/openmp-offload-amdgpu-runtime-experimental/llvm.inst -DCMAKE_BUILD_TYPE=Release -DCLANG_DEFAULT_LINKER=lld '-DLLVM_TARGETS_TO_BUILD=X86;AMDGPU' -DLLVM_ENABLE_ASSERTIONS=ON -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DLIBOMPTARGET_FOUND_AMDGPU_GPU=ON -DLLVM_LIBC_FULL_BUILD=ON -DLIBC_GPU_BUILD=ON -DLIBC_GPU_ARCHITECTURE=gfx906 -DLIBC_GPU_TEST_ARCHITECTURE=gfx906 '-DLLVM_ENABLE_PROJECTS=llvm;libc;clang;lld;compiler-rt;openmp'

So, it only enables those as projects. This leads to a compiler error as we actually need the Clang to build the runtimes.

Is this an error in the automation machinery that we should fix, or shall we go back to enabling those as enable_runtimes?

[gkistanova]

Please keep it as is. I'll take a look later today on what might be wrong.

[gkistanova]

OpenMPBuilder didn't do enable_runtimes="auto". This is fixed in https://github.com/llvm/llvm-zorg/commit/b86d771caf03cb7fe94d0d187cad25d9519ea584.

Could you commit this patch again, please? And do not revert even if something wouldn't work as expected. Since we are all looking at this thing, it would be a bit easier with less moving parts.

[jhuber6]

I'll land it since JP is asleep.

[ronlieb]

[AMD Official Use Only - General]

Thanks Joseph

buildbot/osuosl/master/config/builders.py

'factory' : OpenMPBuilder.getOpenMPCMakeBuildFactory(
                     )},
 
     {'name' : "openmp-offload-amdgpu-runtime-experimental",
     'tags'  : ["openmp"],
     'workernames' : ["omp-vega20-1"],
     'builddir': "openmp-offload-amdgpu-runtime-experimental",
     'factory' : OpenMPBuilder.getOpenMPCMakeBuildFactory(
                         clean=True,
-                        enable_runtimes=['openmp'],
-                        depends_on_projects=['llvm','clang','lld','openmp'],
+                        depends_on_projects=['llvm', 'clang', 'compiler-rt', 'libc', 'lld', 'openmp'],
                         extraCmakeArgs=[
                             "-DCMAKE_BUILD_TYPE=Release",
                             "-DCLANG_DEFAULT_LINKER=lld",
                             "-DLLVM_TARGETS_TO_BUILD=X86;AMDGPU",
                             "-DLLVM_ENABLE_ASSERTIONS=ON",
-                            "-DLLVM_ENABLE_RUNTIMES=openmp",
                             "-DCMAKE_C_COMPILER_LAUNCHER=ccache",
                             "-DCMAKE_CXX_COMPILER_LAUNCHER=ccache",
                             "-DLIBOMPTARGET_FOUND_AMDGPU_GPU=ON",
+                            "-DLLVM_LIBC_FULL_BUILD=ON",
+                            "-DLIBC_GPU_BUILD=ON",
+                            "-DLIBC_GPU_ARCHITECTURE=gfx906",
+                            "-DLIBC_GPU_TEST_ARCHITECTURE=gfx906",
                             ],
                         install=True,
                         testsuite=False,
                         testsuite_sollvevv=False,
                         extraTestsuiteCmakeArgs=[
                             "-DTEST_SUITE_SOLLVEVV_OFFLOADING_CFLAGS=-fopenmp;-fopenmp-targets=amdgcn-amd-amdhsa;-Xopenmp-target=amdgcn-amd-amdhsa;-march=gfx906",
                             "-DTEST_SUITE_SOLLVEVV_OFFLOADING_LDLAGS=-fopenmp;-fopenmp-targets=amdgcn-amd-amdhsa;-Xopenmp-target=amdgcn-amd-amdhsa;-march=gfx906",
                         ],
-                        add_lit_checks=["check-clang", "check-llvm", "check-lld"]
+                        add_lit_checks=["check-clang", "check-llvm", "check-lld", "check-libc"]
                     )},
 
 
 # Whole-toolchain builders.
 
     {'name': "fuchsia-x86_64-linux",
     'tags'  : ["toolchain"],
     'workernames' :["fuchsia-debian-64-us-central1-a-1", "fuchsia-debian-64-us-central1-b-1"],