Changing the ppc64[be|le] bots to TestSuiteBuilders

[kamaub]

Summary

Update the ppc64 lnt bots to use cmake and ninja to build and run test-suite

To achieve this with minimal impact, most of the functionality from UnifiedTreeBuilder.py
is copied into TestSuiteBuilder.py instead of importing it so that the step naming convention
and ninja environment variables can be modified without impacting all the bots that
use getCmakeBuildFactory().

Bots are renamed to reflect this change:

• clang-ppc64le-linux-lnt => clang-ppc64le-linux-test-suite
• clang-ppc64be-linux-lnt => clang-ppc64be-linux-test-suite

The clang-ppc64le-rhel bot is updated to run check-all instead of check on the llvm-lit testing step.

[kamaub]

Planning to move away from the UnifiedTreeBuilder for our TestSuiteBuilder

[kamaub]

This update removes the edits made to the TestSuiteBuilder, as they are not
critical to the PR, we will not be moving away from the UnifiedTreeBuilder
as a base for the TestSuiteBuilder.

[amyk]

I think the patch makes sense - just had a question I wanted to ask.

buildbot/osuosl/master/config/builders.py

689

Can you remind me why we're removing the check here again? Does this correspond to the 'ninja check' that we previously discussed?

[kamaub]

buildbot/osuosl/master/config/builders.py

689

Yes, on https://reviews.llvm.org/source/zorg/browse/main/zorg/buildbot/builders/UnifiedTreeBuilder.py$295 , the bot is set to run ninja check-all as most general bots and certainly our ppc bots do, not sure why we specifically chose to make this ninja check but with this the bot is allowed to use the default as it should.

[gkistanova]

Thanks, Kamau!
LGTM.

I assume you renamed the existing workers keeping the secret part the same respectively. Please send me the new set of credentials otherwise, would you?

[kamaub]

Thank you!

In D128273#3623225, @gkistanova wrote:

I assume you renamed the existing workers keeping the secret part the same respectively. Please send me the new set of credentials otherwise, would you?

Yes, let us keep the secret part the same for both of them.

When is the next build master restart scheduled for? I would like to turn these two workers off at the end of the day before, so I don't spam incorrect credentials again like with the flang and mlir worker.

[gkistanova]

Please feel free to connect with the new names.

buildbot/osuosl/master/config/builders.py

'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     clean=False,
                     checkout_lld=False,
                     useTwoStage=False,
                     stage1_config='Release',
                     extra_cmake_args=['-DLLVM_ENABLE_ASSERTIONS=ON',
                                       '-DLLVM_PARALLEL_LINK_JOBS=4',
                                       '-DLLVM_TARGETS_TO_BUILD=Mips'])},
 
-    {'name' : "clang-ppc64be-linux-lnt",
+    {'name' : "clang-ppc64be-linux-test-suite",
     'tags'  : ["clang", "ppc"],
-    'workernames' : ["ppc64be-clang-lnt-test"],
-    'builddir': "clang-ppc64be-lnt",
-    'factory' : ClangBuilder.getClangCMakeBuildFactory(
-                    clean=False,
-                    checkout_lld=False,
-                    useTwoStage=False,
-                    runTestSuite=True,
-                    stage1_config='Release',
-                    nt_flags=['--threads=16', '--build-threads=16'],
-                    extra_cmake_args=[
+    'workernames' : ["ppc64be-clang-test-suite"],
+    'builddir': "clang-ppc64be-test-suite",
+    'factory' : TestSuiteBuilder.getTestSuiteBuildFactory(
+                    depends_on_projects=["llvm", "clang", "clang-tools-extra",
+                                         "compiler-rt"],
+                    extra_configure_args=[
                         "-DLLVM_ENABLE_ASSERTIONS=ON",
+                        "-DCMAKE_BUILD_TYPE=Release",
+                        "-DLLVM_LIT_ARGS=-v",
                         "-DLLVM_CCACHE_BUILD=ON"])},
 
     {'name' : "clang-ppc64be-linux-multistage",
     'tags'  : ["clang", "ppc"],
     'workernames' : ["ppc64be-clang-multistage-test"],
     'builddir': "clang-ppc64be-multistage",
     'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     clean=False,
                     checkout_lld=False,
                     useTwoStage=True,
                     stage1_config='Release',
                     stage2_config='Release',
                     extra_cmake_args=[
                         "-DLLVM_ENABLE_ASSERTIONS=ON",
                         "-DLLVM_CCACHE_BUILD=ON"])},
 
-    {'name' : "clang-ppc64le-linux-lnt",
+    {'name' : "clang-ppc64le-linux-test-suite",
     'tags'  : ["clang", "ppc", "ppc64le"],
-    'workernames' : ["ppc64le-clang-lnt-test"],
-    'builddir': "clang-ppc64le-lnt",
-    'factory' : ClangBuilder.getClangCMakeBuildFactory(
-                    clean=False,
-                    checkout_lld=False,
-                    useTwoStage=False,
-                    runTestSuite=True,
-                    stage1_config='Release',
-                    nt_flags=['--threads=16', '--build-threads=16'],
-                    extra_cmake_args=[
+    'workernames' : ["ppc64le-clang-test-suite"],
+    'builddir': "clang-ppc64le-test-suite",
+    'factory' : TestSuiteBuilder.getTestSuiteBuildFactory(
+                    depends_on_projects=["llvm", "clang", "clang-tools-extra",
+                                         "compiler-rt"],
+                    extra_configure_args=[
                         "-DLLVM_ENABLE_ASSERTIONS=ON",
+                        "-DCMAKE_BUILD_TYPE=Release",
+                        "-DLLVM_LIT_ARGS=-v",
                         "-DLLVM_CCACHE_BUILD=ON"])},
 
     {'name' : "clang-ppc64le-linux-multistage",
     'tags'  : ["clang", "ppc", "ppc64le"],
     'workernames' : ["ppc64le-clang-multistage-test"],
     'builddir': "clang-ppc64le-multistage",
     'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     clean=False,
                     checkout_lld=False,
                     useTwoStage=True,
                     stage1_config='Release',
                     stage2_config='Release',
                     extra_cmake_args=[
                         '-DLLVM_ENABLE_ASSERTIONS=ON',
                         '-DBUILD_SHARED_LIBS=ON',
                         '-DLLVM_CCACHE_BUILD=ON'])},
 
     {'name' : "clang-ppc64le-rhel",
     'tags'  : ["clang", "ppc", "ppc64le"],
     'workernames' : ["ppc64le-clang-rhel-test"],
     'builddir': "clang-ppc64le-rhel",
     'factory' : TestSuiteBuilder.getTestSuiteBuildFactory(
                     depends_on_projects=["llvm", "clang", "clang-tools-extra", "lld", "compiler-rt"],
-                    checks=["check"],
                     enable_runtimes="auto",
                     extra_configure_args=[
                         "-DLLVM_ENABLE_ASSERTIONS=On", "-DCMAKE_C_COMPILER=clang",
                         "-DCMAKE_CXX_COMPILER=clang++",
                         "-DCLANG_DEFAULT_LINKER=lld",
                         "-DCMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN:PATH=/opt/rh/devtoolset-7/root/usr",
                         "-DCMAKE_CXX_COMPILER_EXTERNAL_TOOLCHAIN:PATH=/opt/rh/devtoolset-7/root/usr",
                         "-DLLVM_BINUTILS_INCDIR=/usr/include", "-DBUILD_SHARED_LIBS=ON", "-DLLVM_ENABLE_WERROR=ON",

buildbot/osuosl/master/config/workers.py

return [
         # MIPS Loongson-3A R4 (Loongson-3A4000) 64-bit little endian (mips64el)
         create_worker("debian-tritium-mips64el", properties={'jobs': 1}, max_builds=1),
 
         # Motorola 68k 32-bit big endian (m68k)
         create_worker("debian-akiko-m68k", properties={'jobs': 1}, max_builds=1),
         create_worker("suse-gary-m68k-cross", properties={'jobs': 4}, max_builds=1),
 
         # POWER7 PowerPC big endian (powerpc64)
-        create_worker("ppc64be-clang-lnt-test", properties={'jobs': 16, 'vcs_protocol': 'https'}, max_builds=1),
+        create_worker("ppc64be-clang-test-suite", max_builds=1),
         create_worker("ppc64be-clang-multistage-test", properties={'jobs': 16}, max_builds=1),
         create_worker("ppc64be-sanitizer", properties={'jobs': 16}, max_builds=1),
 
         # POWER 8 PowerPC little endian (powerpc64le)
-        create_worker("ppc64le-clang-lnt-test", properties={'jobs': 8}, max_builds=1),
+        create_worker("ppc64le-clang-test-suite", max_builds=1),
         create_worker("ppc64le-clang-multistage-test", properties={'jobs': 8}, max_builds=1),
         create_worker("ppc64le-sanitizer", properties={'jobs': 4}, max_builds=1),
         create_worker("ppc64le-lld-multistage-test", max_builds=1),
         create_worker("ppc64le-clang-rhel-test", properties={'jobs': 192}, max_builds=1),
         create_worker("ppc64le-flang-rhel-test", max_builds=1),
         create_worker("ppc64le-mlir-rhel-test", max_builds=1),
 
         # SPARC 64-bit big endian (sparc64)