[PowerPC] [Zorg] Change ppc64le-rhel bot to unified tree instead

[Conanap]

Summary

ppc64le-rhel bot is changed to use unifiedtree builder instead as it has
a factory method with cmake and ninja.

I'm not exactly sure about nt-flags in the original clang builder, so
I would like to see how the buildbot handles this for now. The bot
will be kept on staging for now.

[stefanp]

The point of the nt_flags= is to not overload the machine. We have noticed in the past that if we let the tests run all at the same time without a thread limit we sometimes get intermittent failing tests.
I think this is an option we should try to keep.

buildbot/osuosl/master/config/builders.py

624–625

There are a number of projects listed here for DLLVM_ENABLE_PROJECTS.
However, we only list llvm in depends_on_projects. I feel like the right way to do this would probably be to add the projects to the depends_on_projects.
Also, inside UnifiedTreeBuilder:

CmakeCommand.applyDefaultOptions(cmake_args, [
    ('-DLLVM_ENABLE_PROJECTS=', ";".join(f.depends_on_projects)),
    ])

626

Does this mean that every time the bot runs we will be doing a clean first? This seems like it will slow down the bot quite a bit.

[Conanap]

Changed clean to false, removed a flag in favour of passing into arguments.

[jsji]

buildbot/osuosl/master/config/builders.py

625

This may get same bug as ClangBuilder. Can you review https://reviews.llvm.org/D99685 or even integrate it into this, so that we will get compiler-rt changes in buildbot as well.

[gkistanova]

Thanks, Albion.

Please see my comments inline.

buildbot/osuosl/master/config/builders.py

625–629

No need to specify this. The build factory will take care of the runtimes. It is enough to specify it in the depends_on_projects.

626

No need to specify clean param explicitly, if you do not want to force a clean build on each commit. The default should do a good job for you.

627

Make this a list, please.

636–637

Your worker specifies 4 jobs. But you hard-code here 20 for testing. Could you elaborate why, please?

[Conanap]

buildbot/osuosl/master/config/builders.py

636–637

that's actually a really good point... I'll have to look into that. Thanks for the catch!

[Conanap]

Added compiler-rt compatibility, removed thread limit argument. Bot was tested on a local server.

[Conanap]

Put back -j20 as we're seeing too many threads being used.

[Conanap]

buildbot/osuosl/master/config/builders.py

625

This new patch was able to pick up the compiler-rt changes during testing, so your patch should no longer be needed :)

636–637

I believe we needed -j 20 to prevent too many threads from being utilized - that said, I'm looking at how many jobs we can use for build right now, FYI

[Conanap]

Updated worker to use 8 jobs

[Conanap]

Updated buildbot worker to use 192 jobs.

[Conanap]

@gkistanova I've tested this on a local test server and it seems to run alright. We've changed the job number to 192 and found it to be stable. However, the lld will remain at 20 as more jobs seem to cause some issues on our machines. Please take a look when you get a chance, thank you!

[gkistanova]

LGTM.

Before committing, could you make sure you are based on the current UnifiedTreeBuilder and LLVMBuildFactory, please?
If I remember the context correctly, you may need to specify enable_runtimes="auto", since the default behavior has been changed in https://reviews.llvm.org/rZORGf7b888457641 per request.

However, the lld will remain at 20 as more jobs seem to cause some issues on our machines.

Yes, I noticed that lld memory usage has increased recently.

[jsji]

We want to use runtime build for RHEL bots, so we should set `enable_runtimes=['compiler-rt'] to UnifiedTreeBuilder.

[jsji]

If I remember the context correctly, you may need to specify enable_runtimes="auto", since the default behavior has been changed in https://reviews.llvm.org/rZORGf7b888457641 per request.

Or yes, use enable_runtimes="auto" here should work too.

[Conanap]

Added enable_runtime="auto".

[Conanap]

Thanks @gkistanova ! I've added the option and tested it on a local server, seems to be pretty happy.
@jsji I've updated the patch, please take a look when you get a a chance. Thanks!

[jsji]

Thanks.

[Conanap]

Removed an unintended change

[Conanap]

patch did not auto close; was pushed.

[dblaikie]

In D101199#2759623, @Conanap wrote:

patch did not auto close; was pushed.

For future reference, I think the syntax you need for auto closing (arc creates this automatically) is:

Differential Revision: DXXXX

[dblaikie]

In D101199#2759916, @dblaikie wrote:

In D101199#2759623, @Conanap wrote:

patch did not auto close; was pushed.

For future reference, I think the syntax you need for auto closing (arc creates this automatically) is:

Differential Revision: DXXXX

Ah, nope, full URL:

Differential Revision: https://reviews.llvm.org/DXXXX

[Conanap]

ah maybe I missed the word Revision, thank you!

buildbot/osuosl/master/config/builders.py

'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     stage1_config='Release',
                     extra_cmake_args=[
                         "-DLLVM_ENABLE_ASSERTIONS=ON",
                         "-DPYTHON_EXECUTABLE:FILEPATH=/usr/local/bin/python3"])},
 
     {'name' : "clang-ppc64le-rhel",
     'tags'  : ["clang", "ppc", "ppc64le"],
     'workernames' : ["ppc64le-clang-rhel-test"],
     'builddir': "clang-ppc64le-rhel",
-    'factory' : ClangBuilder.getClangCMakeBuildFactory(clean=False,
-                    checkout_clang_tools_extra=True,
-                    checkout_compiler_rt=False,
-                    checkout_lld=True,
-                    checkout_libcxx=False,
-                    useTwoStage=False,
-                    runTestSuite=True,
-                    stage1_config='Release',
-                    nt_flags=['--threads=16', '--build-threads=16'],
-                    extra_cmake_args=[
-                        '-DLLVM_ENABLE_PROJECTS=clang;llvm;clang-tools-extra;lld',
-                        '-DLLVM_ENABLE_RUNTIMES=compiler-rt',
+    'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
+                    depends_on_projects=["llvm", "clang", "clang-tools-extra", "lld", "compiler-rt"],
+                    checks=["check"],
+                    enable_runtimes="auto",
+                    extra_configure_args=[
                         "-DLLVM_ENABLE_ASSERTIONS=On", "-DCMAKE_C_COMPILER=clang",
                         "-DCMAKE_CXX_COMPILER=clang++",
                         "-DCLANG_DEFAULT_LINKER=lld",
                         "-DCMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN:PATH=/opt/rh/devtoolset-7/root/usr",
                         "-DCMAKE_CXX_COMPILER_EXTERNAL_TOOLCHAIN:PATH=/opt/rh/devtoolset-7/root/usr",
                         "-DLLVM_BINUTILS_INCDIR=/usr/include", "-DBUILD_SHARED_LIBS=ON", "-DLLVM_ENABLE_WERROR=ON",
-                        '-DLLVM_LIT_ARGS=-vj 20'])},
+                        "-DCMAKE_BUILD_TYPE=Release"
+                        "-DLLVM_LIT_ARGS=-vj 20"])},
 
     {'name' : "clang-ppc64-aix-ppc64le",
     'tags'  : ["clang", "aix", "ppc", "ppc64le"],
     'workernames' : ["aix-ppc64-ppc64le"],
     'builddir': "clang-ppc64-aix-ppc64le",
     'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     clean=False,
                     checkout_clang_tools_extra=False,

buildbot/osuosl/master/config/workers.py

return [
         create_worker("ppc64be-sanitizer", properties={'jobs': 16}, max_builds=1),
 
         # POWER 8 PowerPC little endian (powerpc64le)
         create_worker("ppc64le-clang-test", properties={'jobs': 4}, max_builds=1),
         create_worker("ppc64le-clang-lnt-test", properties={'jobs': 8}, max_builds=1),
         create_worker("ppc64le-clang-multistage-test", properties={'jobs': 8}, max_builds=1),
         create_worker("ppc64le-sanitizer", properties={'jobs': 4}, max_builds=1),
         create_worker("ppc64le-lld-multistage-test", max_builds=1),
-        create_worker("ppc64le-clang-rhel-test", properties={'jobs': 4}, max_builds=1),
+        create_worker("ppc64le-clang-rhel-test", properties={'jobs': 192}, max_builds=1),
         create_worker("ppc64le-flang-mlir-rhel-test", max_builds=1),
 
         # SPARC 64-bit big endian (sparc64)
         create_worker("debian-stadler-sparc64", properties={'jobs': 4}, max_builds=1),
 
         # Ubuntu x86-64, Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
         create_worker("hexagon-build-02", properties={'jobs': 12, 'loadaverage': 32},
             max_builds=1),