[zorg] Adjust libcxx buildbot config on AIX

[Xiangling_L]

Summary

Enable libcxx project only

[Xiangling_L]

ping.

[daltenty]

buildbot/osuosl/master/config/builders.py

1352

See comment below.

1353

I think this should be a binary option to select the new "Unified Standalone" build configuration. We can then just set the correct src_root inside the factory rather than having to pass this explicitly.

zorg/buildbot/builders/LibcxxAndAbiBuilder.py

85

We could just append to LLVM_ENABLE_RUNTIMES instead here, if we are in the Unified Standalone build configuration

[Xiangling_L]

Use a boolean flag instead to control if building a unified standalone runtimes;

[daltenty]

LGTM, thanks!

[gkistanova]

I'm not sure I follow what are you trying to achieve by changing the src_root.

Changing src_root would not give you ../llvm/runtimes, it simply changes where the source code gets checked out to. Like <buildbot-worker-directory>/<builder>/runtimes/llvm-project/<the rest of the monorepo tree>.

zorg/buildbot/builders/LibcxxAndAbiBuilder.py

15

There is a mistype here. Should be a comma at the end.
I have fixed this for you with 4cd2689758e95b6811dae41ef9abec92f71e03a7.

81

f.depends_on_projects could contain more than just runtimes.

[gkistanova]

It seems this patch needs more attention.
Please see my comments above.

[Xiangling_L]

In D97326#2599081, @gkistanova wrote:

I'm not sure I follow what are you trying to achieve by changing the src_root.

Changing src_root would not give you ../llvm/runtimes, it simply changes where the source code gets checked out to. Like <buildbot-worker-directory>/<builder>/runtimes/llvm-project/<the rest of the monorepo tree>.

Thanks for pointing this out. What we are trying to do is to specify runtimes project[https://github.com/llvm/llvm-project/tree/main/runtimes] as the source to cmake and enable the building of libcxx;libcxxabi;libunwind with -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi;libunwind" to build a standalone runtimes.

The cmake command I am expecting is something similar to:

cmake ../llvm/runtimes ' -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DCMAKE_AR=/usr/bin/ar -DLIBCXX_CXX_ABI=libcxxabi -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi;libunwind"

Could you point to me what is the proper way to do this?

[Xiangling_L]

Adjust the build_standalone implementation.

[Xiangling_L]

ping.

[Xiangling_L]

ping.

[gkistanova]

I have added the support you need to UnifiedTreeBuilder.

You could pass src_to_build_dir='runtimes' to UnifiedTreeBuilder.getLLVMBuildFactoryAndSourcecodeSteps( to have what you are after.

[Xiangling_L]

Pass runtimes as src_bo_build_dir when building standalone runtimes

[gkistanova]

Almost there.
One problem to fix.

zorg/buildbot/builders/LibcxxAndAbiBuilder.py

38

src_to_build_dir is not defined if the branch is not taken here.

41

How about something like

src_to_build_dir='runtimes' if build_standalone else None,

and getting rid of that if above?

Or if you prefer having if to highlight the logic, then you need to initialize src_to_build_dir to None right before that if.

[Xiangling_L]

Addressed the comments;

[gkistanova]

LGTM

[Xiangling_L]

In D97326#2651355, @gkistanova wrote:

LGTM

Thank you so much for the review. However we have some strategy adjustment at this moment and may need to further change our buildbot config. So I'd like to suspend this patch for the moment. Sorry for the inconvenience.

[Xiangling_L]

Adjust the libcxx buildbot config to enable libcxx project only;
But still keep adding build_standalone functionalities to LibcxxAndAbiBuilder;

[Xiangling_L]

ping.

[Xiangling_L]

ping.

[gkistanova]

Looks fine.

However, now the change to buildbot/osuosl/master/config/builders.py and the change to zorg/buildbot/builders/LibcxxAndAbiBuilder.py are not related.
Could you break them to 2 separate patches, please?

[Xiangling_L]

In D97326#2671773, @gkistanova wrote:

Looks fine.

However, now the change to buildbot/osuosl/master/config/builders.py and the change to zorg/buildbot/builders/LibcxxAndAbiBuilder.py are not related.
Could you break them to 2 separate patches, please?

Sure will do.

[Xiangling_L]

Split this patch into two;

[gkistanova]

LGTM

buildbot/osuosl/master/config/builders.py

# Libc++ builders.
     'workernames' : ['aix-ppc-libcxx'],
     'builddir': 'libcxx-libcxxabi-libunwind-ppc-aix',
     'factory' : LibcxxAndAbiBuilder.getLibcxxAndAbiBuilder(
                     env={'OBJECT_MODE': '32'},
                     cmake_extra_opts={
                         'CMAKE_BUILD_TYPE': 'Release',
                         'CMAKE_C_COMPILER': 'gcc',
                         'CMAKE_CXX_COMPILER': 'g++',
-                        'CMAKE_AR': '/usr/bin/ar'})},
+                        'CMAKE_AR': '/usr/bin/ar'},
+                    build_standalone=True)},
 
     {'name': "fuchsia-x86_64-linux",
     'tags'  : ["toolchain"],
     'workernames' :["fuchsia-debian-64-us-central1-a-1", "fuchsia-debian-64-us-central1-b-1"],
     'builddir': "fuchsia-x86_64-linux",
     'factory': FuchsiaBuilder.getFuchsiaToolchainBuildFactory()},
 
 # libc Builders.
 

zorg/buildbot/builders/LibcxxAndAbiBuilder.py

 import buildbot.steps.shell
 import buildbot.process.properties as properties
 
 from zorg.buildbot.commands.LitTestCommand import LitTestCommand
 from zorg.buildbot.commands.CmakeCommand import CmakeCommand
 from zorg.buildbot.process.factory import LLVMBuildFactory
 from zorg.buildbot.builders import UnifiedTreeBuilder
 
 def getLibcxxAndAbiBuilder(f=None, env=None,
                            cmake_extra_opts=None, lit_extra_opts=None,
                            lit_extra_args=None, check_libcxx_abilist=False,
                            check_libcxx_benchmarks=None,
                            depends_on_projects=None,
                            use_cache=None,
+                           build_standalone=False.
                            **kwargs):
 
     if env is None:
         env = {}
     if cmake_extra_opts is None:
         cmake_extra_opts = {}
     if lit_extra_opts is None:
         lit_extra_opts = {}
     if lit_extra_args is None:
         lit_extra_args = []
 
     if depends_on_projects is None:
         depends_on_projects = ['libcxx','libcxxabi','libunwind']
 
+    if build_standalone:
+        src_root = 'runtimes'
+    else:
         src_root = 'llvm'
+
     build_path = 'build'
 
     if f is None:
         f = UnifiedTreeBuilder.getLLVMBuildFactoryAndSourcecodeSteps(
                 depends_on_projects=depends_on_projects,
                 llvm_srcdir=src_root,
                 obj_dir=build_path,
                 **kwargs) # Pass through all the extra arguments.
 
     rel_src_dir = LLVMBuildFactory.pathRelativeTo(f.llvm_srcdir, build_path)
 
     # Specify the max number of threads using properties so LIT doesn't use
     # all the threads on the system.
     litTestArgs = '-vv --show-unsupported --show-xfail --threads=%(jobs)s'
     if lit_extra_args:
if 'LIBCXX_CXX_ABI' not in cmake_extra_opts:
         cmake_opts.append('-DLIBCXX_CXX_ABI=libcxxabi')
 
     # Nuke/remake build directory and run CMake
     f.addStep(buildbot.steps.shell.ShellCommand(
         name='rm.builddir', command=['rm', '-rf', build_path],
         workdir=".",
         haltOnFailure=False))
 
+    if build_standalone:
+       CmakeCommand.applyRequiredOptions(cmake_opts, [
+        ('-DLLVM_ENABLE_RUNTIMES=', ";".join(f.depends_on_projects)),
+        ])
+    else:
        CmakeCommand.applyRequiredOptions(cmake_opts, [
         ('-DLLVM_ENABLE_PROJECTS=', ";".join(f.depends_on_projects)),
         ])
 
     f.addStep(buildbot.steps.shell.ShellCommand(
         name='cmake', command=['cmake', rel_src_dir] + cmake_opts,
         haltOnFailure=True, workdir=build_path, env=env))
 
     # Build libcxxabi
     jobs_flag = properties.WithProperties('-j%(jobs)s')