Add enable_runtimes option to Clang builder

[DavidSpickett]

Summary

...and set aarch64-full-2stage to "auto" meaning
that compiler-rt will go in RUNTIMES.

The option already exists in the functions below
this it just needed adding to the arguments.

This is the start of moving all the bots that use compiler-rt
and friends to using ENABLE_RUNTIMES rather than ENABLE_PROJECTS.

(https://discourse.llvm.org/t/should-buildbots-switch-to-enable-runtimes-instead-of-enable-projects-for-compiler-rt/65042)

[DavidSpickett]

This requires https://reviews.llvm.org/D132438 to be relanded in some form. I tested the build with that patch applied.

Locally the build was fine but I will move the bot to silent before landing this to make handling any issues easier.

[gkistanova]

Hello David,

You have figured out that LLVMBuildFactoryalready supports proper handling of runtime projects, as described at https://github.com/llvm/llvm-zorg/blob/main/zorg/buildbot/process/factory.py#L24. Having enable_runtimes default to None and passing it through to LLVMBuildFactory is the right way to go.

The patch looks good in general, with a few issues to fix - please see my comments inline.

This patch changes the behavior of existing builders which use the ClangBuilder.getClangCMakeBuildFactory, though. Did you do an impact analyses by any chance? Is any of the existing builder affected?

buildbot/osuosl/master/config/builders.py

470

Don't you want to skip the enable_runtimes arg and let the default do the trick?

zorg/buildbot/builders/ClangBuilder.py

169–171

Missing comma at the and of this line?

185–186

Missing comma at the and of this line?

275–276

Missing comma before enable_runtimes param?

[DavidSpickett]

This patch changes the behavior of existing builders which use the ClangBuilder.getClangCMakeBuildFactory, though. Did you do an impact analyses by any chance? Is any of the existing builder affected?

I don't think that it does, perhaps I am mistaken. My intent is that only clang-aarch64-full-2stage would change.

buildbot/osuosl/master/config/builders.py

470

The default enable_runtimes to getClangCMakeBuildFactory is None.
That calls _getClangCMakeBuildFactory passing on that value.
That creates an LLVMBuildFactory passing on None, that then does this check:

enable_runtimes = kwargs.pop('enable_runtimes', None)
# If specified, we either got a givem list of
# enabled runtimes, or "auto", or "all".

if enable_runtimes is None:
    # For the backward compatibility, we do not use
    # enable_runtimes unless it is requested explicitly.
    self.enable_runtimes = frozenset([])

Prior to this change, no value is passed down and we use LLVMBuildFactory's default, which is None as shown in the code above.

So existing builders were ending up with None and now they just do that by passing down a None from higher up. This one bot then overrides that with "auto" to opt into the runtimes build.

Perhaps you meant that the default to getClangCMakeBuildFactory could be changed to "auto" and yes, eventually I'd like that to happen. For now, as you pointed out, we need to limit the impact.

This first builder is one we (Linaro) control. Once that's working I'll work on our other bots and reach out to the other owners to get them switched over / explicitly marked as not wanting to use the runtimes build.

[DavidSpickett]

Add missing commas.

[DavidSpickett]

And somehow Arc included the entire world in that last update, will fix.

[DavidSpickett]

Attempt to fix the diff.

[DavidSpickett]

Update the diff the old fashioned way.

[DavidSpickett]

Apologies for the spam folks :)

[gkistanova]

LGTM.

[DavidSpickett]

Landed as https://github.com/llvm/llvm-zorg/commit/ca793509f5a9b26395400dcb92b344b1e055e949. Testing will be done on a silent bot.

buildbot/osuosl/master/config/builders.py

# Clang builders.
     {'name' : "clang-aarch64-full-2stage",
     'tags'  : ["clang"],
     'workernames' : ["linaro-clang-aarch64-full-2stage"],
     'builddir': "clang-aarch64-full-2stage",
     'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     clean=False,
                     checkout_flang=True,
                     checkout_lld=False,
+                    enable_runtimes="auto",
                     useTwoStage=True,
                     testStage1=False,
                     runTestSuite=True,
                     env={
                         'NO_STOP_MESSAGE':'1', # For Fortran test-suite
                     },
                     testsuite_flags=[
                         '--cppflags', '-mcpu=cortex-a57',

zorg/buildbot/builders/ClangBuilder.py

def getClangCMakeBuildFactory(
             extra_cmake_args=None,
 
             # Extra repositories
             checkout_clang_tools_extra=True,
             checkout_compiler_rt=True,
             checkout_lld=True,
             checkout_libcxx=False,
             checkout_flang=False,
-            checkout_test_suite=False):
+            checkout_test_suite=False,
+
+            enable_runtimes=None):
     return _getClangCMakeBuildFactory(
                clean=clean, checks=checks, cmake=cmake, jobs=jobs, vs=vs,
                vs_target_arch=vs_target_arch, useTwoStage=useTwoStage,
                testStage1=testStage1, stage1_config=stage1_config,
                stage2_config=stage2_config, runTestSuite=runTestSuite,
                nt_flags=nt_flags, testsuite_flags=testsuite_flags,
                submitURL=submitURL, testerName=testerName,
                env=env, extra_cmake_args=extra_cmake_args,
                checkout_clang_tools_extra=checkout_clang_tools_extra,
                checkout_lld=checkout_lld,
                checkout_compiler_rt=checkout_compiler_rt,
                checkout_libcxx=checkout_libcxx,
                checkout_flang=checkout_flang,
-               checkout_test_suite=checkout_test_suite)
+               checkout_test_suite=checkout_test_suite,
+               enable_runtimes=enable_runtimes)
 
 def _getClangCMakeBuildFactory(
             clean=True,
             checks=None,
             cmake='cmake',
             jobs=None,
 
             # VS tools environment variable if using MSVC. For example,
def _getClangCMakeBuildFactory(
             # Extra repositories
             checkout_clang_tools_extra=True,
             checkout_compiler_rt=True,
             checkout_lld=True,
             checkout_libcxx=False,
             checkout_test_suite=False,
             checkout_flang=False,
 
+            enable_runtimes=None,
+
             # Upload artifacts to Google Cloud Storage (for the llvmbisect tool)
             stage1_upload_directory=None,
 
             # Use a lower compression level to generate the build-cache package faster
             # default is 6 according to documentation
             xz_compression_factor=6,
             use_pixz_compression=False,
 
if checkout_libcxx:
         depends_on_projects.append('libcxxabi')
         depends_on_projects.append('libunwind')
     if checkout_flang:
         depends_on_projects.append('flang')
         depends_on_projects.append('mlir')
 
     f = LLVMBuildFactory(
             depends_on_projects=depends_on_projects,
-            llvm_srcdir='llvm')
+            llvm_srcdir='llvm',
+            enable_runtimes=enable_runtimes)
 
     # Checkout the latest code for LNT
     # and the test-suite separately. Le's do this first,
     # so we wouldn't poison got_revision property.
     if runTestSuite or checkout_test_suite:
         f.addGetSourcecodeForProject(
             project='lnt',
             src_dir='test/lnt',