[zorg] [PowerPC] Limit number of threads to 256 on ppc64le-sanitizer bot

[saghir]

Summary

By default all threads available on the machines are being used
by the Buildbot. This patch limits the use of threads to 256.

[saghir]

fix indentation

[nemanjai]

This is definitely needed as it is really overloading the machine. However I can't really approve it as I am not familiar with the Python code here.

buildbot/osuosl/master/config/builders.py

1201

I don't see this argument being passed to any other calls to SanitizerBuilder.getSanitizerBuildFactory() but I assume it works the same as it does with other factory functions. Would like to get some confirmation from someone more familiar though.

[saghir]

buildbot/osuosl/master/config/builders.py

1201

Thanks, I will wait for someone who is more familiar with the code to take a look at it.

[gkistanova]

Unfortunately, SanitizerBuilder.getSanitizerBuildFactory does not accept extra_configure_args argument and does not pass it down to the script.

You would need to propose a patch to support this.

I also do not see the zorg/buildbot/builders/sanitizers/buildbot_selector.py accepting and passing further additional cmake args, so you would likely need to add support there as well.

[stefanp]

It seems that most builders do have the extra_configure_args parameter but the builder that we use here does not.
You may have to add the parameter to the build factory to get this to work.

buildbot/osuosl/master/config/builders.py

1201

I don't think that this is going to work this easily. It doesn't look like extra_configure_args is part of the call to getSanitizerBuildFactory().
From SanitizerBuilder.py:

def getSanitizerBuildFactory(
    clean=False,
    depends_on_projects=None,
    env=None,
    timeout=1200):

Whereas other builders do have it. For example in ClangLTOBuilder.py:

def getClangWithLTOBuildFactory(
           depends_on_projects = None,
           clean = False,
           jobs  = None,
           extra_configure_args = None,
... <more options here> ...

You may have to add the parameter to getSanitizerBuildFactory and then use it in the body of the function.

[saghir]

Thank you for the inputs everyone. I will take a look at the changes needed to be done to make this work. Thanks!

[zacharyselk]

buildbot/osuosl/master/config/builders.py

1202

You may also want to add -GNinja to the extra_configure_args.

[saghir]

Since significantly different changes are needed to make this work, another patch will be posted for review, closing this one.

buildbot/osuosl/master/config/builders.py

def _get_sanitizer_builders():
           {'name': "sanitizer-ppc64be-linux",
            'slavenames' :["ppc64be-sanitizer"],
            'builddir': "sanitizer-ppc64be",
            'factory': SanitizerBuilder.getSanitizerBuildFactory(timeout=1800)},
 
           {'name': "sanitizer-ppc64le-linux",
            'slavenames' :["ppc64le-sanitizer"],
            'builddir': "sanitizer-ppc64le",
-           'factory': SanitizerBuilder.getSanitizerBuildFactory(timeout=1800)},
+           'factory': SanitizerBuilder.getSanitizerBuildFactory(
+                          timeout=1800,
+                          extra_configure_args=[
+                              "-DLLVM_LIT_ARGS=-v -j256"])},
 
           {'name': "sanitizer-windows",
            'slavenames' :["sanitizer-windows"],
            'builddir': "sanitizer-windows",
            'factory': AnnotatedBuilder.getAnnotatedBuildFactory(
              script="sanitizer-windows.py",
              depends_on_projects=["llvm", "clang", "lld", "compiler-rt"])},