Page MenuHomePhabricator

[zorg] [PowerPC] Limit number of threads to 256 on ppc64le-sanitizer bot
AbandonedPublic

Authored by saghir on Sep 8 2020, 2:32 PM.

Details

Summary

By default all threads available on the machines are being used
by the Buildbot. This patch limits the use of threads to 256.

Event Timeline

saghir created this revision.Sep 8 2020, 2:32 PM
saghir requested review of this revision.Sep 8 2020, 2:32 PM
saghir updated this revision to Diff 290587.Sep 8 2020, 2:40 PM

fix indentation

saghir edited the summary of this revision. (Show Details)Sep 8 2020, 2:42 PM
saghir added reviewers: nemanjai, stefanp.
saghir added a project: Restricted Project.

This is definitely needed as it is really overloading the machine. However I can't really approve it as I am not familiar with the Python code here.

buildbot/osuosl/master/config/builders.py
1201

I don't see this argument being passed to any other calls to SanitizerBuilder.getSanitizerBuildFactory() but I assume it works the same as it does with other factory functions. Would like to get some confirmation from someone more familiar though.

saghir added inline comments.Sep 8 2020, 2:56 PM
buildbot/osuosl/master/config/builders.py
1201

Thanks, I will wait for someone who is more familiar with the code to take a look at it.

Unfortunately, SanitizerBuilder.getSanitizerBuildFactory does not accept accept extra_configure_args argument and does not pass it down to the script.

You would need to propose a patch to support this.

I also do not see the zorg/buildbot/builders/sanitizers/buildbot_selector.py accepting and passing further additional cmake args, so you would likely need to add support there as well.

It seems that most builders do have the extra_configure_args parameter but the builder that we use here does not.
You may have to add the parameter to the build factory to get this to work.

buildbot/osuosl/master/config/builders.py
1201

I don't think that this is going to work this easily. It doesn't look like extra_configure_args is part of the call to getSanitizerBuildFactory().
From SanitizerBuilder.py:

def getSanitizerBuildFactory(
    clean=False,
    depends_on_projects=None,
    env=None,
    timeout=1200):

Whereas other builders do have it. For example in ClangLTOBuilder.py:

def getClangWithLTOBuildFactory(
           depends_on_projects = None,
           clean = False,
           jobs  = None,
           extra_configure_args = None,
... <more options here> ...

You may have to add the parameter to getSanitizerBuildFactory and then use it in the body of the function.

saghir added a comment.Sep 9 2020, 7:02 AM

Thank you for the inputs everyone. I will take a look at the changes needed to be done to make this work. Thanks!

zacharyselk added inline comments.Sep 9 2020, 7:43 AM
buildbot/osuosl/master/config/builders.py
1202

You may also want to add -GNinja to the extra_configure_args.

saghir abandoned this revision.Sep 9 2020, 11:15 AM

Since significantly different changes are needed to make this work, another patch will be posted for review, closing this one.