Page MenuHomePhabricator

[zorg][Fuchsia] Run check-lld with 32 jobs
Needs ReviewPublic

Authored by leonardchan on Nov 6 2020, 10:13 AM.

Details

Summary

We're hitting EAGAIN on the upstream buildbots when constructing threads. Overall buildbot memory usage doesn't seem too high and we've set the number of allowed processes (NPROC) to unlimited. We can try reducing the number of jobs run specifically when running lld tests.

Diff Detail

Event Timeline

leonardchan created this revision.Nov 6 2020, 10:13 AM
leonardchan requested review of this revision.Nov 6 2020, 10:13 AM

I don't think this is going to help, lit tasks are running inside the console Ninja pool so they're already limited to a single job. Instead, we would need to pass -j to lit through LIT_OPTS environment variable, see http://llvm.org/docs/CommandGuide/lit.html#general-options.

Updated to use LIT_OPTS.

phosek added inline comments.Nov 6 2020, 12:02 PM
zorg/buildbot/builders/FuchsiaBuilder.py
148

I've never seen this option used with =, have you checked whether this works? The quotes shouldn't be needed either.

leonardchan added inline comments.
zorg/buildbot/builders/FuchsiaBuilder.py
148

I think = should work. Never used LIT_OPTS before and I just copied from an example I found in some lit tests. ninja check-clang seems to run much slower with this so I would *think* it's working. Removed it regardless since -j32 just seems cleaner.

Specifying 32 jobs as a hard coded value in the build factory does not look right. You may want to parametrize this somehow.

zorg/buildbot/builders/FuchsiaBuilder.py
138

I know that this is how it was, but since you changed this line, could you use the str.format() interface instead, please?

We are Python 3.x only now. No need to keep the backward compatibility.

148

Isn’t the job number specific to a particular builder hardware?
Or you are saying that it must be 32 at any builder?
At a quick glance this looks like a worker property.

leonardchan marked an inline comment as done.

Specifying 32 jobs as a hard coded value in the build factory does not look right. You may want to parametrize this somehow.

Updated such that the number of jobs is passed as an argument to getFuchsiaToolchainBuildFactory.

zorg/buildbot/builders/FuchsiaBuilder.py
148

From what I can tell, the job number can be freely adjusted to whatever value but is only limited by the number of cores on the hardware.

My intention was to have all current fuchsia toolchain builders to run at most 32 jobs for specifically the check-lld step. Is there a more proper way to set this other than in the builder factory?