This is an archive of the discontinued LLVM Phabricator instance.

[lit] Enqueue tests on a separate thread to not hit limits on parallel queues
ClosedPublic

Authored by filcab on Feb 25 2016, 7:47 AM.

Details

Summary

The current implementation just hangs if tests exceed 2^^15, at least
on Mac OS X. This might happen with a ninja check-all if one has a bunch of
llvm projects (especially sanitizers + iOS support)

Diff Detail

Repository
rL LLVM

Event Timeline

filcab updated this revision to Diff 49062.Feb 25 2016, 7:47 AM
filcab retitled this revision from to [lit] Enqueue tests on a separate thread to not hit limits on parallel queues.
filcab updated this object.
filcab added reviewers: delcypher, bkramer.
filcab added a subscriber: llvm-commits.

Interesting. I didn't know such a limit.

silvas added a subscriber: silvas.Feb 25 2016, 9:11 PM

This turns out to be simpler than I was imagining. Nice to finally have this implemented! (I still remember when I initially ran into this when doing crazy things with lit, but thought it would be a bit nastier to fix)

I can't speak for the technical changes in this patch, but I will say that I too was experiencing a deadlock on OS X and this patch fixes the problem.

Anyone else wants to chime in or suggest changes?

bkramer edited edge metadata.Mar 11 2016, 3:09 AM

This at least needs a comment explaining the limit. Ideally with a python bug number if it is a python bug.

filcab added a subscriber: filcab.Mar 11 2016, 7:53 AM

No bug, and there's no specific limit in the docs (well, you can see
how it's implemented and check the limits of the Queue's innards).

multiprocessing.Queue is allowed to behave like this, though:
https://docs.python.org/2/library/multiprocessing.html#multiprocessing.Queue.put

"put(obj[, block[, timeout]])
Put obj into the queue. If the optional argument block is True (the
default) and timeout is None (the default), block if necessary until a
free slot is available."

Do you want me to add this to the commit message? I don't think it's
the most useful thing to have in a comment there, reminding people to
look at the docs. :-)

Thank you,

Filipe
filcab updated this revision to Diff 50639.Mar 14 2016, 1:35 PM
filcab edited edge metadata.

Update with review feedback.

This revision was automatically updated to reflect the committed changes.