Add ctrl-c support to parallel dotest.py.

[tfiala]

Summary

This change does the following:

• If Ctrl-C is hit once, the parallel dotest.py runner will stop any further work from being started, but will wait for the existing tests in progress to complete.
• If Ctrl-C is hit a second time, the parallel dotest.py will kill the process trees for the child worker threads and the inferior dotest.py processes they spawned.

In both cases, the report infrastructure is left intact to report on whatever work was completed before it stopped. Thus, if 42 tests ran when the test was keyboard interrupted, the results for those 42 tests will be displayed.

See https://llvm.org/bugs/show_bug.cgi?id=24709

[zturner]

Ctrl+C once doesn't work on Windows with this patch. It seems to continue starting up new processes, and after all of them are done, I'm left with the original set of Python processes not doing any work, just sitting there. If I Ctrl+C again everything does.

Note that when I press the first Ctrl+C, I don't even get the print that says First Keyboard Interrupt received.

[clayborg]

Looks fine. Zach, do you need to make some modifications for Windows? Or should we just check this in and you can fix windows with a separate patch?

[zturner]

I can try to put a little time into figuring out what's wrong. I'd rather
it not go in broken though, so let me take a look first and we can figure
out what to do after that.

[tfiala]

In D12651#240423, @zturner wrote:

Ctrl+C once doesn't work on Windows with this patch. It seems to continue starting up new processes, and after all of them are done, I'm left with the original set of Python processes not doing any work, just sitting there. If I Ctrl+C again everything does.

Okay. So its like the first Ctrl-C is essentially not even handled, but the second Ctrl-C does the nuclear option and kills everything. Is that right?

Note that when I press the first Ctrl+C, I don't even get the print that says First Keyboard Interrupt received.

That kinda sounds like these lines might be suspect:

# Shut off interrupt handling in the child process.
signal.signal(signal.SIGINT, signal.SIG_IGN)

(at line 224-225). If they're not doing anything on Windows, then it is possible that the Ctrl-C could get gobbled by one of the child multiprocessing workers (?)

I'll dig around and see if there's something different about how Windows handles disabling Ctrl-C. (If you know anything there, that'd be helpful as well).

Worst case we could check os type and skip the soft kill on NT?

[tfiala]

In D12651#240429, @zturner wrote:

I can try to put a little time into figuring out what's wrong. I'd rather
it not go in broken though, so let me take a look first and we can figure
out what to do after that.

Yep absolutely.

[zturner]

Hmm, yea the problem is worse than I thought. Even if I don't do Ctrl+C at
all (I just run the test suite like I always have and let it finish) it's
just leaving a bunch of stale processes at the end without them shutting
down on their own.

[clayborg]

I believe that python is also weird in that if you want to catch a CTRL+C it has to be executing python code (not in a python code the entered C/C++ code) otherwise the CTRL+C might not get to the python itself. This might be only a problem in the embedded python interpreter though so I could be wrong on this.

[clayborg]

Were you running dosep.py on Windows before as the way to run your test suite? Or were you running dotest.py (no multi-threading)?

[tfiala]

In D12651#240439, @zturner wrote:

Hmm, yea the problem is worse than I thought. Even if I don't do Ctrl+C at
all (I just run the test suite like I always have and let it finish) it's
just leaving a bunch of stale processes at the end without them shutting
down on their own.

Ok, and it doesn't do that prior to the patch? (i.e. this isn't just a case of the processes slowly falling off at the end?)

There's a point where all the worker processes are joined, which should mean they're done. So either the join is failing to complete (maybe because something in the process isn't wrapping up right), or the join isn't causing them to die.

Is the initial test runner itself finishing up?

[zturner]

Correct, it doesn't do that prior to the patch. It looks like it's never
exiting this loop:

try:
    for worker in workers:
        worker.join()
except:

either when a Ctrl+C happens, or when all the processes finish. I guess
it's stuck in there for some reason. So even the test runner doesn't
finish. Still investigating.

Also to answer Greg's question, yes we were using dosep before. The
problem seems to be related to swtiching from multiprocess.Pool to doing
the pumping manually.

[tfiala]

In D12651#240443, @zturner wrote:

Correct, it doesn't do that prior to the patch. It looks like it's never
exiting this loop:

try:
    for worker in workers:
        worker.join()
except:

either when a Ctrl+C happens, or when all the processes finish. I guess
it's stuck in there for some reason. So even the test runner doesn't
finish. Still investigating.

Okay let me go ahead and make a few changes that might help and resend the patch. Back in a few...

[tfiala]

Specify tight queue sizes for the job description queue and the job results queue.

This *might* stop unintentional queue blocking when adding items to the queue if they were sized too small by default. Might be a possible difference between Windows and OS X/Linux.

[tfiala]

Hey Zachary,

Can you give the latest patch a try? That might stop some unintentional blockage as adding items to those two queues would be blocking.

I'm going to be relocating home in a minute, but the other thing I'm going to try is to go back to the multiprocessing.Pool() and see if the Ctrl-C woes that others hit are maybe somehow not an issue for us. If that's the case, then maybe the Pool() piece will take care of whatever isn't happy on the Windows side with the manual pooling.

[zturner]

Tried out this patch, unfortunately I'm seeing the same thing. The very
first call to worker.join() is never returning.

It's unfortunate that it's so hard to debug this stuff, do you have any
suggestions for how I can try to nail down what the child dotest instance
is actually doing? I wonder if it's blocking somewhere in its script, or
if this is some quirk of the multiprocessing library's dynamic invocation /
whatever magic is does.

How much of an effort would it be to make the switch to threads now? The
main thing we'd have to do is get rid of all of the globals in dotest, and
make a DoTest class or something.

As a last resort, you can bring back the pool.map logic behind an OS check,
and use the multiprocess.Process logic for other platforms. But it would
be great to have less platform specific branches and logic, not more.

I'm willing to bet you could reproduce this on a Windows VM with a trash
script that all it does is create some processes using the same structure
of loop you're using here. Not sure if that's an option for you. I don't
mind the back and forth diagnostics either, just whatever is easiest for
you.

[tfiala]

In D12651#240480, @zturner wrote:

Tried out this patch, unfortunately I'm seeing the same thing. The very
first call to worker.join() is never returning.

It's unfortunate that it's so hard to debug this stuff, do you have any
suggestions for how I can try to nail down what the child dotest instance
is actually doing? I wonder if it's blocking somewhere in its script, or
if this is some quirk of the multiprocessing library's dynamic invocation /
whatever magic is does.

How much of an effort would it be to make the switch to threads now? The
main thing we'd have to do is get rid of all of the globals in dotest, and
make a DoTest class or something.

It's a bit more work than I want to take on right now. I think we really may want to keep the multiprocessing and just not exec out to dotest.py for a third-ish time for each inferior.

As a last resort, you can bring back the pool.map logic behind an OS check,
and use the multiprocess.Process logic for other platforms. But it would
be great to have less platform specific branches and logic, not more.

I'm willing to bet you could reproduce this on a Windows VM with a trash
script that all it does is create some processes using the same structure
of loop you're using here. Not sure if that's an option for you. I don't
mind the back and forth diagnostics either, just whatever is easiest for
you.

Yeah I will have to get a Windows LLDB setup going again. I've got one at home so I can try with that in the near future here. For the moment I'm going to throw up one with multiprocessing.Pool again and throw it over the wall after I make sure Ctrl-C is working with it here (OS X and Linux).

I will try to get the Pool-based version going now. Back as soon as I have that!

[tfiala]

The latest patch does the following:

• adds a test runner strategy layer for the isolated, concurrent test support (in dosep.py).

The currently-supported test runners are:

1. multiprocessing

This is a multiprocessing-based parallel test runner that supports ctrl-c through rolling its own worker pool. This one is known to currently have an issue on Windows.

2. multiprocessing-pool

This one is exactly like what previously existed - multiprocessing based using multiprocessing.Pool() to do the work. It does not handle Ctrl-C at all (and performs poorly, just like before, when the user does press Ctrl-C).

3. serial

This one is the same test runner used that does no concurrent test running but does use subprocess to provide process isolation to each of the test suite files.

It is not expected that a user specify test-runner-name: sane defaults are provided. The defaults used when no test runner name is given are:

• Use serial if the number of threads determined to be used is 1. (Likely due to the --threads argument on modern hardware).
• If the os.name is 'nt', use multiprocessing-pool.
• Otherwise, use multiprocessing.

@zturner - this should provide the net result of 'no behavior change' on Windows. If you have a chance to verify that with no special options, that'd be great.

Note I plan to make one more change to add in the threading-based test runner before I'm totally done here, so this is just WIP. Zachary, if you want to wait for the "threading" test runner, that seems reasonable. I hope to have that diff in later today.

@dawn, I did add in the logic that says 'if you specify -v and using the dosep-style test runner, assume --output-on-success is specified.

[tfiala]

That last patch also fixes a regression in the previous patch when --threads 1 was specified --- the serial test runner in dosep.py was busted and I missed that.

[zturner]

I won't be able to test this until Tuesday, but thanks for working on it

[tfiala]

In D12651#240633, @zturner wrote:

I won't be able to test this until Tuesday, but thanks for working on it

No worries. Have a great weekend!

(And maybe I'll have a working Windows setup before then).

[tfiala]

This change adds the threading-module-based parallel test runner strategy. It is not selected by default. It is pool-based.

The threading-pool and multiprocessing-pool strategies are 19.8% slower than the newer hand-wrapped ctrl-c-supporting, hand-wrapped multiprocessing pool approach. I'm going to keep the multiprocessing strategy turned on for everything except Windows NT.

I may later try to hand-wrap the pooling on the threading side as well, since it looks like the slowdown may actually be the Pool provided by multiprocessing.pool (which we were always using, so the multiprocessing test runner strategy represents a potential 20% win for same-class Linux boxes vs. the previous approach currently at top of tree). These numbers were on a 24-core Linux box using 24 processes/threads (the defaults for the box).

If I get this working on a Windows box with the current defaults, I'll check it in.

[tfiala]

Adds "threading" test-runner strategy, which mirrors the "multithreading" runner in terms of supporting Ctrl-C and hand-rolling the worker model. Like multiprocessing over multiprocessing-pool, threading outperforms threading-pool.

Both threading and multiprocessing operate at roughly the same speed (within 10s of ms) on a 24 core Linux box. Both these two test runners are substantially faster than the versions that use the Pool class --- the Pool-class versions are 20-25% slower than the hand-rolled worker pool versions, likely due to the more even on-demand scheduling and coarse grain and uneven nature of our tasks being run.

[tfiala]

@zturner, at this point you should be able to run this and see no change on Windows (assuming I did the os check correctly). The Windows test runner is set to be the previous multithreading-pool strategy. For everyone else, they'll get the multiprocessing strategy by default that supports Ctrl-C.

You can also check the threading-pool and the threading strategy to see if you get any speedup on Windows. The only speed difference I saw was that the -pool versions were slower than the non-pool versions, but otherwise no wall-clock difference in the threading vs. multiprocessing versions. I saw a slight (~1.4%) reduction in system time on the threading vs. multiprocessing, but that didn't translate to an overall speedup. Also, it is more difficult to get the Ctrl-C behavior correct in the threading vs. multiprocessing scenario. So unless there was a specific reason to want to use threading over multiprocessing (or the perf difference was significant on some other OS), I'd stick with multiprocessing-based.

On OS X there was at least one test that would hang each run, so I didn't get any kind of real timing numbers there since everything was always maxed out by the hanging test. We'll definitely stick with the multiprocessing test runner until we have some data that suggests a worthy reason for changing.

I'd like to get this in as soon as this is working on your end, Zachary, as I have other changes backed up behind it.

Thanks!

[zturner]

Adrian, can you verify this in the morning? Basically just trying to
ensure that ninja check-lldb still works as it did before. There's a
chance I'm going to be OOO tomorrow (or at the very best late) due to
something unexpected.

[amccarth]

After applying the patch, I get three additional test case failures on Windows. I'm trying to figure out now which ones.

Is there something specific I'm supposed to be trying, with regard to Ctrl+C itself.

[zturner]

If the patch is busted, pretty much every test should start failing. As
long as ninja check-lldb actually runs, completes, and behaves pretty much
the same way as before, that's pretty much all that needs to be verified.

Ctrl+C shouldn't behave any differently with this patch, the idea was to
refactor it so that on Windows it would be identical to pre-patch, but on
other platforms there would be improved Ctrl+C handling

[amccarth]

Ninja check-lldb works.

But...

Before the patch, there are 53 failing test suites and 71 failing test cases.
After the patch there are 54 failing test suites and 74 failing test cases.

Diffing the list of failed suites at the end of the output shows only one difference: TestCPPBreakpoints.py is listed twice after the patch.

That may be unrelated to the patch, but it seems consistent.

[zturner]

Can you confirm with 2 runs before and 2 runs after the patch that you see
the same results every time?

Todd, can you think of a reason why this might happen? I don't think it's
worth holding the patch up too long over this because I want to see the
other improvements come through, but if you can think of an obvious cause
for this Todd (or if you feel like looking into it before I get back
tomorrow Adrian) we could probably fix it before it goes in. Either In any
case, I'm willing to let it go in this way and fix it tomorrow if necessary.

[amccarth]

I'm re-running the tests now, but I agree this is probably not worth holding up this patch.

[tfiala]

In D12651#241810, @zturner wrote:

Can you confirm with 2 runs before and 2 runs after the patch that you see
the same results every time?

Todd, can you think of a reason why this might happen?

Hmm, I vaguely recall finding what looked like a bug in the pre-change code where we might have been losing a count on something, so we might be counting something we previously didn't. But I went through so much code on this in the last 72 hours that I might be misremembering now.

What I can say is that if we're miscounting something (possibly newly so), it shouldn't be too hard to find and fix when we list everything. (i.e. if we're counting something, we should be able to say what it was that was counted, and if there's something being misrepresented, that should be relatively easy to track down).

worth holding the patch up too long over this because I want to see the
other improvements come through, but if you can think of an obvious cause
for this Todd (or if you feel like looking into it before I get back
tomorrow Adrian) we could probably fix it before it goes in. Either In any
case, I'm willing to let it go in this way and fix it tomorrow if necessary.

That would be great.

Let me know when you're comfortable with it going in, Zachary. And thanks for trying it, Adrian!

Diffing the list of failed suites at the end of the output shows only one difference: TestCPPBreakpoints.py is listed twice after the patch.

We should be able to hunt that down. What is your command line?

[amccarth]

It seems to be general flakiness. In three runs without the patch, I got the lower numbers every time. In three runs with the patch, I got higher numbers twice and the lower numbers once.

I have no objection to this patch based on this.

[zturner]

Cool, lgtm as well. Sorry for the holdup

[tfiala]

In D12651#241903, @zturner wrote:

Cool, lgtm as well. Sorry for the holdup

Absolutely no worries. Thanks for checking, Zachary!

It would also be good if we could get either the --test-runner-name with "threading" or "mulltiprocessing" working on Windows at some point (i.e. look into that original failure when ctrl-c was added), if for no other reason than I suspect you'd get a performance win on Windows based on other platforms. You might also find, even if you're stuck with using the pool implementation, that "threading-pool" might just be faster than "multiprocessing-pool" on Windows. The threading-pool test runner strategy should be identical in behavior on Windows to the multiprocessing-pool strategy that you're using over there, with the diff of using "threading" rather than the "multiprocessing" module for the underlying implementation.

I'll get this checked in. Thanks!

[tfiala]

Committed here:

Sending        test/dosep.py
Sending        test/dotest.py
Sending        test/dotest_args.py
Transmitting file data ...
Committed revision 247084.

[tfiala]

On OS X there was at least one test that would hang each run, so I didn't get any kind of real timing numbers there since everything was always maxed out by the hanging test.

Well wonders abound. On my OS X setup, I am not seeing the hanging tests with TOT from mid day. And, surprisingly, the '--test-runner-name threading' is no longer grabbing the parallel test runner's Python global interpreter lock on the exec. This means the threads are really working in parallel as originally intended last year. This might be related to changes since OS X 10.9 when I was last messing with this. The better news is that, unlike Linux, the 'threading' test runner *is* a speed improvement over the 'multiprocessing' test runner: on a 4 core MBP mid 2013 with 8 GB RAM using 8 threads, I shaved off 16% (~1.2 times speedup) on my runtime using 'threading' over 'multiprocessing'.

I'm not sure if this is related to the version of OS X and the bundled Python, so I may do some experimentation before considering flipping it to default to 'threading' for OS X.

test/dosep.py

 echo core.%p | sudo tee /proc/sys/kernel/core_pattern
 """
 
+import fnmatch
 import multiprocessing
 import os
-import fnmatch
 import platform
+import Queue
 import re
-import dotest_args
+import signal
 import subprocess
 import sys
 
+import dotest_args
+
 from optparse import OptionParser
 
 

     return passes, failures, unexpected_successes
 
 
-def call_with_timeout(command, timeout, name):
+def call_with_timeout(command, timeout, name, inferior_pid_events):
     """Run command with a timeout if possible."""
     """-s QUIT will create a coredump if they are enabled on your system"""
     process = None

                                    stdin=subprocess.PIPE,
                                    stdout=subprocess.PIPE,
                                    stderr=subprocess.PIPE)
+    inferior_pid = process.pid
+    if inferior_pid_events:
+        inferior_pid_events.put_nowait(('created', inferior_pid))
     output = process.communicate()
     exit_status = process.returncode
+    if inferior_pid_events:
+        inferior_pid_events.put_nowait(('destroyed', inferior_pid))
+
     passes, failures, unexpected_successes = parse_test_results(output)
     if exit_status == 0:
         # stdout does not have any useful information from 'dotest.py',

     return name, exit_status, passes, failures, unexpected_successes
 
 
-def process_dir(root, files, test_root, dotest_argv):
+def process_dir(root, files, test_root, dotest_argv, inferior_pid_events):
     """Examine a directory for tests, and invoke any found within it."""
     results = []
     for name in files:

         timeout = (os.getenv("LLDB_%s_TIMEOUT" % timeout_name) or
                    getDefaultTimeout(dotest_options.lldb_platform_name))
 
-        results.append(call_with_timeout(command, timeout, name))
+        results.append(call_with_timeout(
+            command, timeout, name, inferior_pid_events))
 
     # result = (name, status, passes, failures, unexpected_successes)
     timed_out = [name for name, status, _, _, _ in results

 out_q = None
 
 
-def process_dir_worker(arg_tuple):
-    """Worker thread main loop when in multithreaded mode.
+def process_dir_worker_multiprocessing(
+        a_output_lock, a_test_counter, a_total_tests, a_test_name_len,
+        a_dotest_options, job_queue, result_queue, inferior_pid_events):
+    """Worker thread main loop when in multiprocessing mode.
     Takes one directory specification at a time and works on it."""
-    return process_dir(*arg_tuple)
 
+    # Shut off interrupt handling in the child process.
+    signal.signal(signal.SIGINT, signal.SIG_IGN)
 
-def walk_and_invoke(test_directory, test_subdir, dotest_argv, num_threads):
-    """Look for matched files and invoke test driver on each one.
-    In single-threaded mode, each test driver is invoked directly.
-    In multi-threaded mode, submit each test driver to a worker
-    queue, and then wait for all to complete.
+    # Setup the global state for the worker process.
+    setup_global_variables(
+        a_output_lock, a_test_counter, a_total_tests, a_test_name_len,
+        a_dotest_options)
 
-    test_directory - lldb/test/ directory
-    test_subdir - lldb/test/ or a subfolder with the tests we're interested in
-                  running
+    # Keep grabbing entries from the queue until done.
+    while not job_queue.empty():
+        try:
+            job = job_queue.get(block=False)
+            result = process_dir(job[0], job[1], job[2], job[3],
+                                 inferior_pid_events)
+            result_queue.put(result)
+        except Queue.Empty:
+            # Fine, we're done.
+            pass
+
+
+def process_dir_worker_multiprocessing_pool(args):
+    process_dir(*args)
+
+
+def process_dir_mapper_inprocess(args):
+    """Map adapter for running the subprocess-based test runner.
+
+    @param args the process work item tuple
+    @return the test result tuple
+"""
+    return process_dir(*args)
+
+
+def collect_active_pids_from_pid_events(event_queue):
     """
+    Returns the set of what should be active inferior pids based on
+    the event stream.
 
-    # Collect the test files that we'll run.
-    test_work_items = []
-    for root, dirs, files in os.walk(test_subdir, topdown=False):
-        def is_test(name):
+    @param event_queue a multiprocessing.Queue containing events of the
+    form:
+         ('created', pid)
+         ('destroyed', pid)
+
+    @return set of inferior dotest.py pids activated but never completed.
+    """
+    active_pid_set = set()
+    while not event_queue.empty():
+        pid_event = event_queue.get_nowait()
+        if pid_event[0] == 'created':
+            active_pid_set.add(pid_event[1])
+        elif pid_event[0] == 'destroyed':
+            active_pid_set.remove(pid_event[1])
+    return active_pid_set
+
+
+def kill_all_worker_processes(workers, inferior_pid_events):
+    """
+    Kills all specified worker processes and their process tree.
+
+    @param workers a list of multiprocess.Process worker objects.
+    @param inferior_pid_events a multiprocess.Queue that contains
+    all inferior create and destroy events.  Used to construct
+    the list of child pids still outstanding that need to be killed.
+    """
+    for worker in workers:
+        worker.terminate()
+        worker.join()
+
+    # Add all the child test pids created.
+    active_pid_set = collect_active_pids_from_pid_events(
+        inferior_pid_events)
+    for inferior_pid in active_pid_set:
+        print "killing inferior pid {}".format(inferior_pid)
+        os.kill(inferior_pid, signal.SIGKILL)
+
+
+def find_test_files_in_dir_tree(dir_root, found_func):
+    """Calls found_func for all the test files in the given dir hierarchy.
+
+    @param dir_root the path to the directory to start scanning
+    for test files.  All files in this directory and all its children
+    directory trees will be searched.
+
+    @param found_func a callable object that will be passed
+    the parent directory (relative to dir_root) and the list of
+    test files from within that directory.
+    """
+    for root, _, files in os.walk(dir_root, topdown=False):
+        def is_test_filename(test_dir, base_filename):
+            """Returns True if the given filename matches the test name format.
+
+            @param test_dir the directory to check.  Should be absolute or
+            relative to current working directory.
+
+            @param base_filename the base name of the filename to check for a
+            dherence to the python test case filename format.
+
+            @return True if name matches the python test case filename format.
+            """
             # Not interested in symbolically linked files.
-            if os.path.islink(os.path.join(root, name)):
+            if os.path.islink(os.path.join(test_dir, base_filename)):
                 return False
             # Only interested in test files with the "Test*.py" naming pattern.
-            return name.startswith("Test") and name.endswith(".py")
+            return (base_filename.startswith("Test") and
+                    base_filename.endswith(".py"))
+
+        tests = [filename for filename in files
+                 if is_test_filename(root, filename)]
+        if tests:
+            found_func(root, tests)
 
-        tests = filter(is_test, files)
-        test_work_items.append((root, tests, test_directory, dotest_argv))
 
+def initialize_global_vars_multiprocessing(num_threads, test_work_items):
+    # Initialize the global state we'll use to communicate with the
+    # rest of the flat module.
     global output_lock, test_counter, total_tests, test_name_len
     output_lock = multiprocessing.RLock()
-    # item = (root, tests, test_directory, dotest_argv)
     total_tests = sum([len(item[1]) for item in test_work_items])
     test_counter = multiprocessing.Value('i', 0)
     test_name_len = multiprocessing.Value('i', 0)

         total_tests, num_threads, (num_threads > 1) * "s")
     update_progress()
 
-    # Run the items, either in a pool (for multicore speedup) or
-    # calling each individually.
-    if num_threads > 1:
-        pool = multiprocessing.Pool(
-            num_threads,
-            initializer=setup_global_variables,
-            initargs=(output_lock, test_counter, total_tests, test_name_len,
-                      dotest_options))
-        test_results = pool.map(process_dir_worker, test_work_items)
-    else:
-        test_results = map(process_dir_worker, test_work_items)
 
-    # result = (timed_out, failed, passed, unexpected_successes, fail_count, pass_count)
+def multiprocessing_test_runner(num_threads, test_work_items):
+    """Provides multiprocessing.Pool.map() test runner adapter.
+
+    This concurrent test runner is based on the multiprocessing
+    library, and rolls its own worker pooling strategy so it
+    can handle Ctrl-C properly.
+
+    This test runner is known to have an issue running on
+    Windows platforms.
+
+    @param num_threads the number of worker processes to use.
+
+    @param test_work_items the iterable of test work item tuples
+    to run.
+    """
+
+    # Initialize our global state.
+    initialize_global_vars_multiprocessing(num_threads, test_work_items)
+
+    # Create jobs.
+    job_queue = multiprocessing.Queue(len(test_work_items))
+    for test_work_item in test_work_items:
+        job_queue.put(test_work_item)
+
+    result_queue = multiprocessing.Queue(len(test_work_items))
+
+    # Create queues for started child pids.  Terminating
+    # the multiprocess processes does not terminate the
+    # child processes they spawn.  We can remove this tracking
+    # if/when we move to having the multiprocess process directly
+    # perform the test logic.  The Queue size needs to be able to
+    # hold 2 * (num inferior dotest.py processes started) entries.
+    inferior_pid_events = multiprocessing.Queue(4096)
+
+    # Create workers.  We don't use multiprocessing.Pool due to
+    # challenges with handling ^C keyboard interrupts.
+    workers = []
+    for _ in range(num_threads):
+        worker = multiprocessing.Process(
+            target=process_dir_worker_multiprocessing,
+            args=(output_lock,
+                  test_counter,
+                  total_tests,
+                  test_name_len,
+                  dotest_options,
+                  job_queue,
+                  result_queue,
+                  inferior_pid_events))
+        worker.start()
+        workers.append(worker)
+
+    # Wait for all workers to finish, handling ^C as needed.
+    try:
+        for worker in workers:
+            worker.join()
+    except KeyboardInterrupt:
+        # First try to drain the queue of work and let the
+        # running tests complete.
+        while not job_queue.empty():
+            try:
+                # Just drain it to stop more work from being started.
+                job_queue.get_nowait()
+            except Queue.Empty:
+                pass
+
+        print ('\nFirst KeyboardInterrupt received, stopping '
+               'future work.  Press again to hard-stop existing tests.')
+        try:
+            for worker in workers:
+                worker.join()
+        except KeyboardInterrupt:
+            print ('\nSecond KeyboardInterrupt received, killing '
+                   'all worker process trees.')
+            kill_all_worker_processes(workers, inferior_pid_events)
+
+    test_results = []
+    while not result_queue.empty():
+        test_results.append(result_queue.get(block=False))
+    return test_results
+
+
+def multiprocessing_test_runner_pool(num_threads, test_work_items):
+    # Initialize our global state.
+    initialize_global_vars_multiprocessing(num_threads, test_work_items)
+
+    pool = multiprocessing.Pool(
+        num_threads,
+        initializer=setup_global_variables,
+        initargs=(output_lock, test_counter, total_tests, test_name_len,
+                  dotest_options))
+    return pool.map(process_dir_worker_multiprocessing_pool, test_work_items)
+
+
+def inprocess_exec_test_runner(test_work_items):
+    # Initialize our global state.
+    initialize_global_vars_multiprocessing(1, test_work_items)
+    return map(process_dir_mapper_inprocess, test_work_items)
+
+
+def walk_and_invoke(test_directory, test_subdir, dotest_argv,
+                    test_runner_func):
+    """Look for matched files and invoke test driver on each one.
+    In single-threaded mode, each test driver is invoked directly.
+    In multi-threaded mode, submit each test driver to a worker
+    queue, and then wait for all to complete.
+
+    test_directory - lldb/test/ directory
+    test_subdir - lldb/test/ or a subfolder with the tests we're interested in
+                  running
+    """
+
+    # Collect the test files that we'll run.
+    test_work_items = []
+    find_test_files_in_dir_tree(
+        test_subdir, lambda testdir, test_files: test_work_items.append([
+            test_subdir, test_files, test_directory, dotest_argv, None]))
+
+    # Convert test work items into test results using whatever
+    # was provided as the test run function.
+    test_results = test_runner_func(test_work_items)
+
+    # Summarize the results and return to caller.
     timed_out = sum([result[0] for result in test_results], [])
     passed = sum([result[1] for result in test_results], [])
     failed = sum([result[2] for result in test_results], [])

     pass_count = sum([result[4] for result in test_results])
     fail_count = sum([result[5] for result in test_results])
 
-    return (timed_out, passed, failed, unexpected_successes, pass_count, fail_count)
+    return (timed_out, passed, failed, unexpected_successes, pass_count,
+            fail_count)
 
 
 def getExpectedTimeouts(platform_name):

     return result
 
 
-def main(print_details_on_success, num_threads, test_subdir):
+def get_test_runner_strategies(num_threads):
+    """Returns the test runner strategies by name in a dictionary.
+
+    @param num_threads specifies the number of threads/processes
+    that will be used for concurrent test runners.
+
+    @return dictionary with key as test runner strategy name and
+    value set to a callable object that takes the test work item
+    and returns a test result tuple.
+    """
+    return {
+        # multiprocessing supports ctrl-c and does not use
+        # multiprocessing.Pool.
+        "multiprocessing":
+        (lambda work_items: multiprocessing_test_runner(
+            num_threads, work_items)),
+
+        # multiprocessing-pool uses multiprocessing.Pool but
+        # does not support Ctrl-C.
+        "multiprocessing-pool":
+        (lambda work_items: multiprocessing_test_runner_pool(
+            num_threads, work_items)),
+
+        # serial uses the subprocess-based, single process
+        # test runner.  This provides process isolation but
+        # no concurrent test running.
+        "serial":
+        inprocess_exec_test_runner
+        }
+
+
+def main(print_details_on_success, num_threads, test_subdir,
+         test_runner_name):
     """Run dotest.py in inferior mode in parallel.
 
     @param print_details_on_success the parsed value of the output-on-success

     @param test_subdir optionally specifies a subdir to limit testing
     within.  May be None if the entire test tree is to be used.  This subdir
     is assumed to be relative to the lldb/test root of the test hierarchy.
+
+    @param test_runner_name if specified, contains the test runner
+    name which selects the strategy used to run the isolated and
+    optionally concurrent test runner. Specify None to allow the
+    system to choose the most appropriate test runner given desired
+    thread count and OS type.
+
     """
 
     dotest_argv = sys.argv[1:]

         num_threads = 1
 
     system_info = " ".join(platform.uname())
-    (timed_out, passed, failed, unexpected_successes, pass_count, fail_count) = walk_and_invoke(
-        test_directory, test_subdir, dotest_argv, num_threads)
+
+    # Figure out which testrunner strategy we'll use.
+    runner_strategies_by_name = get_test_runner_strategies(num_threads)
+
+    # If the user didn't specify a test runner strategy, determine
+    # the default now based on number of threads and OS type.
+    if not test_runner_name:
+        if num_threads == 1:
+            # Use the serial runner.
+            test_runner_name = "serial"
+        elif os.name == "nt":
+            # Currently the multiprocessing test runner with ctrl-c
+            # support isn't running correctly on nt.  Use the pool
+            # support without ctrl-c.
+            test_runner_name = "multiprocessing-pool"
+        else:
+            # For everyone else, use the ctrl-c-enabled
+            # multiprocessing support.
+            test_runner_name = "multiprocessing"
+
+    if test_runner_name not in runner_strategies_by_name:
+        raise ("specified testrunner name '{}' unknown. "
+               "Valid choices: {}".format(
+                   test_runner_name,
+                   runner_strategies_by_name.keys()))
+    test_runner_func = runner_strategies_by_name[test_runner_name]
+
+    summary_results = walk_and_invoke(
+        test_directory, test_subdir, dotest_argv, test_runner_func)
+
+    (timed_out, passed, failed, unexpected_successes, pass_count,
+     fail_count) = summary_results
 
     timed_out = set(timed_out)
     num_test_files = len(passed) + len(failed)

test/dotest.py

 num_threads = None
 output_on_success = False
 no_multiprocess_test_runner = False
+test_runner_name = None
 
 def usage(parser):
     parser.print_help()

     global num_threads
     global output_on_success
     global no_multiprocess_test_runner
+    global test_runner_name
 
     do_help = False
 

     if args.inferior:
         is_inferior_test_runner = True
 
-    if args.output_on_success:
+    # Turn on output_on_sucess if either explicitly added or -v specified.
+    if args.output_on_success or args.v:
         output_on_success = True
 
     if args.num_threads:

     if args.test_subdir:
         multiprocess_test_subdir = args.test_subdir
 
+    if args.test_runner_name:
+        test_runner_name = args.test_runner_name
+
     if args.lldb_platform_name:
         lldb_platform_name = args.lldb_platform_name
     if args.lldb_platform_url:

     # multiprocess test runner here.
     if isMultiprocessTestRunner():
         import dosep
-        dosep.main(output_on_success, num_threads, multiprocess_test_subdir)
+        dosep.main(output_on_success, num_threads, multiprocess_test_subdir,
+                   test_runner_name)
         raise "should never get here"
 
     setupSysPath()

test/dotest_args.py

         dest='num_threads',
         help=('The number of threads/processes to use when running tests '
               'separately, defaults to the number of CPU cores available'))
-    parser.add_argument(
+    group.add_argument(
         '--test-subdir',
         action='store',
         help='Specify a test subdirectory to use relative to the test root dir'
     )
+    group.add_argument(
+        '--test-runner-name',
+        action='store',
+        help=('Specify a test runner strategy.  Valid values: multiprocessing,'
+              ' multiprocessing-pool, serial')
+    )
 
     # Remove the reference to our helper function
     del X