add hook for calling platform-dependent pre-kill action on a timed out test

[tfiala]

Summary

This change introduces the concept of a platform-specific, pre-kill-hook mechanism. If a platform defines the hook, then the hook gets called right after a timeout is detected in a test run, but before the process is killed.

The pre-kill-hook mechanism works as follows:

• When a timeout is detected in the process_control.ProcessDriver class that runs the per-test lldb process, a new overridable on_timeout_pre_kill() method is called on the ProcessDriver instance.

• The concurrent test driver's derived ProcessDriver overrides this method. It looks to see if a module called "lldbsuite.pre_kill_hook.{platform-system-name}" module exists, where platform-system-name is replaced with platform.system().lower():
  • If that module doesn't exist, the rest of the new behavior is skipped.
  • If that module does exist, it is loaded, and the method "do_pre_kill(process_id, output_stream)" is called. If that method throws an exception, we log it and we ignore further processing of the pre-killed process.
  • The process_id arg of the do_pre_kill function is the process id as returned by the ProcessDriver.pid property.
  • The output_stream arg of the do_pre_kill function takes a file-like object. Output to be collected from doing any processing on the process-to-be-killed should be written into the file-like object. The current impl uses a six.StringIO and then writes this output to {TestFilename}-{pid}.sample in the session directory.

Platforms where platform.system() is "Darwin" will get a pre-kill action that runs the 'sample' program on the lldb that has timed out. That data will be collected on CI and analyzed to determine what is happening during timeouts. (This has an advantage over a core in that it is much smaller and that it clearly demonstrates any liveness of the process, if there is any).

I will also hunt around on Linux to see if there might be something akin to 'sample' that might be available. If so, it would be nice to hook something up for that.

[tfiala]

packages/Python/lldbsuite/test/dosep.py

238–245

I suspect we will need to tweak this a bit. We need to be able to dispatch on more than just the host platform.system(). It may be sufficient to pass along the test platform info as an argument.

[tfiala]

Greg also had the idea of having a fallback mechanism that uses a newly-spun-up lldb to attach to the to-be-killed process, and retrieves the threads and backdtraces, to dump out a compact description. That's nice in that it should work on any host that has a working lldb with python support.

[labath]

Sounds like a useful thing to have. I've found turning on logging very helpful when looking for these issues, as it can tell you what was happening in the past, in addition to the current state (also it allows you to compare the logs from a successful and unsuccessful run).

packages/Python/lldbsuite/test/test_runner/process_control.py

514

Is this actually used anywhere?

[tfiala]

packages/Python/lldbsuite/test/test_runner/process_control.py

514

No - I originally was parsing some options out of it in the handler, but I no longer am doing that. I will take this out of the final. It's easy enough to add in later if we ever need it.

[tfiala]

I'm about to check this in. I just wanted to put up my final change, which includes the following:

• README.md - documents the new pre_kill_hook package in lldbsuite.

• added a runner_context/context_dict argument to the pre-kill-hook logic. This dictionary will contain:
  • entry 'args': array containing configuration.args
  • entry 'platform_name': contains configuration.lldb_platform_name
  • entry 'platform_url': contains configuration.lldb_platform_url
  • entry 'platform_working_dir': contains configuration.lldb_platform_working_dir

I pass the dictionary in to decouple the pre_kill_hook package from the test runner configuration module. (Also, the configuration module logic is not guaranteed to be run on any of those test queue workers, which may be in separate processes when using the multiprocessing* test runner strategies).

[tfiala]

Closed by r282258

packages/Python/lldbsuite/pre_kill_hook/__init__.py

+"""Initialize the package."""

packages/Python/lldbsuite/pre_kill_hook/darwin.py

+"""Provides a pre-kill method to run on macOS."""
+from __future__ import print_function
+
+# system imports
+import subprocess
+
+# third-party module imports
+import six
+
+
+def do_pre_kill(process_id, output_stream, sample_time=3):
+    """Samples the given process id, and puts the output to output_stream.
+
+    @param process_id:
+        The local process to sample.
+
+    @param output_stream:
+        A File-like object that should be used to write the results of
+        sampling.
+    """
+    output = subprocess.check_output(['sample', six.text_type(process_id),
+                                      str(sample_time)])
+    output_stream.write(output)

packages/Python/lldbsuite/pre_kill_hook/tests/test_darwin.py

+"""Test the pre-kill hook on Darwin."""
+from __future__ import print_function
+
+# system imports
+from multiprocessing import Process, Queue
+from platform import system
+import re
+from unittest import main, TestCase
+
+# third party
+from six import StringIO
+
+
+def do_child_process(child_work_queue, parent_work_queue, verbose):
+    import os
+
+    pid = os.getpid()
+    if verbose:
+        print("child: pid {} started, sending to parent".format(pid))
+    parent_work_queue.put(pid)
+    if verbose:
+        print("child: waiting for shut-down request from parent")
+    child_work_queue.get()
+    if verbose:
+        print("child: received shut-down request.  Child exiting.")
+
+
+class DarwinPreKillTestCase(TestCase):
+
+    def __init__(self, methodName):
+        super(DarwinPreKillTestCase, self).__init__(methodName)
+        self.process = None
+        self.child_work_queue = None
+        self.verbose = False
+
+    def tearDown(self):
+        if self.verbose:
+            print("parent: sending shut-down request to child")
+        if self.process:
+            self.child_work_queue.put("hello, child")
+        self.process.join()
+        if self.verbose:
+            print("parent: child is fully shut down")
+
+    def test_sample(self):
+        # Ensure we're Darwin.
+        if system() != 'Darwin':
+            self.skipTest("requires a Darwin-based OS")
+
+        # Start the child process.
+        self.child_work_queue = Queue()
+        parent_work_queue = Queue()
+        self.process = Process(target=do_child_process,
+                               args=(self.child_work_queue, parent_work_queue,
+                                     self.verbose))
+        if self.verbose:
+            print("parent: starting child")
+        self.process.start()
+
+        # Wait for the child to report its pid.  Then we know we're running.
+        if self.verbose:
+            print("parent: waiting for child to start")
+        child_pid = parent_work_queue.get()
+
+        # Sample the child process.
+        from darwin import do_pre_kill
+
+        if self.verbose:
+            print("parent: running pre-kill action on child")
+        output_io = StringIO()
+        do_pre_kill(child_pid, output_io)
+        output = output_io.getvalue()
+
+        if self.verbose:
+            print("parent: do_pre_kill() wrote the following output:", output)
+        self.assertIsNotNone(output)
+
+        # We should have a line with:
+        # Process:  .* [{pid}]
+        process_re = re.compile(r"Process:[^[]+\[([^]]+)\]")
+        match = process_re.search(output)
+        self.assertIsNotNone(match, "should have found process id for "
+                             "sampled process")
+        self.assertEqual(1, len(match.groups()))
+        self.assertEqual(child_pid, int(match.group(1)))
+
+        # We should see a Call graph: section.
+        callgraph_re = re.compile(r"Call graph:")
+        match = callgraph_re.search(output)
+        self.assertIsNotNone(match, "should have found the Call graph section"
+                             "in sample output")
+
+        # We should see a Binary Images: section.
+        binary_images_re = re.compile(r"Binary Images:")
+        match = binary_images_re.search(output)
+        self.assertIsNotNone(match, "should have found the Binary Images "
+                             "section in sample output")
+
+
+if __name__ == "__main__":
+    main()

packages/Python/lldbsuite/test/dosep.py

 import multiprocessing.pool
 import os
 import platform
 import re
 import signal
 import sys
 import threading
 
+from six import StringIO
 from six.moves import queue
 
 # Our packages and modules
 import lldbsuite
 import lldbsuite.support.seven as seven
 
 from . import configuration
 from . import dotest_args
 from lldbsuite.support import optional_with
 from lldbsuite.test_event import dotest_channels
 from lldbsuite.test_event.event_builder import EventBuilder
 from lldbsuite.test_event import formatter
 
 from .test_runner import process_control
 
 # Status codes for running command with timeout.
 eTimedOut, ePassed, eFailed = 124, 0, 1
 
+g_session_dir = None
 output_lock = None
 test_counter = None
 total_tests = None
 test_name_len = None
 dotest_options = None
 RESULTS_FORMATTER = None
 RUNNER_PROCESS_ASYNC_MAP = None
 RESULTS_LISTENER_CHANNEL = None
def on_process_exited(self, command, output, was_timeout, exit_status):
         # Save off the results for the caller.
         self.results = (
             self.file_name,
             exit_status,
             passes,
             failures,
             unexpected_successes)
 
+    def on_timeout_pre_kill(self):
+        # We're just about to have a timeout take effect.  Here's our chance
+        # to do a pre-kill action.
+
+        # For now, we look to see if the lldbsuite.pre_kill module has a
+        # runner for our platform.
+        module_name = "lldbsuite.pre_kill_hook." + platform.system().lower()
+        try:
+            import importlib
+            module = importlib.import_module(module_name)
+        except ImportError:
+            # We don't have one for this platform.  Skip.
+            sys.stderr.write("\nwarning: no timeout handler module: " + module_name)
+            return
+
+        # Try to run the pre-kill-hook method.
+        try:
+            # Run the pre-kill command.
+            output_io = StringIO()
+            module.do_pre_kill(self.pid, output_io)
+
+            # Write the output to a filename associated with the test file and pid.
+            basename = "{}-{}.sample".format(self.file_name, self.pid)
+            sample_path = os.path.join(g_session_dir, basename)
+            with open(sample_path, "w") as output_file:
+                output_file.write(output_io.getvalue())
+        except Exception as e:
+            sys.stderr.write("caught exception while running "
+                             "pre-kill action: {}".format(e))
+            return
+
     def is_exceptional_exit(self):
         """Returns whether the process returned a timeout.
 
         Not valid to call until after on_process_exited() completes.
 
         @return True if the exit is an exceptional exit (e.g. signal on
         POSIX); False otherwise.
         """
for root, _, files in os.walk(dir_root, topdown=False):
         tests = [
             (filename, os.path.join(root, filename))
             for filename in files
             if is_test_filename(root, filename)]
         if tests:
             found_func(root, tests)
 
 
-def initialize_global_vars_common(num_threads, test_work_items):
-    global total_tests, test_counter, test_name_len
+def initialize_global_vars_common(num_threads, test_work_items, session_dir):
+    global g_session_dir, total_tests, test_counter, test_name_len
 
     total_tests = sum([len(item[1]) for item in test_work_items])
     test_counter = multiprocessing.Value('i', 0)
     test_name_len = multiprocessing.Value('i', 0)
+    g_session_dir = session_dir
     if not (RESULTS_FORMATTER and RESULTS_FORMATTER.is_using_terminal()):
         print(
             "Testing: %d test suites, %d thread%s" %
             (total_tests,
              num_threads,
              (num_threads > 1) *
                 "s"),
             file=sys.stderr)
     update_progress()
 
 
-def initialize_global_vars_multiprocessing(num_threads, test_work_items):
+def initialize_global_vars_multiprocessing(num_threads, test_work_items,
+                                           session_dir):
     # Initialize the global state we'll use to communicate with the
     # rest of the flat module.
     global output_lock
     output_lock = multiprocessing.RLock()
 
-    initialize_global_vars_common(num_threads, test_work_items)
+    initialize_global_vars_common(num_threads, test_work_items, session_dir)
 
 
-def initialize_global_vars_threading(num_threads, test_work_items):
+def initialize_global_vars_threading(num_threads, test_work_items, session_dir):
     """Initializes global variables used in threading mode.
     @param num_threads specifies the number of workers used.
     @param test_work_items specifies all the work items
     that will be processed.
     """
     # Initialize the global state we'll use to communicate with the
     # rest of the flat module.
     global output_lock
     output_lock = threading.RLock()
 
     index_lock = threading.RLock()
     index_map = {}
 
     def get_worker_index_threading():
         """Returns a 0-based, thread-unique index for the worker thread."""
         thread_id = threading.current_thread().ident
         with index_lock:
             if thread_id not in index_map:
                 index_map[thread_id] = len(index_map)
             return index_map[thread_id]
 
     global GET_WORKER_INDEX
     GET_WORKER_INDEX = get_worker_index_threading
 
-    initialize_global_vars_common(num_threads, test_work_items)
+    initialize_global_vars_common(num_threads, test_work_items, session_dir)
 
 
 def ctrl_c_loop(main_op_func, done_func, ctrl_c_handler):
     """Provides a main loop that is Ctrl-C protected.
 
     The main loop calls the main_op_func() repeatedly until done_func()
     returns true.  The ctrl_c_handler() method is called with a single
     int parameter that contains the number of times the ctrl_c has been
if workers is not None and len(workers) > 0:
         # We're not done if we still have workers left.
         return False
     if async_map is not None and len(async_map) > 0:
         return False
     # We're done.
     return True
 
 
-def multiprocessing_test_runner(num_threads, test_work_items):
+def multiprocessing_test_runner(num_threads, test_work_items, session_dir):
     """Provides hand-wrapped pooling test runner adapter with Ctrl-C support.
 
     This concurrent test runner is based on the multiprocessing
     library, and rolls its own worker pooling strategy so it
     can handle Ctrl-C properly.
 
     This test runner is known to have an issue running on
     Windows platforms.
 
     @param num_threads the number of worker processes to use.
 
     @param test_work_items the iterable of test work item tuples
     to run.
     """
 
     # Initialize our global state.
-    initialize_global_vars_multiprocessing(num_threads, test_work_items)
+    initialize_global_vars_multiprocessing(num_threads, test_work_items,
+                                           session_dir)
 
     # Create jobs.
     job_queue = multiprocessing.Queue(len(test_work_items))
     for test_work_item in test_work_items:
         job_queue.put(test_work_item)
 
     result_queue = multiprocessing.Queue(len(test_work_items))
 
while not done:
             # is complete.
             if len(channel_map) > 0:
                 # We still have an asyncore channel running.  Not done yet.
                 done = False
 
     return map_results
 
 
-def multiprocessing_test_runner_pool(num_threads, test_work_items):
+def multiprocessing_test_runner_pool(num_threads, test_work_items, session_dir):
     # Initialize our global state.
-    initialize_global_vars_multiprocessing(num_threads, test_work_items)
+    initialize_global_vars_multiprocessing(num_threads, test_work_items,
+                                           session_dir)
 
     manager = multiprocessing.Manager()
     worker_index_map = manager.dict()
 
     pool = multiprocessing.Pool(
         num_threads,
         initializer=setup_global_variables,
         initargs=(output_lock, test_counter, total_tests, test_name_len,
                   dotest_options, worker_index_map))
 
     # Start the map operation (async mode).
     map_future = pool.map_async(
         process_dir_worker_multiprocessing_pool, test_work_items)
     return map_async_run_loop(
         map_future, RUNNER_PROCESS_ASYNC_MAP, RESULTS_LISTENER_CHANNEL)
 
 
-def threading_test_runner(num_threads, test_work_items):
+def threading_test_runner(num_threads, test_work_items, session_dir):
     """Provides hand-wrapped pooling threading-based test runner adapter
     with Ctrl-C support.
 
     This concurrent test runner is based on the threading
     library, and rolls its own worker pooling strategy so it
     can handle Ctrl-C properly.
 
     @param num_threads the number of worker processes to use.
 
     @param test_work_items the iterable of test work item tuples
     to run.
     """
 
     # Initialize our global state.
-    initialize_global_vars_threading(num_threads, test_work_items)
+    initialize_global_vars_threading(num_threads, test_work_items, session_dir)
 
     # Create jobs.
     job_queue = queue.Queue()
     for test_work_item in test_work_items:
         job_queue.put(test_work_item)
 
     result_queue = queue.Queue()
 
def threading_test_runner(num_threads, test_work_items, session_dir):
 
     # Reap the test results.
     test_results = []
     while not result_queue.empty():
         test_results.append(result_queue.get(block=False))
     return test_results
 
 
-def threading_test_runner_pool(num_threads, test_work_items):
+def threading_test_runner_pool(num_threads, test_work_items, session_dir):
     # Initialize our global state.
-    initialize_global_vars_threading(num_threads, test_work_items)
+    initialize_global_vars_threading(num_threads, test_work_items, session_dir)
 
     pool = multiprocessing.pool.ThreadPool(num_threads)
     map_future = pool.map_async(
         process_dir_worker_threading_pool, test_work_items)
 
     return map_async_run_loop(
         map_future, RUNNER_PROCESS_ASYNC_MAP, RESULTS_LISTENER_CHANNEL)
 
 
 def asyncore_run_loop(channel_map):
     try:
         asyncore.loop(None, False, channel_map)
     except:
         # Swallow it, we're seeing:
         #   error: (9, 'Bad file descriptor')
         # when the listener channel is closed.  Shouldn't be the case.
         pass
 
 
-def inprocess_exec_test_runner(test_work_items):
+def inprocess_exec_test_runner(test_work_items, session_dir):
     # Initialize our global state.
-    initialize_global_vars_multiprocessing(1, test_work_items)
+    initialize_global_vars_multiprocessing(1, test_work_items, session_dir)
 
     # We're always worker index 0
     global GET_WORKER_INDEX
     GET_WORKER_INDEX = lambda: 0
 
     # Run the listener and related channel maps in a separate thread.
     # global RUNNER_PROCESS_ASYNC_MAP
     global RESULTS_LISTENER_CHANNEL
def find(pattern, path):
     result = []
     for root, dirs, files in os.walk(path):
         for name in files:
             if fnmatch.fnmatch(name, pattern):
                 result.append(os.path.join(root, name))
     return result
 
 
-def get_test_runner_strategies(num_threads):
+def get_test_runner_strategies(num_threads, session_dir):
     """Returns the test runner strategies by name in a dictionary.
 
     @param num_threads specifies the number of threads/processes
     that will be used for concurrent test runners.
 
+    @param session_dir specifies the session dir to use for
+    auxiliary files.
+
     @return dictionary with key as test runner strategy name and
     value set to a callable object that takes the test work item
     and returns a test result tuple.
     """
     return {
         # multiprocessing supports ctrl-c and does not use
         # multiprocessing.Pool.
         "multiprocessing":
         (lambda work_items: multiprocessing_test_runner(
-            num_threads, work_items)),
+            num_threads, work_items, session_dir)),
 
         # multiprocessing-pool uses multiprocessing.Pool but
         # does not support Ctrl-C.
         "multiprocessing-pool":
         (lambda work_items: multiprocessing_test_runner_pool(
-            num_threads, work_items)),
+            num_threads, work_items, session_dir)),
 
         # threading uses a hand-rolled worker pool much
         # like multiprocessing, but instead uses in-process
         # worker threads.  This one supports Ctrl-C.
         "threading":
-        (lambda work_items: threading_test_runner(num_threads, work_items)),
+        (lambda work_items: threading_test_runner(
+            num_threads, work_items, session_dir)),
 
         # threading-pool uses threading for the workers (in-process)
         # and uses the multiprocessing.pool thread-enabled pool.
         # This does not properly support Ctrl-C.
         "threading-pool":
         (lambda work_items: threading_test_runner_pool(
-            num_threads, work_items)),
+            num_threads, work_items, session_dir)),
 
         # serial uses the subprocess-based, single process
         # test runner.  This provides process isolation but
         # no concurrent test execution.
         "serial":
-        inprocess_exec_test_runner
+        (lambda work_items: inprocess_exec_test_runner(
+            work_items, session_dir))
     }
 
 
 def _remove_option(
         args, long_option_name, short_option_name, takes_arg):
     """Removes option and related option arguments from args array.
 
     This method removes all short/long options that match the given
def default_test_runner_name(num_threads):
     else:
         # For everyone else, use the ctrl-c-enabled threading support.
         # Should use fewer system resources than the multprocessing
         # variant.
         test_runner_name = "threading"
     return test_runner_name
 
 
-def rerun_tests(test_subdir, tests_for_rerun, dotest_argv):
+def rerun_tests(test_subdir, tests_for_rerun, dotest_argv, session_dir):
     # Build the list of test files to rerun.  Some future time we'll
     # enable re-run by test method so we can constrain the rerun set
     # to just the method(s) that were in issued within a file.
 
     # Sort rerun files into subdirectories.
     print("\nRerunning the following files:")
     rerun_files_by_subdir = {}
     for test_filename in tests_for_rerun.keys():
def rerun_tests(test_subdir, tests_for_rerun, dotest_argv, session_dir):
     # Do not update legacy counts, I am getting rid of
     # them so no point adding complicated merge logic here.
     rerun_thread_count = 1
     # Force the parallel test runner to choose a multi-worker strategy.
     rerun_runner_name = default_test_runner_name(rerun_thread_count + 1)
     print("rerun will use the '{}' test runner strategy".format(
         rerun_runner_name))
 
-    runner_strategies_by_name = get_test_runner_strategies(rerun_thread_count)
+    runner_strategies_by_name = get_test_runner_strategies(
+        rerun_thread_count, session_dir)
     rerun_runner_func = runner_strategies_by_name[
         rerun_runner_name]
     if rerun_runner_func is None:
         raise Exception(
             "failed to find rerun test runner "
             "function named '{}'".format(rerun_runner_name))
 
     walk_and_invoke(
def main(num_threads, test_subdir, test_runner_name, results_formatter):
 
     # Figure out which test files should be enabled for expected
     # timeout
     expected_timeout = getExpectedTimeouts(dotest_options.lldb_platform_name)
     if results_formatter is not None:
         results_formatter.set_expected_timeouts_by_basename(expected_timeout)
 
     # Figure out which testrunner strategy we'll use.
-    runner_strategies_by_name = get_test_runner_strategies(num_threads)
+    runner_strategies_by_name = get_test_runner_strategies(
+        num_threads, session_dir)
 
     # If the user didn't specify a test runner strategy, determine
     # the default now based on number of threads and OS type.
     if not test_runner_name:
         test_runner_name = default_test_runner_name(num_threads)
 
     if test_runner_name not in runner_strategies_by_name:
         raise Exception(
if results_formatter is not None:
 
             # Check if the number of files exceeds the max cutoff.  If so,
             # we skip the rerun step.
             if rerun_file_count > configuration.rerun_max_file_threshold:
                 print("Skipping rerun: max rerun file threshold ({}) "
                       "exceeded".format(
                           configuration.rerun_max_file_threshold))
             else:
-                rerun_tests(test_subdir, tests_for_rerun, dotest_argv)
+                rerun_tests(test_subdir, tests_for_rerun, dotest_argv,
+                            session_dir)
 
     # The results formatter - if present - is done now.  Tell it to
     # terminate.
     if results_formatter is not None:
         results_formatter.send_terminate_as_needed()
 
     timed_out = set(timed_out)
     num_test_files = len(passed) + len(failed)

packages/Python/lldbsuite/test/test_runner/process_control.py

def __init__(self, soft_terminate_timeout=10.0):
         # Might want this longer if core dumps are generated and
         # take a long time to write out.
         self.soft_terminate_timeout = soft_terminate_timeout
         # Number of seconds to wait for the hard terminate to
         # wrap up, before giving up on the io thread.  This should
         # be fast.
         self.hard_terminate_timeout = 5.0
         self.returncode = None
+        self.command = None
 
     # =============================================
     # Methods for subclasses to override if desired.
     # =============================================
 
     def on_process_started(self):
         pass
 
     def on_process_exited(self, command, output, was_timeout, exit_status):
         pass
 
+    def on_timeout_pre_kill(self):
+        """Called after the timeout interval elapses but before killing it.
+
+        This method is added to enable derived classes the ability to do
+        something to the process prior to it being killed.  For example,
+        this would be a good spot to run a program that samples the process
+        to see what it was doing (or not doing).
+
+        Do not attempt to reap the process (i.e. use wait()) in this method.
+        That will interfere with the kill mechanism and return code processing.
+        """
+        pass
+
     def write(self, content):
         # pylint: disable=no-self-use
         # Intended - we want derived classes to be able to override
         # this and use any self state they may contain.
         sys.stdout.write(content)
 
     # ==============================================================
     # Operations used to drive processes.  Clients will want to call
     # one of these.
     # ==============================================================
 
     def run_command(self, command):
+        # Remember the command we were running so we can refer back
+        # to it later.
+        self.command = command
+
         # Start up the child process and the thread that does the
         # communication pump.
         self._start_process_and_io_thread(command)
 
         # Wait indefinitely for the child process to finish
         # communicating.  This indicates it has closed stdout/stderr
         # pipes and is done.
         self.io_thread.join()
class ProcessDriver(object):
     def _wait_with_timeout(self, timeout_seconds, command, want_core):
         # Allow up to timeout seconds for the io thread to wrap up.
         # If that completes, the child process should be done.
         completed_normally = self.done_event.wait(timeout_seconds)
         if completed_normally:
             # Reap the child process here.
             self.returncode = self.process.wait()
         else:
+
+            # Allow derived classes to do some work after we detected
+            # a timeout but before we touch the timed-out process.
+            self.on_timeout_pre_kill()
+
             # Prepare to stop the process
             process_terminated = completed_normally
             terminate_attempt_count = 0
 
             # Try as many attempts as we support for trying to shut down
             # the child process if it's not already shut down.
             while not process_terminated:
                 terminate_attempt_count += 1