[lit] Implement support of per test timeout in lit.

[delcypher]

Summary

[lit] Implement support of per test timeout in lit.

This should work with ShTest (executed externally or internally) and GTest
test formats.

To set the timeout a new option `--timeout=` has
been added which specifies the maximum run time of an individual test
in seconds. By default this 0 which causes no timeout to be enforced.

The timeout can also be set from a lit configuration file by modifying
the `lit_config.maxIndividualTestTime` property.

To implement a timeout we now require the psutil Python module if a
timeout is requested. This dependency is confined to the newly added
`lit.util.killProcessAndChildren()`. A note has been added into the
TODO document describing how we can remove the dependency on the
`pustil` module in the future. It would be nice to remove this
immediately but that is a lot more work and Daniel Dunbar believes it is
better that we get a working implementation first and then improve it.

To avoid breaking the existing behaviour the psutil module will not be
imported if no timeout is requested.

The included testcases are derived from test cases provided by
Jonathan Roelofs which were in an previous attempt to add a per test
timeout to lit (http://reviews.llvm.org/D6584). Thanks Jonathan!

[jroelofs]

utils/lit/lit/main.py

208

Please name the flag "--timeout" so it's easier to type out on the llvm-lit command line. It's probably a good idea to keep the descriptive variable name though.

utils/lit/tests/Inputs/per_test_timeout/slow.py

8

The infinite loop testcase was actually useful to guarantee that the timeout detection & handling code isn't broken. With this testcase, it could be broken and we wouldn't notice (apart from the tests running a little slower).

[delcypher]

utils/lit/tests/Inputs/per_test_timeout/slow.py

8

If the `slow.py program doesn't cause a timeout then several tests will fail. If you replace time.sleep(6) with time.sleep(0)` then run lit on the test cases you will see failures.

python2 ~/dev/llvm-upstream/src/utils/lit/lit.py -v --debug  tests/Inputs/per_test_timeout/
********************
Testing Time: 0.41s
********************
Failing Tests (3):
    per_test_timeout :: timeout_external.txt
    per_test_timeout :: timeout_internal.txt
    per_test_timeout :: timeout_internal_set_in_config.txt

  Expected Passes    : 2
  Unexpected Failures: 3

What you had before (an infinite loop and an `XFAIL line) was not helpful because running the above command would **always** hang. One work around if you really want a test case with an infinite loop is to do some trickery with lit.cfg so that when we invoke it from the command line slow.py` doesn't get detected but when we run lit inside lit it is detected.

[jroelofs]

utils/lit/tests/Inputs/per_test_timeout/slow.py

8

I don't understand why "trickery" is needed in order to make the infinite loop testcase work. The test worked in the diff I uploaded to your llvm-dev thread about this...

[delcypher]

• Rename `--max-individual-test-time to --timeout` as requested by Jonathan Roelofs
• Add infinite loop test case as requested by Jonathan Roelofs

[delcypher]

utils/lit/tests/Inputs/per_test_timeout/slow.py

9

If I did lit.py tests/Inputs/per_test_timeout/ lit would try to run the .py scripts (in addition to the .txt) which would hang.

I've changed lit.cfg just to run the .txt scripts. I was mistaken about needing lit.cfg trickery here.
I thought that would also prevent the .py scripts from being in the .txt test cases that invoke lit (which is why I mentioned the lit.cfg trickery) but if you manually specify the test case names it doesn't seem to matter that .py isn't in config.suffixes`.

The latest patch I just uploaded should address this.

[jroelofs]

utils/lit/tests/Inputs/per_test_timeout/lit.cfg

27

Please revert this part of the change.

utils/lit/tests/Inputs/per_test_timeout/slow.py

9

Ohh, I see what you were running into now. I don't think we have to worry about running lit on that particular test directory without the requisite --timeout flag.

I've changed lit.cfg just to run the .txt scripts.

Running lit.py tests/Inputs/per_test_timeout/ should hang, but running lit.py tests/Inputs/per_test_timeout/ --timeout=1 should not.

[delcypher]

utils/lit/tests/Inputs/per_test_timeout/slow.py

9

If this is the case why did the patch you sent to post with the subject [lit] RFC: Per test timeout have timeout.py invoke lit inside lit with the --timeout flag? If you expect the caller to use --timeout= then having an additional test case that invokes lit on the other test cases with that flag seems redundant.

In my current implementation the "lit invoking lit" (timeout_*.txt tests) tests are not redundant because they invoke lit in three different ways

• Use internal shell, set timeout on command line
• Use external shell, set timeout on command line
• Use internal shell, set timeout in configuration file (i.e. don't use command line flag)

I can revert the small change I made to lit.py but this begs the question

• Who is running these tests?
• How does the person running these tests know how to run them correctly and interpret their results?

[jroelofs]

utils/lit/tests/Inputs/per_test_timeout/slow.py

9

If this is the case why did the patch you sent to post with the subject [lit] RFC: Per test timeout have timeout.py invoke lit inside lit with the --timeout flag? If you expect the caller to use --timeout= then having an additional test case that invokes lit on the other test cases with that flag seems redundant.

Because that's how you're supposed to test lit features? Each of the python files directly in path/to/llvm/utils/lit/tests is itself a testcase for lit (some (all?) of which have their own testsuite in the Inputs folder), and that's where I expect the caller to know to have the --timeout= arg. Let me know if I can clarify this any better... it's a little hard to explain.

In my current implementation the "lit invoking lit" (timeout_*.txt tests) tests are not redundant because they invoke lit in three different ways

Use internal shell, set timeout on command line
Use external shell, set timeout on command line
Use internal shell, set timeout in configuration file (i.e. don't use command line flag)

Each of those three ways of invoking lit to test this feature should be its own file in path/to/llvm/utils/lit/tests... i.e. move them up one directory.

Who is running these tests?

People who make changes to lit should be running them. It'd be ideal if they got run as part of llvm's make check, but that isn't the case currently. I've been meaning to put in a patch for this, so thanks for reminding me.

How does the person running these tests know how to run them correctly and interpret their results?

The correct way to run the lit testsuite for lit, is to run lit.py path/to/llvm/utils/lit/tests. The results are inretpreted by the top-level testsuite's tests, and if lit is not broken, they should all pass.

[delcypher]

utils/lit/tests/Inputs/per_test_timeout/slow.py

9

Thanks for clarifying. I had completely misunderstood how the tests were meant to be run (and consequently how they were meant to be written).

I'll try to rewrite the tests.

[jroelofs]

utils/lit/tests/Inputs/per_test_timeout/slow.py

9

Ok, cool... glad we're on the same page now.

One thing I just thought of: you'll need to set up the top-level tests such that they detect whether import psutil fails, so they can decide whether to test --timeout= at all. I'm not sure what the best way to do that would be, but here's an idea to start from: set an 'available feature' based on the presence of psutil, and then add REQUIRES: python-psutil in them.

[delcypher]

• Rewrite tests. Tests now include testing of a per test timeout for GTest tests.

[delcypher]

utils/lit/tests/Inputs/per_test_timeout/slow.py

9

Please see the latest revision (Diff 3). I've rewritten the tests and added some new ones too.

[jroelofs]

utils/lit/lit/main.py

477

s/timeouts/Timeouts/

utils/lit/tests/Inputs/googletest-timeout/DummySubDir/OneTest

31 ↗

(On Diff #40345)

copy-pasta? s/subtest B/subtest C/

utils/lit/tests/Inputs/per_test_timeout/slow.py

9

Cool, thanks!

utils/lit/tests/lit.cfg

53 ↗

(On Diff #40345)

I'd make a note that --timeout functionality will be disabled too.

utils/lit/tests/shtest-timeout.py

12 ↗

(On Diff #40345)

I think you only need one set of CHECK lines after the three sets of RUN lines. Since the only part that's different is the ERR output, only those checks need to be duplicated/specialized.

[delcypher]

utils/lit/lit/main.py

477

Will fix.

utils/lit/tests/Inputs/googletest-timeout/DummySubDir/OneTest

31 ↗

(On Diff #40345)

Oops yes that's a mistake.

utils/lit/tests/lit.cfg

53 ↗

(On Diff #40345)

I'll make a note of it.

utils/lit/tests/shtest-timeout.py

12 ↗

(On Diff #40345)

That's a good idea. I'll do that.

[delcypher]

• Fix some typos
• Note that --timeout will be disabled in the warning emitted during testing if psutil can't be found
• Add TIMEOUT tests to the list of failed tests that are listed in main.py

[delcypher]

Hopefully that's most of the testing issues fixed.

[jroelofs]

LGTM with one more fix:

utils/lit/tests/shtest-timeout.py

41 ↗

(On Diff #40355)

It'd be better to just put them here, and delete this comment (as well as the other one).

[delcypher]

• Minor re-order of FileCheck CHECK directives

[delcypher]

utils/lit/tests/shtest-timeout.py

41 ↗

(On Diff #40356)

Done

[delcypher]

• Made lit output more useful (when -v is passed) when a timeout occurs, the exit code, stdout and stderr are now shown. If using the internal shell it also indicated which command(s) were killed by the timeout.
• Improve tests to check for the improved output from lit when a timeout occurs.

[jroelofs]

Looks good.

utils/lit/lit/TestRunner.py

582

Oh cool, I've never noticed Phabricator highlighting rebase changes before.

[delcypher]

utils/lit/lit/TestRunner.py

582

I don't see anything different. Maybe that's because you added a comment?

That change (r253556) was a bug I found in lit while working on this patch. I committed that fix directly to trunk
because I thought it was trivial enough to not need reviewing.

[MatzeB]

This is a good feature to have.

I think this implementation will not apply a timeout to the TestRunner.py:executeShCmd codepath (which seems to be a /bin/sh emulation for windows users).
The requirement of an external python package is unfortunate, what about things possible with built-in python libraries such as "os.kill(pid, signal.SIGKILL)" or "resource.setrlimit(resource.RLIMIT_CPU ...)"?

[delcypher]

In D14706#303269, @MatzeB wrote:

This is a good feature to have.

Thanks for taking a look

I think this implementation will not apply a timeout to the TestRunner.py:executeShCmd codepath (which seems to be a /bin/sh emulation for windows users).

That is incorrect. The timeout is applied on all code paths when the timeout is
enabled. For the executeShCmd() (internal shell) case the timeout is
implemented in a different way to lit.util.executeCommand() (external shell)
but this definitely works (at least on Linux). I added tests cases for the three
cases: External Shell, Internal Shell and GTest.

The requirement of an external python package is unfortunate, what about things possible with built-in python libraries such as "os.kill(pid, signal.SIGKILL)" or "resource.setrlimit(resource.RLIMIT_CPU ...)"?

While it is not impossible to remove the dependency on the pustil module,
doing so would be difficult due to platform differences. The issue here is
that when killing a process, we must kill the process and its children (and
their children and so on...). If we don't do this we will have processes left
behind that can continue executing past the timeout.

This is even an issue in the most basic case where we use the external shell to
execute a single command because the parent process is the shell and the child
is the actual command we wanted to run.

There are ways to achieve killing of a group of processes on Linux (not sure
about OSX) by having the parent be in its own session (via
preexe_fn=cos.setsid()) and then later killing using os.killpg() but
that isn't available on Windows and so isn't portable. The aproach I use is
because pustil provides a way to easily iterate over a process's children
and kill them. I don't know of any cross platform way of doing this using just
Python's standard library.

I have kept the dependency on pustil confined to the
lit.util.killProcessAndChildren() function so in the future we could
potentially remove the dependency. It's not something I can do right now
though as I don't have access to a Windows or OSX machine.

[cmatthews]

I think needing a package that is not included in the Python standard release this is a serious issue. lit has no dependencies. That makes it easy to build and test anywhere.

[delcypher]

In D14706#303390, @cmatthews wrote:

I think needing a package that is not included in the Python standard release this is a serious issue. lit has no dependencies. That makes it easy to build and test anywhere.

It is not a hard dependency. It is only required if you want use the per test timeout. If you don't use the timeout lit will work fine without pustil. Although it would be nice to not have psutil as a dependency I'd like to point out that:

• Installing pustil is ridiculously easy. If you are root its just a single command`pip install psutil`, if you're not root you can just use virtualenv and then install it there.
• Not using pustil will likely lead to less portability (at least initially). The pustil module is awesome and a lot of hard work has gone into making it work on all platforms. It's likely that our attempts to emulate this work inside lit would not be as portable.
• lit is used by llvm developers or it is obtained via the package on PyPi. In the PyPi case pustil can be made a dependency so users will automatically get pustil. In the case for LLVM developers I believe that they are competent enough to know how to install pustil if they need it.

[jroelofs]

When I tried to implement this a year ago, it turned out not to be possible on python 2.7 (without the external library). The features needed to make it work weren't added until 3.2, I believe.

I think having psutil as a soft dependency is fine.

utils/lit/lit/TestRunner.py

582

I was looking at the diff of Diff 5 vs Diff 6. And yeah, I'd consider that trivial too.

[delcypher]

• Downgrade warning about `--timeout=` command line option overriding the LitConfig setting to a note
• To not print the per test timeout when running
• Improve testing of `--timeout=` overriding the LitConfig setting
• Remove implementing timeout from TODO
• Add notes on improving per test timeout implementation in TODO

utils/lit/lit/LitConfig.py

 from __future__ import absolute_import
 import inspect
 import os
 import sys
 
 import lit.Test
 import lit.formats
 import lit.TestingConfig
 import lit.util
 
-class LitConfig:
+# LitConfig must be a new style class for properties to work
+class LitConfig(object):
     """LitConfig - Configuration data for a 'lit' test runner instance, shared
     across all tests.
 
     The LitConfig object is also used to communicate with client configuration
     files, it is always passed in as the global variable 'lit' so that
     configuration files can access common functionality and internal components
     easily.
     """
 
     def __init__(self, progname, path, quiet,
                  useValgrind, valgrindLeakCheck, valgrindArgs,
                  noExecute, debug, isWindows,
-                 params, config_prefix = None):
+                 params, config_prefix = None,
+                 maxIndividualTestTime = 0):
         # The name of the test runner.
         self.progname = progname
         # The items to add to the PATH environment variable.
         self.path = [str(p) for p in path]
         self.quiet = bool(quiet)
         self.useValgrind = bool(useValgrind)
         self.valgrindLeakCheck = bool(valgrindLeakCheck)
         self.valgrindUserArgs = list(valgrindArgs)
def __init__(self, progname, path, quiet,
                                  '--error-exitcode=123']
             if self.valgrindLeakCheck:
                 self.valgrindArgs.append('--leak-check=full')
             else:
                 # The default is 'summary'.
                 self.valgrindArgs.append('--leak-check=no')
             self.valgrindArgs.extend(self.valgrindUserArgs)
 
+        self.maxIndividualTestTime = maxIndividualTestTime
+
+    @property
+    def maxIndividualTestTime(self):
+        """
+            Interface for getting maximum time to spend executing
+            a single test
+        """
+        return self._maxIndividualTestTime
+
+    @maxIndividualTestTime.setter
+    def maxIndividualTestTime(self, value):
+        """
+            Interface for setting maximum time to spend executing
+            a single test
+        """
+        self._maxIndividualTestTime = value
+        if self.maxIndividualTestTime > 0:
+            # The current implementation needs psutil to set
+            # a timeout per test. Check it's available.
+            # See lit.util.killProcessAndChildren()
+            try:
+                import psutil
+            except ImportError:
+                self.fatal("Setting a timeout per test requires the"
+                           " Python psutil module but it could not be"
+                           " found. Try installing it via pip or via"
+                           " your operating system's package manager.")
+        elif self.maxIndividualTestTime < 0:
+            self.fatal('The timeout per test must be >= 0 seconds')
 
     def load_config(self, config, path):
         """load_config(config, path) - Load a config object from an alternate
         path."""
         if self.debug:
             self.note('load_config from %r' % path)
         config.load_from_path(path, self)
         return config

utils/lit/lit/Test.py

 
 PASS        = ResultCode('PASS', False)
 FLAKYPASS   = ResultCode('FLAKYPASS', False)
 XFAIL       = ResultCode('XFAIL', False)
 FAIL        = ResultCode('FAIL', True)
 XPASS       = ResultCode('XPASS', True)
 UNRESOLVED  = ResultCode('UNRESOLVED', True)
 UNSUPPORTED = ResultCode('UNSUPPORTED', False)
+TIMEOUT     = ResultCode('TIMEOUT', True)
 
 # Test metric values.
 
 class MetricValue(object):
     def format(self):
         """
         format() -> str
 

utils/lit/lit/TestRunner.py

 from __future__ import absolute_import
 import os, signal, subprocess, sys
 import re
 import platform
 import tempfile
+import threading
 
 import lit.ShUtil as ShUtil
 import lit.Test as Test
 import lit.util
 from lit.util import to_bytes, to_string
 
 class InternalShellError(Exception):
     def __init__(self, command, message):
         self.command = command
         self.message = message
 
+class TestTimeoutError(Exception):
+    def __init__(self, command, message):
+        self.command = command
+        self.message = message
+
 kIsWindows = platform.system() == 'Windows'
 
 # Don't use close_fds on Windows.
 kUseCloseFDs = not kIsWindows
 
 # Use temporary files to replace /dev/null on Windows.
 kAvoidDevNull = kIsWindows
 
 class ShellEnvironment(object):
 
     """Mutable shell environment containing things like CWD and env vars.
 
     Environment variables are not implemented, but cwd tracking is.
     """
 
     def __init__(self, cwd, env):
         self.cwd = cwd
         self.env = dict(env)
 
-def executeShCmd(cmd, shenv, results):
+class TimeoutHelper(object):
+    """
+        Object used to helper manage enforcing a timeout in
+        _executeShCmd(). It is passed through recursive calls
+        to collect processes that have been executed so that when
+        the timeout happens they can be killed.
+    """
+    def __init__(self, timeout):
+        self.timeout = timeout
+        self._procs = []
+        self._timeoutReached = False
+        self._doneKillPass = False
+        # This lock will be used to protect concurrent access
+        # to _procs and _doneKillPass
+        self._lock = None
+        self._timer = None
+
+    def cancel(self):
+        if not self.active():
+            return
+        self._timer.cancel()
+
+    def active(self):
+        return self.timeout > 0
+
+    def addProcess(self, proc):
+        if not self.active():
+            return
+        needToRunKill = False
+        with self._lock:
+            self._procs.append(proc)
+            # Avoid re-entering the lock by finding out if kill needs to be run
+            # again here but call it if necessary once we have left the lock.
+            # We could use a reentrant lock here instead but this code seems
+            # clearer to me.
+            needToRunKill = self._doneKillPass
+
+        # The initial call to _kill() from the timer thread already happened so
+        # we need to call it again from this thread, otherwise this process
+        # will be left to run even though the timeout was already hit
+        if needToRunKill:
+            assert self.timeoutReached()
+            self._kill()
+
+    def startTimer(self):
+        if not self.active():
+            return
+
+        # Do some late initialisation that's only needed
+        # if there is a timeout set
+        self._lock = threading.Lock()
+        self._timer = threading.Timer(self.timeout, self._handleTimeoutReached)
+        self._timer.start()
+
+    def _handleTimeoutReached(self):
+        self._timeoutReached = True
+        self._kill()
+
+    def timeoutReached(self):
+        return self._timeoutReached
+
+    def _kill(self):
+        """
+            This method may be called multiple times as we might get unlucky
+            and be in the middle of creating a new process in _executeShCmd()
+            which won't yet be in ``self._procs``. By locking here and in
+            addProcess() we should be able to kill processes launched after
+            the initial call to _kill()
+        """
+        with self._lock:
+            for p in self._procs:
+                lit.util.killProcessAndChildren(p.pid)
+            # Empty the list and note that we've done a pass over the list
+            self._procs = [] # Python2 doesn't have list.clear()
+            self._doneKillPass = True
+
+def executeShCmd(cmd, shenv, results, timeout=0):
+    """
+        Wrapper around _executeShCmd that handles
+        timeout
+    """
+    # Use the helper even when no timeout is required to make
+    # other code simpler (i.e. avoid bunch of ``!= None`` checks)
+    timeoutHelper = TimeoutHelper(timeout)
+    if timeout > 0:
+        timeoutHelper.startTimer()
+    finalExitCode = _executeShCmd(cmd, shenv, results, timeoutHelper)
+    timeoutHelper.cancel()
+    if timeoutHelper.timeoutReached():
+        raise TestTimeoutError(cmd, 'Reached test timeout of {} seconds'.format(timeout))
+    return finalExitCode
+
+
+def _executeShCmd(cmd, shenv, results, timeoutHelper):
+    if timeoutHelper.timeoutReached():
+        # Prevent further recursion if the timeout has been hit
+        # as we should try avoid launching more processes.
+        return None
+
     if isinstance(cmd, ShUtil.Seq):
         if cmd.op == ';':
-            res = executeShCmd(cmd.lhs, shenv, results)
-            return executeShCmd(cmd.rhs, shenv, results)
+            res = _executeShCmd(cmd.lhs, shenv, results, timeoutHelper)
+            return _executeShCmd(cmd.rhs, shenv, results, timeoutHelper)
 
         if cmd.op == '&':
             raise InternalShellError(cmd,"unsupported shell operator: '&'")
 
         if cmd.op == '||':
-            res = executeShCmd(cmd.lhs, shenv, results)
+            res = _executeShCmd(cmd.lhs, shenv, results, timeoutHelper)
             if res != 0:
-                res = executeShCmd(cmd.rhs, shenv, results)
+                res = _executeShCmd(cmd.rhs, shenv, results, timeoutHelper)
             return res
 
         if cmd.op == '&&':
-            res = executeShCmd(cmd.lhs, shenv, results)
+            res = _executeShCmd(cmd.lhs, shenv, results, timeoutHelper)
             if res is None:
                 return res
 
             if res == 0:
-                res = executeShCmd(cmd.rhs, shenv, results)
+                res = _executeShCmd(cmd.rhs, shenv, results, timeoutHelper)
             return res
 
         raise ValueError('Unknown shell command: %r' % cmd.op)
     assert isinstance(cmd, ShUtil.Pipeline)
 
     # Handle shell builtins first.
     if cmd.commands[0].args[0] == 'cd':
         if len(cmd.commands) != 1:
for i,j in enumerate(cmd.commands):
         try:
             procs.append(subprocess.Popen(args, cwd=cmd_shenv.cwd,
                                           executable = executable,
                                           stdin = stdin,
                                           stdout = stdout,
                                           stderr = stderr,
                                           env = cmd_shenv.env,
                                           close_fds = kUseCloseFDs))
+            # Let the helper know about this process
+            timeoutHelper.addProcess(procs[-1])
         except OSError as e:
             raise InternalShellError(j, 'Could not create process ({}) due to {}'.format(executable, e))
 
         # Immediately close stdin for any process taking stdin from us.
         if stdin == subprocess.PIPE:
             procs[-1].stdin.close()
             procs[-1].stdin = None
 
def executeScriptInternal(test, litConfig, tmpBase, commands, cwd):
 
     cmd = cmds[0]
     for c in cmds[1:]:
         cmd = ShUtil.Seq(cmd, '&&', c)
 
     results = []
     try:
         shenv = ShellEnvironment(cwd, test.config.environment)
-        exitCode = executeShCmd(cmd, shenv, results)
+        exitCode = executeShCmd(cmd, shenv, results, timeout=litConfig.maxIndividualTestTime)
     except InternalShellError:
         e = sys.exc_info()[1]
         exitCode = 127
         results.append((e.command, '', e.message, exitCode))
 
     out = err = ''
     for i,(cmd, cmd_out,cmd_err,res) in enumerate(results):
         out += 'Command %d: %s\n' % (i, ' '.join('"%s"' % s for s in cmd.args))
else:
             command = [bashPath, script]
         else:
             command = ['/bin/sh', script]
         if litConfig.useValgrind:
             # FIXME: Running valgrind on sh is overkill. We probably could just
             # run on clang with no real loss.
             command = litConfig.valgrindArgs + command
 
+    try:
         return lit.util.executeCommand(command, cwd=cwd,
-                                   env=test.config.environment)
+                                       env=test.config.environment,
+                                       timeout=litConfig.maxIndividualTestTime)
+    except lit.util.ExecuteCommandTimeoutException:
+        raise TestTimeoutError(command, 'Hit timeout of {} seconds'.format(litConfig.maxIndividualTestTime))
 
 def parseIntegratedTestScriptCommands(source_path, keywords):
     """
     parseIntegratedTestScriptCommands(source_path) -> commands
 
     Parse the commands in an integrated test script file into a list of
     (line_number, command_type, line).
     """
def processLine(ln):
         # Apply substitutions
         for a,b in substitutions:
             if kIsWindows:
                 b = b.replace("\\","\\\\")
             ln = re.sub(a, b, ln)
 
         # Strip the trailing newline and any extra whitespace.
         return ln.strip()
     return map(processLine, script)
 
 def parseIntegratedTestScript(test, require_script=True):
     """parseIntegratedTestScript - Scan an LLVM/Clang style integrated test
     script and extract the lines to 'RUN' as well as 'XFAIL' and 'REQUIRES'
     and 'UNSUPPORTED' information. If 'require_script' is False an empty script
     may be returned. This can be used for test formats where the actual script
     is optional or ignored.
     """
def parseIntegratedTestScript(test, require_script=True):
 
     return script
 
 def _runShTest(test, litConfig, useExternalSh, script, tmpBase):
     # Create the output directory if it does not already exist.
     lit.util.mkdir_p(os.path.dirname(tmpBase))
 
     execdir = os.path.dirname(test.getExecPath())
+    try:
         if useExternalSh:
             res = executeScript(test, litConfig, tmpBase, script, execdir)
         else:
             res = executeScriptInternal(test, litConfig, tmpBase, script, execdir)
         if isinstance(res, lit.Test.Result):
             return res
+    except TestTimeoutError as e:
+        return lit.Test.Result(Test.TIMEOUT, e.message)
 
     out,err,exitCode = res
     if exitCode == 0:
         status = Test.PASS
     else:
         status = Test.FAIL
 
     # Form the output log.

utils/lit/lit/formats/googletest.py

def execute(self, test, litConfig):
 
         cmd = [testPath, '--gtest_filter=' + testName]
         if litConfig.useValgrind:
             cmd = litConfig.valgrindArgs + cmd
 
         if litConfig.noExecute:
             return lit.Test.PASS, ''
 
+        try:
             out, err, exitCode = lit.util.executeCommand(
-            cmd, env=test.config.environment)
+                cmd, env=test.config.environment,
+                timeout=litConfig.maxIndividualTestTime)
+        except lit.util.ExecuteCommandTimeoutException:
+            return (lit.Test.TIMEOUT,
+                    'Reached timeout of {} seconds'.format(
+                        litConfig.maxIndividualTestTime)
+                   )
 
         if exitCode:
             return lit.Test.FAIL, out + err
 
         passing_test_line = '[  PASSED  ] 1 test.'
         if passing_test_line not in out:
             msg = ('Unable to find %r in gtest output:\n\n%s%s' %
                    (passing_test_line, out, err))
             return lit.Test.UNRESOLVED, msg
 
         return lit.Test.PASS,''
 

utils/lit/lit/main.py

group.add_option("", "--time-tests", dest="timeTests",
                      help="Track elapsed wall time for each test",
                      action="store_true", default=False)
     group.add_option("", "--no-execute", dest="noExecute",
                      help="Don't execute any tests (assume PASS)",
                      action="store_true", default=False)
     group.add_option("", "--xunit-xml-output", dest="xunit_output_file",
                       help=("Write XUnit-compatible XML test reports to the"
                             " specified file"), default=None)
+    group.add_option("", "--max-individual-test-time", dest="maxIndividualTestTime",
+                     help="Maximum time to spend running a single test (in seconds)."
+                     "0 means no time limit. [Default: 0]",
+                    type=int, default=None)
     parser.add_option_group(group)
 
     group = OptionGroup(parser, "Test Selection")
     group.add_option("", "--max-tests", dest="maxTests", metavar="N",
                      help="Maximum number of tests to run",
                      action="store", type=int, default=None)
     group.add_option("", "--max-time", dest="maxTime", metavar="N",
                      help="Maximum time to spend testing (in seconds)",
# threads by default there.
     userParams = dict(builtinParameters)
     for entry in opts.userParameters:
         if '=' not in entry:
             name,val = entry,''
         else:
             name,val = entry.split('=', 1)
         userParams[name] = val
 
+    # Decide what the requested maximum indvidual test time should be
+    if opts.maxIndividualTestTime != None:
+        maxIndividualTestTime = opts.maxIndividualTestTime
+    else:
+        # Default is zero
+        maxIndividualTestTime = 0
+
+
     # Create the global config object.
     litConfig = lit.LitConfig.LitConfig(
         progname = os.path.basename(sys.argv[0]),
         path = opts.path,
         quiet = opts.quiet,
         useValgrind = opts.useValgrind,
         valgrindLeakCheck = opts.valgrindLeakCheck,
         valgrindArgs = opts.valgrindArgs,
         noExecute = opts.noExecute,
         debug = opts.debug,
         isWindows = isWindows,
         params = userParams,
-        config_prefix = opts.configPrefix)
+        config_prefix = opts.configPrefix,
+        maxIndividualTestTime = maxIndividualTestTime)
 
     # Perform test discovery.
     run = lit.run.Run(litConfig,
                       lit.discovery.find_tests_for_inputs(litConfig, inputs))
 
+    # After test discovery the configuration might have changed
+    # the maxIndividualTestTime. If we explicitly set this on the
+    # command line then override what was set in the test configuration
+    if opts.maxIndividualTestTime != None:
+        if opts.maxIndividualTestTime != litConfig.maxIndividualTestTime:
+            litConfig.warning(('The test suite configuration requested an individual'
+                ' test timeout of {0} seconds but a timeout of {1} seconds was'
+                ' requested on the command line. Forcing timeout to be {1}'
+                ' seconds')
+                .format(litConfig.maxIndividualTestTime,
+                        opts.maxIndividualTestTime))
+            litConfig.maxIndividualTestTime = opts.maxIndividualTestTime
+
     if opts.showSuites or opts.showTests:
         # Aggregate the tests by suite.
         suitesAndTests = {}
         for result_test in run.tests:
             if result_test.suite not in suitesAndTests:
                 suitesAndTests[result_test.suite] = []
             suitesAndTests[result_test.suite].append(result_test)
         suitesAndTests = list(suitesAndTests.items())
# threads by default there.
     except:
         pass
 
     extra = ''
     if len(run.tests) != numTotalTests:
         extra = ' of %d' % numTotalTests
     header = '-- Testing: %d%s tests, %d threads --'%(len(run.tests), extra,
                                                       opts.numThreads)
-
+    print('Using individual test timeout of {} seconds'.format(litConfig.maxIndividualTestTime))
     progressBar = None
     if not opts.quiet:
         if opts.succinct and opts.useProgressBar:
             try:
                 tc = lit.ProgressBar.TerminalController()
                 progressBar = lit.ProgressBar.ProgressBar(tc, header)
             except ValueError:
                 print(header)
if opts.timeTests and run.tests:
         lit.util.printHistogram(test_times, title='Tests')
 
     for name,code in (('Expected Passes    ', lit.Test.PASS),
                       ('Passes With Retry  ', lit.Test.FLAKYPASS),
                       ('Expected Failures  ', lit.Test.XFAIL),
                       ('Unsupported Tests  ', lit.Test.UNSUPPORTED),
                       ('Unresolved Tests   ', lit.Test.UNRESOLVED),
                       ('Unexpected Passes  ', lit.Test.XPASS),
-                      ('Unexpected Failures', lit.Test.FAIL)):
+                      ('Unexpected Failures', lit.Test.FAIL),
+                      ('Individual timeouts', lit.Test.TIMEOUT)):
         if opts.quiet and not code.isFailure:
             continue
         N = len(byCode.get(code,[]))
         if N:
             print('  %s: %d' % (name,N))
 
     if opts.xunit_output_file:
         # Collect the tests, indexed by test suite

utils/lit/lit/util.py

 import errno
 import itertools
 import math
 import os
 import platform
 import signal
 import subprocess
 import sys
+import threading
 
 def to_bytes(str):
     # Encode to UTF-8 to get binary data.
     return str.encode('utf-8')
 
 def to_string(bytes):
     if isinstance(bytes, str):
         return bytes
def printHistogram(items, title = 'Items'):
     print(hr)
     for i,row in enumerate(histo):
         pct = float(len(row)) / len(items)
         w = int(barW * pct)
         print("[%*.*fs,%*.*fs) :: [%s%s] :: [%*d/%*d]" % (
             pDigits, pfDigits, i*barH, pDigits, pfDigits, (i+1)*barH,
             '*'*w, ' '*(barW-w), cDigits, len(row), cDigits, len(items)))
 
+class ExecuteCommandTimeoutException(Exception):
+    pass
 # Close extra file handles on UNIX (on Windows this cannot be done while
 # also redirecting input).
 kUseCloseFDs = not (platform.system() == 'Windows')
-def executeCommand(command, cwd=None, env=None, input=None):
+def executeCommand(command, cwd=None, env=None, input=None, timeout=0):
     p = subprocess.Popen(command, cwd=cwd,
                          stdin=subprocess.PIPE,
                          stdout=subprocess.PIPE,
                          stderr=subprocess.PIPE,
                          env=env, close_fds=kUseCloseFDs)
+    timerObject = None
+    # FIXME: Because of the way nested function scopes work in Python 2.x we
+    # need to use a reference to a mutable object rather than a plain
+    # bool. In Python 3 we could use the "nonlocal" keyword but we need
+    # to support Python 2 as well.
+    hitTimeOut = [False]
+    try:
+        if timeout > 0:
+            def killProcess():
+                # We may be invoking a shell so we need to kill the
+                # process and all its children.
+                hitTimeOut[0] = True
+                killProcessAndChildren(p.pid)
+
+            timerObject = threading.Timer(timeout, killProcess)
+            timerObject.start()
+
         out,err = p.communicate(input=input)
         exitCode = p.wait()
+    finally:
+        if timerObject != None:
+            timerObject.cancel()
+
+    if hitTimeOut[0]:
+        raise ExecuteCommandTimeoutException()
 
     # Detect Ctrl-C in subprocess.
     if exitCode == -signal.SIGINT:
         raise KeyboardInterrupt
 
     # Ensure the resulting output is always of string type.
     out = convert_string(out)
     err = convert_string(err)
if 'darwin' in config.target_triple:
             out = out.strip()
             res = cmd.wait()
         except OSError:
             res = -1
         if res == 0 and out:
             sdk_path = out
             lit_config.note('using SDKROOT: %r' % sdk_path)
             config.environment['SDKROOT'] = sdk_path
+
+def killProcessAndChildren(pid):
+    """
+    This function kills a process with ``pid`` and all its
+    running children (recursively). It is currently implemented
+    using the psutil module which provides a simple platform
+    neutral implementation.
+
+    TODO: Reimplement this without using psutil so we can
+          remove our dependency on it.
+    """
+    import psutil
+    try:
+        psutilProc = psutil.Process(pid)
+        for child in psutilProc.children(recursive=True):
+            try:
+                child.kill()
+            except psutil.NoSuchProcess:
+                pass
+        psutilProc.kill()
+    except psutil.NoSuchProcess:
+        pass

utils/lit/tests/Inputs/per_test_timeout/lit.cfg

+# -*- Python -*-
+import os
+import sys
+
+import lit.formats
+
+config.name = 'per_test_timeout'
+
+shellType = lit_config.params.get('external', '1')
+
+if shellType == '0':
+    lit_config.note('Using internal shell')
+    externalShell = False
+else:
+    lit_config.note('Using external shell')
+    externalShell = True
+
+configSetTimeout = lit_config.params.get('set_timeout', '0')
+
+if configSetTimeout == '1':
+    # Try setting the max individual test time in the configuration
+    lit_config.maxIndividualTestTime = 1
+
+# FIXME: GTest needs testing too
+config.test_format = lit.formats.ShTest(execute_external=externalShell)
+config.suffixes = ['.txt', '.py']
+config.test_source_root = os.path.dirname(__file__)
+config.test_exec_root = config.test_source_root
+config.target_triple = '(unused)'
+src_root = os.path.join(config.test_source_root, '..')
+config.environment['PYTHONPATH'] = src_root
+config.substitutions.append(('%{python}', sys.executable))
+config.substitutions.append(('%{inputs}', config.test_source_root))
+config.substitutions.append(('%{lit}', sys.modules['__main__'].__file__))

utils/lit/tests/Inputs/per_test_timeout/short.py

+# RUN: %{python} %s
+from __future__ import print_function
+
+import sys
+
+print("short program")

utils/lit/tests/Inputs/per_test_timeout/slow.py

+# RUN: %{python} %s
+from __future__ import print_function
+
+import time
+import sys
+
+print("slow program")
+time.sleep(6)

utils/lit/tests/Inputs/per_test_timeout/timeout_external.txt

+# RUN: not %{lit} \
+# RUN: %{inputs}/slow.py \
+# RUN: %{inputs}/short.py \
+# RUN: -j 1 -v --debug --max-individual-test-time 1 --param external=1 > %t.out 2> %t.err
+# RUN: FileCheck --check-prefix=CHECK-OUT < %t.out %s
+# RUN: FileCheck --check-prefix=CHECK-ERR < %t.err %s
+
+# CHECK-OUT: PASS: per_test_timeout :: short.py
+# CHECK-OUT: TIMEOUT: per_test_timeout :: slow.py
+# CHECK-OUT: Expected Passes{{ *}}: 1
+# CHECK-OUT: Individual timeouts{{ *}}: 1
+# CHECK-ERR: Using external shell

utils/lit/tests/Inputs/per_test_timeout/timeout_internal.txt

+# RUN: not %{lit} \
+# RUN: %{inputs}/slow.py \
+# RUN: %{inputs}/short.py \
+# RUN: -j 1 -v --debug --max-individual-test-time 1 --param external=0 > %t.out 2> %t.err
+# RUN: FileCheck --check-prefix=CHECK-OUT < %t.out %s
+# RUN: FileCheck --check-prefix=CHECK-ERR < %t.err %s
+
+# CHECK-OUT: PASS: per_test_timeout :: short.py
+# CHECK-OUT: TIMEOUT: per_test_timeout :: slow.py
+# CHECK-OUT: Expected Passes{{ *}}: 1
+# CHECK-OUT: Individual timeouts{{ *}}: 1
+# CHECK-ERR: Using internal shell

utils/lit/tests/Inputs/per_test_timeout/timeout_internal_set_in_config.txt

+# RUN: not %{lit} \
+# RUN: %{inputs}/slow.py \
+# RUN: %{inputs}/short.py \
+# RUN: -j 1 -v --debug --param external=0 --param set_timeout=1 > %t.out 2> %t.err
+# RUN: FileCheck --check-prefix=CHECK-OUT < %t.out %s
+# RUN: FileCheck --check-prefix=CHECK-ERR < %t.err %s
+
+# CHECK-OUT: PASS: per_test_timeout :: short.py
+# CHECK-OUT: TIMEOUT: per_test_timeout :: slow.py
+# CHECK-OUT: Expected Passes{{ *}}: 1
+# CHECK-OUT: Individual timeouts{{ *}}: 1
+# CHECK-ERR: Using internal shell