[lit] Enable the use of custom user-defined lit commands

[zturner]

Summary

Currently lit supports running shell commands through the use of the RUN: prefix. This patch allows individual test suites to install their own run handlers that can do things other than run shell commands. RUN commands still work as they always do, just that now if a different kind of command appears it will be appropriately sequenced along with the run command.

The commands the user installs can execute arbitrary Python code. As such, they can in theory write directly to stdout or stderr, but a well-behaved command should return its stdout and stderr from the function so that this can be reported to the user in a manner consistent with output from RUN lines.

The motivating use case for this is being able to provide a richer and more powerful syntax by which to compile test programs in LLDB tests. Currently everything is based off of substitutions and explicitly shell commands, but this is problematic when you get into interesting compilation scenarios.

For example, one could imagine wanting to write a test that tested the behavior of the debugger with optimized code. Each driver has different sets of flags that control the optimization behavior.

Another example is in cross-compilation scenarios. Certain types of PDB tests don't need to run a process, so the tests can be run anywhere, but they need to be linked with special flags to avoid pulling in system libraries.

We can try to make substitutions for all of these cases, but it will quickly become unwieldy and you will end up with a command line like: RUN: %cxx %debug %opt %norun, and this still isn't as flexible as you'd like.

With this patch, we could (in theory) do the compilation directly from Python. Instead of a shell command like above, we could write something like:

COMPILE: source=%p/Inputs/foo.cpp \
COMPILE:     mode=debug \
COMPILE:     opt=none \
COMPILE:     link=no \
COMPILE:     output=%t.o \
COMPILE:     clean=yes

and let the function figure out how best to do this for each platform. This is similar in spirit to how lldb's dotest.py already works with its platform specific builders, but the mechanism here is general enough that it can be used for anything a test suite wants, not just compiling.

[stella.stamenova]

LGTM as long as the tests are all still passing :)

[rnk]

I think the code to implement this is fine, but before we add this complexity to lit, I just wanted to know if other folks who work on the LLDB test suite are on board and want to use this approach to abstract over building apps for different targets. I see @stella.stamenova is, but I wanted to hear from other people involved in the LLDB lit test suite stuff. The %compile %debug %opt %s lit substitution approach is limiting, but do people feel strongly that this is much better?

llvm/utils/lit/tests/Inputs/shtest-keyword-command/keyword_helper.py

2

Oh, the joys of multiprocessing and pickling...

[zturner]

+vsk. One thing I need input from LLDB people on is whether this seems like a good "general direction" for writing lit tests. Does wrapping compilation and linker commands behind a python function seem like it would address these use cases?

Can we think of any other use cases for custom commands like this?

[vsk]

For compiling/linking, I think we can get by using lit substitutions to fill in platform-specific options? iOS testing for Swift is done this way (both on-device and simulator), as is testing for the profiling runtime. Dan and @filcab are more active in the area of sanitizer runtime testing, so they may have more informed opinions to share about how well that model works.

I’m not sure what we’d use custom lit commands for beyond compiling tests.

[labath]

I think that something like this would go a long way towards solving the problems with lit tests we're having in lldb.

However, the part that is not clear to me is whether it is actually necessary to modify lit (shtest) to achieve this. It seems to me an equivalent effect to the command from the motivating example could be achieved via something like

RUN: %compile --source=%p/Inputs/foo.cpp --mode=debug --opt=none --link=no --output=%t.o --clean=yes

where %compile expands to a python script living somewhere in the lldb repository. This script could do the same thing that the implementation of COMPILE: would do, except it would be done in a separate process.

The only downside of that I see is the extra process will incur some overhead, slowing down the testing, but I am not sure if we care about that (or if it would even be measurable). OTOH, the benefits are:

• decreased complexity of lit
• decreased level of surprise of developers seeing new lit commands
• easier reproducibility of tests when debugging (just copy paste the %compile run-line to rebuild the executable)

[zturner]

In D54731#1305183, @labath wrote:

I think that something like this would go a long way towards solving the problems with lit tests we're having in lldb.

However, the part that is not clear to me is whether it is actually necessary to modify lit (shtest) to achieve this. It seems to me an equivalent effect to the command from the motivating example could be achieved via something like

RUN: %compile --source=%p/Inputs/foo.cpp --mode=debug --opt=none --link=no --output=%t.o --clean=yes

where %compile expands to a python script living somewhere in the lldb repository. This script could do the same thing that the implementation of COMPILE: would do, except it would be done in a separate process.

The only downside of that I see is the extra process will incur some overhead, slowing down the testing, but I am not sure if we care about that (or if it would even be measurable). OTOH, the benefits are:

• decreased complexity of lit
• decreased level of surprise of developers seeing new lit commands
• easier reproducibility of tests when debugging (just copy paste the %compile run-line to rebuild the executable)

I did consider this, and I'm still open to the possibility of doing things this way. Two reasons I chose this route instead are:

1. We have a lot of setup that runs in lit before we ever get to this point, and a builder could re-use this. For example, the environment, any additional lit configuration parameters specified on the command line, etc. We could of course pass these in to the compile.py script via hidden arguments, so this isn't a total blocker, it was just something I thought of.

2. We could re-purpose this machinery for other uses. For example, I could imagine re-writing many lldb inline tests in terms of a custom command prefix. For example, here's test/functionalities/data-formatter/dump_dynamic/main.cpp:

class Base {
public:
  Base () = default;
  virtual int func() { return 1; }
  virtual ~Base() = default;
};

class Derived : public Base {
private:
  int m_derived_data;
public:
  Derived () : Base(), m_derived_data(0x0fedbeef) {}
  virtual ~Derived() = default;
  virtual int func() { return m_derived_data; }
};

int main (int argc, char const *argv[])
{
  Base *base = new Derived();
    return 0; //% stream = lldb.SBStream()
    //% base = self.frame().FindVariable("base")
    //% base.SetPreferDynamicValue(lldb.eDynamicDontRunTarget)
    //% base.GetDescription(stream)
    //% if self.TraceOn(): print(stream.GetData())
    //% self.assertTrue(stream.GetData().startswith("(Derived *"))
}

I could imagine writing this as:

class Base {
public:
  Base () = default;
  virtual int func() { return 1; }
  virtual ~Base() = default;
};

class Derived : public Base {
private:
  int m_derived_data;
public:
  Derived () : Base(), m_derived_data(0x0fedbeef) {}
  virtual ~Derived() = default;
  virtual int func() { return m_derived_data; }
};

int main (int argc, char const *argv[])
{
  Base *base = new Derived();
    return 0;
}

//SCRIPT: stream = lldb.SBStream()
//SCRIPT: base = self.frame().FindVariable("base")
//SCRIPT: base.SetPreferDynamicValue(lldb.eDynamicDontRunTarget)
//SCRIPT: base.GetDescription(stream)
//EXPECT: stream.GetData().startswith("(Derived *"))

where the lldb module is loaded in-process similar to how it is with dotest.py. (I do wonder if all lldbinline tests could actually be convereted to lit / FileCheck tests right now, today, using an lldbinit file such as:

script stream = lldb.SBStream()
script base = self.frame().FindVariable("base")
script base.SetPreferDynamicValue(lldb.eDynamicDontRunTarget)
script base.GetDescription(stream)
script stream.GetData()

and then FileCheck'ing that, but I haven't tried and I haven't investigated every single lldbinline test to see if they would all fit into this model.

In any case, the point being that being able to run python code in-process opens up a lot of interesting possibilities, considering that's how all the dotest tests are written. Whether we need that flexibility is open for discussion though. Like I said, i'm willing to give the external script a try if people think we should try a more conservative approach first.

[labath]

I'd go with the "conservative" approach first. The idea of having lldb loaded inside a lit process does not excite me. One of the problems we have with dotest is that when lldb crashes during the test, it takes a part of the test driver with it which causes some tests to be skipped and the complicates the reporting of the result of the crashed test. It's not as bad right now, as there is still the main process left to report some kind of an error (back in the days when tests were run sequentially in a single process, the entire test suite would just stop), but I still think it would be nice to avoid these issues in the new framework.

[aprantl]

Currently lit supports running shell commands through the use of the RUN: prefix. This patch allows individual test suites to install their own run handlers that can do things other than run shell commands. RUN commands still work as they always do, just that now if a different kind of command appears it will be appropriately sequenced along with the run command.

I'm not convinced that this is the best direction to evolve the LLDB testsuite to. Let me know if I'm missing something; I'm willing to be convinced otherwise :-)

It sounds like the problem you want to solve is having a more flexible build system for tests, and the ability to run python code as part of a tests. That is exactly the feature set that dotest.py provides. Tests are written in fully flexible Python, and in order to compile inferiors, we can fan out to a dedicated build system that is really good at compiling programs, namely make.

I don't see how any of your stated goals couldn't be achieved within the existing Makefile.rules. Establishing a second, parallel way of doing something similar would only serve to bifurcate the test infrastructure and make maintenance a lot harder in the future. It also makes the system more difficult to explain to new developers.

Specifically:

The commands the user installs can execute arbitrary Python code.

dotest.py already does that. Currently, we are using lit.py as a test scheduler and dotest.py as an LLDB-specific test harness. I think that's reasonable design.

As such, they can in theory write directly to stdout or stderr, but a well-behaved command should return its stdout and stderr from the function so that this can be reported to the user in a manner consistent with output from RUN lines.

The motivating use case for this is being able to provide a richer and more powerful syntax by which to compile test programs in LLDB tests. Currently everything is based off of substitutions and explicitly shell commands, but this is problematic when you get into interesting compilation scenarios.

I disagree with this statement. Building tests is done in an explicit, portable build system: make. I don't think it is a good idea to *also* add all the complexity of the dotest.py tests to the lit-based tests. Lit-based tests are very useful for certain (specifically non-interactive) use-cases, but if you need more build system support, or need to more complex test logic, I'd rather use make+dotest.py.

For example, one could imagine wanting to write a test that tested the behavior of the debugger with optimized code. Each driver has different sets of flags that control the optimization behavior.

This is mostly a solved problem with our Makefile system.

Another example is in cross-compilation scenarios. Certain types of PDB tests don't need to run a process, so the tests can be run anywhere, but they need to be linked with special flags to avoid pulling in system libraries.

We can try to make substitutions for all of these cases, but it will quickly become unwieldy and you will end up with a command line like: RUN: %cxx %debug %opt %norun, and this still isn't as flexible as you'd like.

With this patch, we could (in theory) do the compilation directly from Python. Instead of a shell command like above, we could write something like:

COMPILE: source=%p/Inputs/foo.cpp \
COMPILE: mode=debug \
COMPILE: opt=none \
COMPILE: link=no \
COMPILE: output=%t.o \
COMPILE: clean=yes
and let the function figure out how best to do this for each platform. This is similar in spirit to how LLDB's dotest.py already works with its platform specific builders, but the mechanism here is general enough that it can be used for anything a test suite wants, not just compiling.

In the dotest tests you generally don't need to write explicit build commands at all. All the platform-specific build logic is implemented once in Makefile.rules and the individual tests merely specify what source files need to be built and whether you want o build a binary or a shared library.

llvm/utils/lit/lit/LitConfig.py

 from __future__ import absolute_import
 import inspect
 import os
 import sys
 
 import lit.Test
 import lit.formats
 import lit.TestingConfig
+import lit.TestRunner
 import lit.util
 
 # LitConfig must be a new style class for properties to work
 class LitConfig(object):
     """LitConfig - Configuration data for a 'lit' test runner instance, shared
     across all tests.
 
     The LitConfig object is also used to communicate with client configuration
def __init__(self, progname, path, quiet,
         self.valgrindLeakCheck = bool(valgrindLeakCheck)
         self.valgrindUserArgs = list(valgrindArgs)
         self.noExecute = noExecute
         self.debug = debug
         self.singleProcess = singleProcess
         self.isWindows = bool(isWindows)
         self.params = dict(params)
         self.bashPath = None
+        self.additionalCommands = []
 
         # Configuration files to look for when discovering test suites.
         self.config_prefix = config_prefix or 'lit'
         self.suffixes = ['cfg.py', 'cfg']
         self.config_names = ['%s.%s' % (self.config_prefix,x) for x in self.suffixes]
         self.site_config_names = ['%s.site.%s' % (self.config_prefix,x) for x in self.suffixes]
         self.local_config_names = ['%s.local.%s' % (self.config_prefix,x) for x in self.suffixes]
 
class LitConfig(object):
     def load_config(self, config, path):
         """load_config(config, path) - Load a config object from an alternate
         path."""
         if self.debug:
             self.note('load_config from %r' % path)
         config.load_from_path(path, self)
         return config
 
+    def installKeywordCommand(self, keyword, command):
+        self.additionalCommands.append((keyword, command))
+
     def getBashPath(self):
         """getBashPath - Get the path to 'bash'"""
         if self.bashPath is not None:
             return self.bashPath
 
         self.bashPath = lit.util.which('bash', os.pathsep.join(self.path))
         if self.bashPath is None:
             self.bashPath = lit.util.which('bash')

llvm/utils/lit/lit/ShCommands.py

def toShell(self, file):
                 raise NotImplementedError('Unable to quote %r' % arg)
 
         for r in self.redirects:
             if len(r[0]) == 1:
                 file.write("%s '%s'" % (r[0][0], r[1]))
             else:
                 file.write("%s%s '%s'" % (r[0][1], r[0][0], r[1]))
 
+class CustomCommand:
+    def __init__(self, command, location, parameter):
+        self.command = command
+        self.location = location
+        self.parameter = parameter
+
 class GlobItem:
     def __init__(self, pattern):
         self.pattern = pattern
 
     def __repr__(self):
         return self.pattern
 
     def __eq__(self, other):

llvm/utils/lit/lit/TestRunner.py

 
 import io
 try:
     from StringIO import StringIO
 except ImportError:
     from io import StringIO
 
 from lit.ShCommands import GlobItem
+from lit.ShCommands import CustomCommand
 import lit.ShUtil as ShUtil
 import lit.Test as Test
 import lit.util
 from lit.util import to_bytes, to_string
 from lit.BooleanExpression import BooleanExpression
 
 class InternalShellError(Exception):
     def __init__(self, command, message):
def _executeShCmd(cmd, shenv, results, timeoutHelper):
 
     if cmd.negate:
         exitCode = not exitCode
 
     return exitCode
 
 def executeScriptInternal(test, litConfig, tmpBase, commands, cwd):
     cmds = []
-    for i, ln in enumerate(commands):
-        ln = commands[i] = re.sub(kPdbgRegex, ": '\\1'; ", ln)
+    for i, (keyword, ln) in enumerate(commands):
+        if keyword == 'RUN:':
+            c = None
+            ln = re.sub(kPdbgRegex, ": '\\1'; ", ln)
             try:
-            cmds.append(ShUtil.ShParser(ln, litConfig.isWindows,
-                                        test.config.pipefail).parse())
+                parser = ShUtil.ShParser(ln, litConfig.isWindows, test.config.pipefail)
+                c = parser.parse()
             except:
                 return lit.Test.Result(Test.FAIL, "shell parser error on: %r" % ln)
-
-    cmd = cmds[0]
-    for c in cmds[1:]:
-        cmd = ShUtil.Seq(cmd, '&&', c)
+            if cmds and not isinstance(cmds[-1], CustomCommand):
+                cmds[-1] = ShUtil.Seq(cmds[-1], '&&', c)
+            else:
+                cmds.append(c)
+        else:
+            match = re.match('%dbg\((.* at line \\d+)\)(.*)', ln)
+            command = next(iter(filter(lambda x : x[0] == keyword, litConfig.additionalCommands)))
+            cmds.append(CustomCommand(command, match.group(1), match.group(2)))
 
     results = []
     timeoutInfo = None
+    for cmd in cmds:
         try:
+            if isinstance(cmd, CustomCommand):
+                fn = cmd.command[1]
+                stdout, stderr = fn(cmd.parameter)
+                result = ShellCommandResult(command=cmd, stdout=stdout, stderr=stderr, exitCode=0, timeoutReached=False)
+                results.append(result)
+                exitCode = 0
+            else:
                 shenv = ShellEnvironment(cwd, test.config.environment)
                 exitCode, timeoutInfo = executeShCmd(cmd, shenv, results, timeout=litConfig.maxIndividualTestTime)
         except InternalShellError:
             e = sys.exc_info()[1]
             exitCode = 127
             results.append(
                 ShellCommandResult(e.command, '', e.message, exitCode, False))
 
     out = err = ''
     for i,result in enumerate(results):
         # Write the command line run.
-        out += '$ %s\n' % (' '.join('"%s"' % s
-                                    for s in result.command.args),)
+        if isinstance(result.command, CustomCommand):
+            temp = result.command.location + result.command.parameter
+        else:
+            temp = ' '.join('"%s"' % s for s in result.command.args)
+        out += '$ %s\n' % (temp,)
 
         # If nothing interesting happened, move on.
         if litConfig.maxIndividualTestTime == 0 and \
                result.exitCode == 0 and \
                not result.stdout.strip() and not result.stderr.strip():
             continue
 
         # Otherwise, something failed or was printed, show it.
substitutions.extend([
             ])
     return substitutions
 
 def applySubstitutions(script, substitutions):
     """Apply substitutions to the script.  Allow full regular expression syntax.
     Replace each matching occurrence of regular expression pattern a with
     substitution b in line ln."""
     def processLine(ln):
+        keyword, ln = ln
         # Apply substitutions
         for a,b in substitutions:
             if kIsWindows:
                 b = b.replace("\\","\\\\")
             ln = re.sub(a, b, ln)
 
         # Strip the trailing newline and any extra whitespace.
-        return ln.strip()
+        return (keyword, ln.strip())
     # Note Python 3 map() gives an iterator rather than a list so explicitly
     # convert to list before returning.
     return list(map(processLine, script))
 
 
 class ParserKind(object):
     """
     An enumeration representing the style of an integrated test keyword or
def __init__(self, keyword, kind, parser=None, initial_value=None):
                              "ParserKind.CUSTOM")
         self.keyword = keyword
         self.kind = kind
         self.parsed_lines = []
         self.value = initial_value
         self.parser = parser
 
         if kind == ParserKind.COMMAND:
-            self.parser = lambda line_number, line, output: \
-                                 self._handleCommand(line_number, line, output,
-                                                     self.keyword)
+            self.parser = self._handleCommand
         elif kind == ParserKind.LIST:
             self.parser = self._handleList
         elif kind == ParserKind.BOOLEAN_EXPR:
             self.parser = self._handleBooleanExpr
         elif kind == ParserKind.TAG:
             self.parser = self._handleTag
         elif kind == ParserKind.CUSTOM:
             if parser is None:
                 raise ValueError("ParserKind.CUSTOM requires a custom parser")
             self.parser = parser
         else:
             raise ValueError("Unknown kind '%s'" % kind)
 
-    def parseLine(self, line_number, line):
+    def parseLine(self, line_number, line, keyword):
         try:
             self.parsed_lines += [(line_number, line)]
-            self.value = self.parser(line_number, line, self.value)
+            self.value = self.parser(line_number, line, self.value, keyword)
         except ValueError as e:
             raise ValueError(str(e) + ("\nin %s directive on test line %d" %
                                        (self.keyword, line_number)))
 
     def getValue(self):
         return self.value
 
     @staticmethod
-    def _handleTag(line_number, line, output):
+    def _handleTag(line_number, line, output, keyword):
         """A helper for parsing TAG type keywords"""
         return (not line.strip() or output)
 
     @staticmethod
     def _handleCommand(line_number, line, output, keyword):
         """A helper for parsing COMMAND type keywords"""
         # Trim trailing whitespace.
         line = line.rstrip()
         # Substitute line number expressions
         line = re.sub('%\(line\)', str(line_number), line)
 
         def replace_line_number(match):
             if match.group(1) == '+':
                 return str(line_number + int(match.group(2)))
             if match.group(1) == '-':
                 return str(line_number - int(match.group(2)))
         line = re.sub('%\(line *([\+-]) *(\d+)\)', replace_line_number, line)
         # Collapse lines with trailing '\\'.
-        if output and output[-1][-1] == '\\':
-            output[-1] = output[-1][:-1] + line
+        if output and output[-1][1][-1] == '\\':
+            output[-1] = (keyword, output[-1][1][:-1] + line)
         else:
             if output is None:
                 output = []
             pdbg = "%dbg({keyword} at line {line_number})".format(
                 keyword=keyword,
                 line_number=line_number)
             assert re.match(kPdbgRegex + "$", pdbg), \
                    "kPdbgRegex expected to match actual %dbg usage"
             line = "{pdbg} {real_command}".format(
                 pdbg=pdbg,
                 real_command=line)
-            output.append(line)
+            output.append((keyword, line))
         return output
 
     @staticmethod
-    def _handleList(line_number, line, output):
+    def _handleList(line_number, line, output, keyword):
         """A parser for LIST type keywords"""
         if output is None:
             output = []
         output.extend([s.strip() for s in line.split(',')])
         return output
 
     @staticmethod
-    def _handleBooleanExpr(line_number, line, output):
+    def _handleBooleanExpr(line_number, line, output, keyword):
         """A parser for BOOLEAN_EXPR type keywords"""
         if output is None:
             output = []
         output.extend([s.strip() for s in line.split(',')])
         # Evaluate each expression to verify syntax.
         # We don't want any results, just the raised ValueError.
         for s in output:
             if s != '*':
                 BooleanExpression.evaluate(s, [])
         return output
 
     @staticmethod
-    def _handleRequiresAny(line_number, line, output):
+    def _handleRequiresAny(line_number, line, output, keyword):
         """A custom parser to transform REQUIRES-ANY: into REQUIRES:"""
 
         # Extract the conditions specified in REQUIRES-ANY: as written.
         conditions = []
-        IntegratedTestKeywordParser._handleList(line_number, line, conditions)
+        IntegratedTestKeywordParser._handleList(line_number, line, conditions, keyword)
 
         # Output a `REQUIRES: a || b || c` expression in its place.
         expression = ' || '.join(conditions)
         IntegratedTestKeywordParser._handleBooleanExpr(line_number,
-                                                       expression, output)
+                                                       expression, output, keyword)
         return output
 
 def parseIntegratedTestScript(test, additional_parsers=[],
                               require_script=True):
     """parseIntegratedTestScript - Scan an LLVM/Clang style integrated test
     script and extract the lines to 'RUN' as well as 'XFAIL' and 'REQUIRES'
     and 'UNSUPPORTED' information.
 
def parseIntegratedTestScript(test, additional_parsers=[],
     # Install user-defined additional parsers.
     for parser in additional_parsers:
         if not isinstance(parser, IntegratedTestKeywordParser):
             raise ValueError('additional parser must be an instance of '
                              'IntegratedTestKeywordParser')
         if parser.keyword in keyword_parsers:
             raise ValueError("Parser for keyword '%s' already exists"
                              % parser.keyword)
+        parser.value = script
         keyword_parsers[parser.keyword] = parser
         
     # Collect the test lines from the script.
     sourcepath = test.getSourcePath()
     for line_number, command_type, ln in \
             parseIntegratedTestScriptCommands(sourcepath,
                                               keyword_parsers.keys()):
         parser = keyword_parsers[command_type]
-        parser.parseLine(line_number, ln)
+        parser.parseLine(line_number, ln, parser.keyword)
+        if command_type != 'RUN:' and parser.kind == ParserKind.COMMAND:
+            has_custom_command_lines = True
         if command_type == 'END.' and parser.getValue() is True:
             break
 
     # Verify the script contains a run line.
     if require_script and not script:
         return lit.Test.Result(Test.UNRESOLVED, "Test has no run line!")
 
     # Check for unterminated run lines.
-    if script and script[-1][-1] == '\\':
+    if script and script[-1][-1][-1] == '\\':
         return lit.Test.Result(Test.UNRESOLVED,
                                "Test has unterminated run lines (with '\\')")
 
     # Enforce REQUIRES:
     missing_required_features = test.getMissingRequiredFeatures()
     if missing_required_features:
         msg = ', '.join(missing_required_features)
         return lit.Test.Result(Test.UNSUPPORTED,
def parseIntegratedTestScript(test, additional_parsers=[],
     if not test.isWithinFeatureLimits():
         msg = ', '.join(test.config.limit_to_features)
         return lit.Test.Result(Test.UNSUPPORTED,
                                "Test does not require any of the features "
                                "specified in limit_to_features: %s" % msg)
 
     return script
 
-
 def _runShTest(test, litConfig, useExternalSh, script, tmpBase):
     # Create the output directory if it does not already exist.
     lit.util.mkdir_p(os.path.dirname(tmpBase))
 
+    disableExternalSh = any(x[0] != 'RUN:' for x in script)
+    if useExternalSh and disableExternalSh:
+        litConfig.note("External shell disabled since custom command was encountered.")
+
     execdir = os.path.dirname(test.getExecPath())
-    if useExternalSh:
-        res = executeScript(test, litConfig, tmpBase, script, execdir)
+    scriptCommands = [x[1] for x in script]
+    if useExternalSh and not disableExternalSh:
+        res = executeScript(test, litConfig, tmpBase, scriptCommands, execdir)
     else:
         res = executeScriptInternal(test, litConfig, tmpBase, script, execdir)
     if isinstance(res, lit.Test.Result):
         return res
 
     out,err,exitCode,timeoutInfo = res
     if exitCode == 0:
         status = Test.PASS
     else:
         if timeoutInfo is None:
             status = Test.FAIL
         else:
             status = Test.TIMEOUT
 
     # Form the output log.
     output = """Script:\n--\n%s\n--\nExit Code: %d\n""" % (
-        '\n'.join(script), exitCode)
+        '\n'.join(scriptCommands), exitCode)
 
     if timeoutInfo is not None:
         output += """Timeout: %s\n""" % (timeoutInfo,)
     output += "\n"
 
     # Append the outputs, if present.
     if out:
         output += """Command Output (stdout):\n--\n%s\n--\n""" % (out,)
     if err:
         output += """Command Output (stderr):\n--\n%s\n--\n""" % (err,)
 
     return lit.Test.Result(status, output)
 
 
 def executeShTest(test, litConfig, useExternalSh,
                   extra_substitutions=[]):
     if test.config.unsupported:
         return lit.Test.Result(Test.UNSUPPORTED, 'Test is unsupported')
 
-    script = parseIntegratedTestScript(test)
+    additional_parsers = [IntegratedTestKeywordParser(x[0], ParserKind.COMMAND) for x in litConfig.additionalCommands]
+    script = parseIntegratedTestScript(test, additional_parsers=additional_parsers)
+
+    # If there are custom command lines we can't use an external shell because
+    # it won't understand them.
     if isinstance(script, lit.Test.Result):
         return script
     if litConfig.noExecute:
         return lit.Test.Result(Test.PASS)
 
     tmpDir, tmpBase = getTempPaths(test)
     substitutions = list(extra_substitutions)
     substitutions += getDefaultSubstitutions(test, tmpDir, tmpBase,

llvm/utils/lit/tests/Inputs/shtest-keyword-command/keyword-command.txt

+MYCOMMAND: Command1
+MYCOMMAND: Command2
+MYCOMMAND: Multi-line \
+MYCOMMAND:   Command

llvm/utils/lit/tests/Inputs/shtest-keyword-command/keyword_helper.py

+
+def customCommand(line):
+  return ('STDOUT: ' + line, 'STDERR: ' + line)

llvm/utils/lit/tests/Inputs/shtest-keyword-command/lit.cfg

+
+import os
+import site
+
+site.addsitedir(os.path.dirname(__file__))
+
+
+import lit.formats
+import keyword_helper
+
+config.name = 'shtest-keyword-command'
+config.suffixes = ['.txt']
+config.test_format = lit.formats.ShTest()
+config.test_source_root = None
+config.test_exec_root = None
+config.substitutions.append(('%{python}', '"%s"' % (sys.executable)))
+
+lit_config.installKeywordCommand('MYCOMMAND:', keyword_helper.customCommand)

llvm/utils/lit/tests/shtest-keyword-command.py

+# Check the that keyword commands work as expected.
+#
+# RUN: %{lit} -j 1 -sav %{inputs}/shtest-keyword-command > %t.out
+# RUN: FileCheck --input-file %t.out %s
+#
+# END.
+
+# CHECK: $ MYCOMMAND: at line 1  Command1
+# CHECK: command output:
+# CHECK: STDOUT:   Command1
+# CHECK: command stderr:
+# CHECK: STDERR:   Command1
+# CHECK: $ MYCOMMAND: at line 2  Command2
+# CHECK: command output:
+# CHECK: STDOUT:   Command2
+# CHECK: command stderr:
+# CHECK: STDERR:   Command2
+# CHECK: $ MYCOMMAND: at line 3  Multi-line    Command
+# CHECK: command output:
+# CHECK: STDOUT:   Multi-line    Command
+# CHECK: command stderr:
+# CHECK: STDERR:   Multi-line    Command
 No newline at end of file