[lit] Report line number for failed RUN command

[jdenny]

Summary

When debugging test failures with -vv (or -v in the case of the
internal shell), this makes it easier to locate the RUN line that
failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN
lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines
after concatenation for line continuations.

When reading the generated shell script, this also makes it easier to
locate the RUN line that produced each command.

To support reporting RUN line numbers in the case of the internal
shell, this patch extends the internal shell to support the null
command, ":", except pipelines are not supported.

To support reporting RUN line numbers in the case of windows cmd.exe
as the external shell, this patch extends -vv to set "echo on" instead
of "echo off" in bat files.

[delcypher]

Your description of needing to pass -vv doesn't look accurate, doesn't the test case just pass -v?

[jdenny]

In D44598#1041038, @delcypher wrote:

Your description of needing to pass -vv doesn't look accurate, doesn't the test case just pass -v?

Hi Dan,

Thanks for your question, and sorry for the confusion. The existing test case isn't a good example as it exercises a test parse error rather than a normal failed RUN command. When using the external shell, -vv is indeed needed. When using the internal shell, -v would be sufficient... except I have a bug in that case.

I'm about to update the diff with new tests that should hopefully demonstrate the new feature better. It will also fix this feature for the internal shell.

[jdenny]

This update makes several improvements:

1. Add new tests that actually check the feature this patch introduces.

2. Fix internal shell to have that feature (only the external shell had it before).

3. Add new test that the internal shell ":" command cannot be part of a pipeline.

[jdenny]

Sorry, one more minor fix: use END. instead of trying to escape RUN commands in CHECK lines.

[jdenny]

A few more improvements:

1. Quote entire "RUN: at line N" argument so that output from the internal shell looks better.

2. Add tests for the case of line continuations.

3. Adjust tests not to depend on the external shell's exact "set -x" output format.

4. Rebase to a more recent master.

[jdenny]

utils/lit/tests/shtest-shell.py

6

This change is unintentional. Will revert later.

[jdenny]

This update does the following:

1. Adds a line about this feature to the lit documentation.
2. Adds more motivation to the patch summary.
3. Removes the unintended change I commented on before.
4. Rebases onto a more recent master.

[jdenny]

Rebased onto a more recent master.

[jdenny]

Rebased onto a more recent master.

Added more reviewers who have recently modified lit.

[jdenny]

Brief discussion about this commit on llvm-dev:

http://lists.llvm.org/pipermail/llvm-dev/2018-April/122308.html

[asmith]

My team has some Windows 10 and Ubuntu 14 Azure VM's that build llvm and run the clang, llvm, lld, and lldb unit tests. I kicked off a test run with this patch and can report everything looks normal. It's cool to see the run line showing up on failing tests. LGTM.

2018-04-11T19:03:16.5491363Z FAIL: lldb :: Breakpoint/case-insensitive.test (335 of 343)
2018-04-11T19:03:16.5491683Z **** TEST 'lldb :: Breakpoint/case-insensitive.test' FAILED ****
2018-04-11T19:03:16.5492308Z Script:
2018-04-11T19:03:16.5492996Z --
2018-04-11T19:03:16.5493638Z : 'RUN: at line 3'; C:/agent/_work/62/b/release/bin/clang.exe C:\agent\_work\62\s\llvm\tools\lldb\lit\Breakpoint/Inputs/case-sensitive.c -g -o C:\agent\_work\62\b\tools\lldb\lit\Breakpoint\Output\case-insensitive.test.tmp
2018-04-11T19:03:16.5494360Z : 'RUN: at line 4'; C:\agent\_work\62\b\release\bin\lldb-test.EXE breakpoints C:\agent\_work\62\b\tools\lldb\lit\Breakpoint\Output\case-insensitive.test.tmp

[jdenny]

In D44598#1064835, @asmith wrote:

My team has some Windows 10 and Ubuntu 14 Azure VM's that build llvm and run the clang, llvm, lld, and lldb unit tests. I kicked off a test run with this patch and can report everything looks normal. It's cool to see the run line showing up on failing tests. LGTM.

Hi Aaron. That's great to hear. Thanks.

Are you accepting the patch, or should we wait for more eyes?

[asmith]

Sure, I will accept it :) Good luck with the buildbots!

[jdenny]

In D44598#1064874, @asmith wrote:

Sure, I will accept it :)

Thanks!

Good luck with the buildbots!

Good point. I'll wait until morning so I can deal with any trouble.

[delcypher]

utils/lit/lit/TestRunner.py

744

The null command is pretty esoteric but occasionally useful in normal shell script. What's the reason it's not supported?
I guess we don't normally need it and it's a waste of time to implement it for the internal shell?

1313

I think this would be more readable as

line = ": '{keyword} at line {line}'; {real_command}".format(
  keyword=keyword,
  line=line,
  real_command=line)

Or something similar. There's no reason for the \' if you use " quotes instead.

utils/lit/tests/unit/TestRunner.py

103

Again if you use " quotes you can avoid having to escape the ' quotes.

[jdenny]

utils/lit/lit/TestRunner.py

744

That would be my guess. Do you object to implementing it for this purpose?

1313

Sure, I'll change that and the other occurrence you pointed out.

[delcypher]

utils/lit/lit/TestRunner.py

744

I'm fine with you not implementing the null command. I was just curious. You could perhaps leave a TODO or FIXME commenting that we don't need to support it right now.

1313

Great. I also much prefer use of python's string format function as in the example I gave but I'll leave that up to you.

[jdenny]

utils/lit/lit/TestRunner.py

744

It seems we're miscommunicating. This does implement the null command. It just rejects the null command in a pipeline because I don't see a use for that right now.

1313

Yeah, that looks nice too.

[jdenny]

1. Applied suggested changes to string quoting and formatting.

2. Rebased onto a more recent master.

[jdenny]

utils/lit/lit/TestRunner.py

744

In case it helps to make this thread intelligible: I made my initial response here under the assumption that you were asking about the existing state of lit without my patch. There, the null command indeed is not implemented.

The null command implementation is needed by this patch so that the internal shell reports run lines as well.

Any objection?

[delcypher]

utils/lit/lit/TestRunner.py

744

Ah sorry. I misread the code. I see now that your are adding support for the null command to the external shell in the case that there is no pipeline. This isn't mentioned in the commit message so perhaps you should mention it?

I have no objection to the code here.

[jdenny]

utils/lit/lit/TestRunner.py

744

Good point. I've added that.

[jdenny]

Rebased onto a more recent master.

Fixed some indentation.

Improved a test case.

[jdenny]

Hi Dan. It seems as if you reviewed the whole patch, and I've made the changes you suggested. Is this ready to land?

[jdenny]

Ping.

[delcypher]

In D44598#1075801, @jdenny wrote:

Ping.

Sorry for the delay. LGTM. I've tested the patch on upstream LLVM and the tests seem to pass. Are you able to commit this?

[jdenny]

In D44598#1076891, @delcypher wrote:

In D44598#1075801, @jdenny wrote:

Ping.

Sorry for the delay. LGTM. I've tested the patch on upstream LLVM and the tests seem to pass.

No problem. Thanks!

Are you able to commit this?

Yes. Will do so shortly.

[jdenny]

This broke windows and has been reverted. Discussion is here:

https://bugs.llvm.org/show_bug.cgi?id=37239

[jdenny]

I've attempted to replace the ":" command with "echo > null" in the case of windows cmd.exe. I don't build on windows, so someone else will have to try it, or we can let the buildbots tell us if windows is happy now.

I cleaned up the patch not to pass the "RUN:" keyword (or a customized version of it) around to parsers that don't need it.

[stella.stamenova]

In D44598#1084281, @jdenny wrote:

I've attempted to replace the ":" command with "echo > null" in the case of windows cmd.exe. I don't build on windows, so someone else will have to try it, or we can let the buildbots tell us if windows is happy now.

The tests work correctly on our setup (Windows 10/Python 3).

[jdenny]

In D44598#1084413, @stella.stamenova wrote:

In D44598#1084281, @jdenny wrote:

I've attempted to replace the ":" command with "echo > null" in the case of windows cmd.exe. I don't build on windows, so someone else will have to try it, or we can let the buildbots tell us if windows is happy now.

I meant "echo > nul", but the code had it right.

The tests work correctly on our setup (Windows 10/Python 3).

Good to know! Thanks!

Before committing, I'll wait a bit for a code review as I made some small but non-trivial changes.

[jmorse]

Hi, this broadly works on the Sony setup (Windows 10, python 2, batch) but I get a failing test case for "shtest-run-at-line.py", see inline comments. It seems to just be minor output differences.

For full clarity, here's the output of running lit on shtest-run-at-line.py, followed by the output that that test is testing: https://pastebin.com/XLfw4k8D

[jmorse]

Aaannnddd, I didn't hit save in phabricator sorry, so here are those inline comments.

utils/lit/tests/shtest-run-at-line.py

22

On Windows, the output I get here is:

Script:

echo "RUN: at line 1" > nul && true
echo "RUN: at line 2" > nul && false
echo "RUN: at line 3" > nul && true

And no subsequent trailing stderr

32

Similar to above, on Windows I get

echo "RUN: at line 1" > nul && : first line continued to second line

Here, as well as no stderr.

[jdenny]

In D44598#1085092, @jmorse wrote:

Aaannnddd, I didn't hit save in phabricator sorry, so here are those inline comments.

Ah, I forgot those tests would need to be updated. Thanks! I'll work on it.

[jdenny]

In D44598#1084413, @stella.stamenova wrote:

In D44598#1084281, @jdenny wrote:

I've attempted to replace the ":" command with "echo > null" in the case of windows cmd.exe. I don't build on windows, so someone else will have to try it, or we can let the buildbots tell us if windows is happy now.

The tests work correctly on our setup (Windows 10/Python 3).

Hi Stella. Can you confirm that your setup uses bash in Windows? That would explain why you didn't see the errors Jeremy saw.

[jdenny]

Fixed test failures for windows cmd.exe, hopefully....

I'm setting up a windows build so that others don't have to keep trying out my failed attempts at this patch. In the meantime, please try it if you like.

[jdenny]

This fixes windows 10 with cmd.exe for me. If anyone else wants to try it out, that would be great, but it should be ready for a patch review.

[jmorse]

Works perfectly for me on Win10 / cmd.exe, many thanks for getting that working!

[jdenny]

In D44598#1091098, @jmorse wrote:

Works perfectly for me on Win10 / cmd.exe, many thanks for getting that working!

Great. Thanks for trying it out!

[jdenny]

Ping.

[jdenny]

Ping.

[jdenny]

Ping.

[lebedev.ri]

In D44598#1115221, @jdenny wrote:

Ping.

What seems to be the problem?
This was accepted before, reverted due to windows bot breakage,
https://reviews.llvm.org/D44598#1091098 suggests that breakage is fixed.
Just commit again?

[jdenny]

In D44598#1115229, @lebedev.ri wrote:

In D44598#1115221, @jdenny wrote:

Ping.

What seems to be the problem?
This was accepted before, reverted due to windows bot breakage,
https://reviews.llvm.org/D44598#1091098 suggests that breakage is fixed.
Just commit again?

There was enough of a code change that I thought the reviewer would want to take another look. Is that not the right approach?

[delcypher]

In D44598#1115233, @jdenny wrote:

In D44598#1115229, @lebedev.ri wrote:

In D44598#1115221, @jdenny wrote:

Ping.

What seems to be the problem?
This was accepted before, reverted due to windows bot breakage,
https://reviews.llvm.org/D44598#1091098 suggests that breakage is fixed.
Just commit again?

There was enough of a code change that I thought the reviewer would want to take another look. Is that not the right approach?

A high level question. Why are we even testing using cmd.exe with an external shell? IIRC that's not a use case LLVM uses for Windows. Instead
the internal shell is used on Windows. Is this (cmd.exe as an external shell) really something we want to support? I don't see anything desirable about supporting cmd.exe as an
external shell because that means it can be very easy to write non-portable shell tests.

[jdenny]

In D44598#1115242, @delcypher wrote:

In D44598#1115233, @jdenny wrote:

In D44598#1115229, @lebedev.ri wrote:

In D44598#1115221, @jdenny wrote:

Ping.

What seems to be the problem?
This was accepted before, reverted due to windows bot breakage,
https://reviews.llvm.org/D44598#1091098 suggests that breakage is fixed.
Just commit again?

There was enough of a code change that I thought the reviewer would want to take another look. Is that not the right approach?

A high level question. Why are we even testing using cmd.exe with an external shell? IIRC that's not a use case LLVM uses for Windows. Instead
the internal shell is used on Windows. Is this (cmd.exe as an external shell) really something we want to support? I don't see anything desirable about supporting cmd.exe as an
external shell because that means it can be very easy to write non-portable shell tests.

I'm not a windows developer. I fixed that because others reverted my patch when it broke windows buildbots. Someone else will have to answer. Here's the bugzilla, in case that helps:

https://bugs.llvm.org/show_bug.cgi?id=37239

[jdenny]

In D44598#1115251, @jdenny wrote:

In D44598#1115242, @delcypher wrote:

A high level question. Why are we even testing using cmd.exe with an external shell? IIRC that's not a use case LLVM uses for Windows. Instead
the internal shell is used on Windows. Is this (cmd.exe as an external shell) really something we want to support? I don't see anything desirable about supporting cmd.exe as an
external shell because that means it can be very easy to write non-portable shell tests.

I'm not a windows developer. I fixed that because others reverted my patch when it broke windows buildbots. Someone else will have to answer. Here's the bugzilla, in case that helps:

https://bugs.llvm.org/show_bug.cgi?id=37239

Hmm. I'm now remembering that this patch broke only lit's own tests -- both in that bugzilla and when I eventually built it in windows myself. That supports your assessment. Then again, I didn't attempt to build all the LLVM test suites in windows.

I'll post the question in that bugzilla in case they're not following here.

[jdenny]

Would there be any harm in committing this patch as is? If we decide to remove all support for windows cmd.exe, that should be a separate patch anyway.

[rnk]

In D44598#1115606, @jdenny wrote:

Would there be any harm in committing this patch as is? If we decide to remove all support for windows cmd.exe, that should be a separate patch anyway.

Nope, go for it.

Sorry, there were many updates, and I think every reviewer assumed someone else would stamp it.

docs/CommandGuide/lit.rst

  Show more information on test failures, for example the entire test output
  instead of just the test result.
 
 .. option:: -vv, --echo-all-commands
 
  Echo all commands to stdout, as they are being executed.
  This can be valuable for debugging test failures, as the last echoed command
  will be the one which has failed.
+ To help you find the source RUN line, :program:`lit` inserts a no-op ``:``
+ command with argument ``'RUN: at line N'`` before each command pipeline.
  This option implies ``--verbose``.
 
 .. option:: -a, --show-all
 
  Show more information about all tests, for example the entire test
  commandline and output.
 
 .. option:: --no-progress-bar

utils/lit/lit/TestRunner.py

def _executeShCmd(cmd, shenv, results, timeoutHelper):
     if cmd.commands[0].args[0] == 'rm':
         if len(cmd.commands) != 1:
             raise InternalShellError(cmd.commands[0], "Unsupported: 'rm' "
                                      "cannot be part of a pipeline")
         cmdResult = executeBuiltinRm(cmd.commands[0], shenv)
         results.append(cmdResult)
         return cmdResult.exitCode
 
+    if cmd.commands[0].args[0] == ':':
+        if len(cmd.commands) != 1:
+            raise InternalShellError(cmd.commands[0], "Unsupported: ':' "
+                                     "cannot be part of a pipeline")
+        results.append(ShellCommandResult(cmd.commands[0], '', '', 0, False))
+        return 0;
+
     procs = []
     default_stdin = subprocess.PIPE
     stderrTempFiles = []
     opened_files = []
     named_temp_files = []
     builtin_commands = set(['cat'])
     builtin_commands_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "builtin_commands")
     # To avoid deadlock, we use a single stderr stream for piped
def __init__(self, keyword, kind, parser=None, initial_value=None):
                 raise ValueError("ParserKind.CUSTOM requires a custom parser")
             self.parser = parser
         else:
             raise ValueError("Unknown kind '%s'" % kind)
 
     def parseLine(self, line_number, line):
         try:
             self.parsed_lines += [(line_number, line)]
-            self.value = self.parser(line_number, line, self.value)
+            self.value = self.parser(line_number, line, self.value,
+                                     self.keyword)
         except ValueError as e:
             raise ValueError(str(e) + ("\nin %s directive on test line %d" %
                                        (self.keyword, line_number)))
 
     def getValue(self):
         return self.value
 
     @staticmethod
-    def _handleTag(line_number, line, output):
+    def _handleTag(line_number, line, output, keyword):
         """A helper for parsing TAG type keywords"""
         return (not line.strip() or output)
 
     @staticmethod
-    def _handleCommand(line_number, line, output):
+    def _handleCommand(line_number, line, output, keyword):
         """A helper for parsing COMMAND type keywords"""
         # Trim trailing whitespace.
         line = line.rstrip()
         # Substitute line number expressions
         line = re.sub('%\(line\)', str(line_number), line)
 
         def replace_line_number(match):
             if match.group(1) == '+':
                 return str(line_number + int(match.group(2)))
             if match.group(1) == '-':
                 return str(line_number - int(match.group(2)))
         line = re.sub('%\(line *([\+-]) *(\d+)\)', replace_line_number, line)
         # Collapse lines with trailing '\\'.
         if output and output[-1][-1] == '\\':
             output[-1] = output[-1][:-1] + line
         else:
             if output is None:
                 output = []
+            line = ': \'' + keyword + ' at line ' + str(line_number) + \
+                   '\'; ' + line
             output.append(line)
         return output
 
     @staticmethod
-    def _handleList(line_number, line, output):
+    def _handleList(line_number, line, output, keyword):
         """A parser for LIST type keywords"""
         if output is None:
             output = []
         output.extend([s.strip() for s in line.split(',')])
         return output
 
     @staticmethod
-    def _handleBooleanExpr(line_number, line, output):
+    def _handleBooleanExpr(line_number, line, output, keyword):
         """A parser for BOOLEAN_EXPR type keywords"""
         if output is None:
             output = []
         output.extend([s.strip() for s in line.split(',')])
         # Evaluate each expression to verify syntax.
         # We don't want any results, just the raised ValueError.
         for s in output:
             if s != '*':
                 BooleanExpression.evaluate(s, [])
         return output
 
     @staticmethod
-    def _handleRequiresAny(line_number, line, output):
+    def _handleRequiresAny(line_number, line, output, keyword):
         """A custom parser to transform REQUIRES-ANY: into REQUIRES:"""
 
         # Extract the conditions specified in REQUIRES-ANY: as written.
         conditions = []
-        IntegratedTestKeywordParser._handleList(line_number, line, conditions)
+        IntegratedTestKeywordParser._handleList(line_number, line, conditions,
+                                                keyword)
 
         # Output a `REQUIRES: a || b || c` expression in its place.
         expression = ' || '.join(conditions)
-        IntegratedTestKeywordParser._handleBooleanExpr(line_number,
-                                                       expression, output)
+        IntegratedTestKeywordParser._handleBooleanExpr(line_number, expression,
+                                                       output, keyword)
         return output
 
 def parseIntegratedTestScript(test, additional_parsers=[],
                               require_script=True):
     """parseIntegratedTestScript - Scan an LLVM/Clang style integrated test
     script and extract the lines to 'RUN' as well as 'XFAIL' and 'REQUIRES'
     and 'UNSUPPORTED' information.
 

utils/lit/tests/Inputs/shtest-run-at-line/external-shell/basic.txt

+# RUN: true
+# RUN: false
+# RUN: true

utils/lit/tests/Inputs/shtest-run-at-line/external-shell/line-continuation.txt

+# RUN: : first line continued \
+# RUN:   to second line
+# RUN: echo 'foo bar' \
+# RUN: | FileCheck %s
+# RUN: echo \
+# RUN: 'foo baz' \
+# RUN: | FileCheck %s
+# RUN: echo 'foo bar' \
+# RUN: | FileCheck %s
+
+# CHECK: foo bar

utils/lit/tests/Inputs/shtest-run-at-line/external-shell/lit.local.cfg

+import lit.formats
+config.test_format = lit.formats.ShTest(execute_external=True)

utils/lit/tests/Inputs/shtest-run-at-line/internal-shell/basic.txt

+# RUN: true
+# RUN: false
+# RUN: true

utils/lit/tests/Inputs/shtest-run-at-line/internal-shell/line-continuation.txt

+# RUN: : first line continued \
+# RUN:   to second line
+# RUN: echo 'foo bar' \
+# RUN: | FileCheck %s
+# RUN: echo \
+# RUN: 'foo baz' \
+# RUN: | FileCheck %s
+# RUN: echo 'foo bar' \
+# RUN: | FileCheck %s
+
+# CHECK: foo bar

utils/lit/tests/Inputs/shtest-run-at-line/internal-shell/lit.local.cfg

+import lit.formats
+config.test_format = lit.formats.ShTest(execute_external=False)

utils/lit/tests/Inputs/shtest-run-at-line/lit.cfg

+import lit.formats
+config.name = 'shtest-run-at-line'
+config.suffixes = ['.txt']

utils/lit/tests/Inputs/shtest-shell/colon-error.txt

+# Check error on an unsupported ":". (cannot be part of a pipeline)
+#
+# RUN: : | echo "hello"

utils/lit/tests/max-failures.py

 # Check the behavior of --max-failures option.
 #
 # RUN: not %{lit} -j 1 -v %{inputs}/max-failures > %t.out
 # RUN: not %{lit} --max-failures=1 -j 1 -v %{inputs}/max-failures >> %t.out
 # RUN: not %{lit} --max-failures=2 -j 1 -v %{inputs}/max-failures >> %t.out
 # RUN: not %{lit} --max-failures=0 -j 1 -v %{inputs}/max-failures 2>> %t.out
 # RUN: FileCheck < %t.out %s
 #
 # END.
 
-# CHECK: Failing Tests (26)
+# CHECK: Failing Tests (27)
 # CHECK: Failing Tests (1)
 # CHECK: Failing Tests (2)
 # CHECK: error: Setting --max-failures to 0 does not have any effect.

utils/lit/tests/shtest-format.py

 # CHECK-NEXT: --
 # CHECK-NEXT: printf "line 1
 # CHECK-NEXT: false
 # CHECK-NEXT: --
 # CHECK-NEXT: Exit Code: 1
 #
 # CHECK: Command Output (stdout):
 # CHECK-NEXT: --
+# CHECK-NEXT: $ ":" "RUN: at line 1"
 # CHECK-NEXT: $ "printf"
 # CHECK-NEXT: # command output:
 # CHECK-NEXT: line 1: failed test output on stdout
 # CHECK-NEXT: line 2: failed test output on stdout
 
 # CHECK: UNRESOLVED: shtest-format :: no-test-line.txt
 # CHECK: PASS: shtest-format :: pass.txt
 # CHECK: UNSUPPORTED: shtest-format :: requires-any-missing.txt

utils/lit/tests/shtest-output-printing.py

 # CHECK-NEXT: *** TEST 'shtest-output-printing :: basic.txt' FAILED ***
 # CHECK-NEXT: Script:
 # CHECK-NEXT: --
 # CHECK:      --
 # CHECK-NEXT: Exit Code: 1
 #
 # CHECK:      Command Output
 # CHECK-NEXT: --
+# CHECK-NEXT: $ ":" "RUN: at line 1"
 # CHECK-NEXT: $ "true"
+# CHECK-NEXT: $ ":" "RUN: at line 2"
 # CHECK-NEXT: $ "echo" "hi"
 # CHECK-NEXT: # command output:
 # CHECK-NEXT: hi
 #
-# CHECK:      $ "wc" "missing-file"
+# CHECK:      $ ":" "RUN: at line 3"
+# CHECK-NEXT: $ "wc" "missing-file"
 # CHECK-NEXT: # redirected output from '{{.*(/|\\\\)}}basic.txt.tmp.out':
 # CHECK-NEXT: missing-file{{.*}} No such file or directory
 # CHECK:      note: command had no output on stdout or stderr
 # CHECK-NEXT: error: command failed with exit status: 1

utils/lit/tests/shtest-run-at-line.py

+# Check that -vv makes the line number of the failing RUN command clear.
+# (-v is actually sufficient in the case of the internal shell.)
+#
+# RUN: not %{lit} -j 1 -vv %{inputs}/shtest-run-at-line > %t.out
+# RUN: FileCheck --input-file %t.out %s
+#
+# END.
+
+
+# In the case of the external shell, we check for only RUN lines in stderr in
+# case some shell implementations format "set -x" output differently.
+
+# CHECK-LABEL: FAIL: shtest-run-at-line :: external-shell/basic.txt
+
+# CHECK:      Script:
+# CHECK:      : 'RUN: at line 1';  true
+# CHECK-NEXT: : 'RUN: at line 2';  false
+# CHECK-NEXT: : 'RUN: at line 3';  true
+
+# CHECK:     Command Output (stderr)
+# CHECK:     RUN: at line 1
+# CHECK:     RUN: at line 2
+# CHECK-NOT: RUN
+
+# CHECK-LABEL: FAIL: shtest-run-at-line :: external-shell/line-continuation.txt
+
+# CHECK:      Script:
+# CHECK:      : 'RUN: at line 1';  : first line continued    to second line
+# CHECK-NEXT: : 'RUN: at line 3';  echo 'foo bar'  | FileCheck
+# CHECK-NEXT: : 'RUN: at line 5';  echo  'foo baz'  | FileCheck
+# CHECK-NEXT: : 'RUN: at line 8';  echo 'foo bar'  | FileCheck
+
+# CHECK:     Command Output (stderr)
+# CHECK:     RUN: at line 1
+# CHECK:     RUN: at line 3
+# CHECK:     RUN: at line 5
+# CHECK-NOT: RUN
+
+
+# CHECK-LABEL: FAIL: shtest-run-at-line :: internal-shell/basic.txt
+
+# CHECK:      Script:
+# CHECK:      : 'RUN: at line 1';  true
+# CHECK-NEXT: : 'RUN: at line 2';  false
+# CHECK-NEXT: : 'RUN: at line 3';  true
+
+# CHECK:      Command Output (stdout)
+# CHECK:      $ ":" "RUN: at line 1"
+# CHECK-NEXT: $ "true"
+# CHECK-NEXT: $ ":" "RUN: at line 2"
+# CHECK-NEXT: $ "false"
+# CHECK-NOT:  RUN
+
+# CHECK-LABEL: FAIL: shtest-run-at-line :: internal-shell/line-continuation.txt
+
+# CHECK:      Script:
+# CHECK:      : 'RUN: at line 1';  : first line continued to second line
+# CHECK-NEXT: : 'RUN: at line 3';  echo 'foo bar'  | FileCheck
+# CHECK-NEXT: : 'RUN: at line 5';  echo  'foo baz'  | FileCheck
+# CHECK-NEXT: : 'RUN: at line 8';  echo 'foo bar'  | FileCheck
+
+# CHECK:      Command Output (stdout)
+# CHECK:      $ ":" "RUN: at line 1"
+# CHECK-NEXT: $ ":" "first" "line" "continued" "to" "second" "line"
+# CHECK-NEXT: $ ":" "RUN: at line 3"
+# CHECK-NEXT: $ "echo" "foo bar"
+# CHECK-NEXT: $ "FileCheck" "{{.*}}"
+# CHECK-NEXT: $ ":" "RUN: at line 5"
+# CHECK-NEXT: $ "echo" "foo baz"
+# CHECK-NEXT: $ "FileCheck" "{{.*}}"
+# CHECK-NOT:  RUN

utils/lit/tests/shtest-shell.py

 # Check the internal shell handling component of the ShTest format.
 #
 # RUN: not %{lit} -j 1 -v %{inputs}/shtest-shell > %t.out
 # FIXME: Temporarily dump test output so we can debug failing tests on
 # buildbots.
 # RUN: cat %t.out
 # RUN: FileCheck --input-file %t.out %s
 #
 # END.
 
 # CHECK: -- Testing:
 
 # CHECK: FAIL: shtest-shell :: cat-error-0.txt
 # CHECK: *** TEST 'shtest-shell :: cat-error-0.txt' FAILED ***
 # CHECK: $ "cat" "-b" "temp1.txt"
 # CHECK: # command stderr:
 # CHECK: Unsupported: 'cat':  option -b not recognized
 # CHECK: error: command failed with exit status: 1
 # CHECK: ***
 
 # CHECK: FAIL: shtest-shell :: cat-error-1.txt
 # CHECK: *** TEST 'shtest-shell :: cat-error-1.txt' FAILED ***
 # CHECK: $ "cat" "temp1.txt"
 # CHECK: # command stderr:
 # CHECK: [Errno 2] No such file or directory: 'temp1.txt'
 # CHECK: error: command failed with exit status: 1
 # CHECK: ***
 
+# CHECK: FAIL: shtest-shell :: colon-error.txt
+# CHECK: *** TEST 'shtest-shell :: colon-error.txt' FAILED ***
+# CHECK: $ ":"
+# CHECK: # command stderr:
+# CHECK: Unsupported: ':' cannot be part of a pipeline
+# CHECK: error: command failed with exit status: 127
+# CHECK: ***
+
 # CHECK: FAIL: shtest-shell :: diff-error-0.txt
 # CHECK: *** TEST 'shtest-shell :: diff-error-0.txt' FAILED ***
 # CHECK: $ "diff" "diff-error-0.txt" "diff-error-0.txt"
 # CHECK: # command stderr:
 # CHECK: Unsupported: 'diff' cannot be part of a pipeline
 # CHECK: error: command failed with exit status: 127
 # CHECK: ***
 

 # CHECK: 'not-a-real-command': command not found
 # CHECK: error: command failed with exit status: 127
 # CHECK: ***
 
 # FIXME: The output here sucks.
 #
 # CHECK: FAIL: shtest-shell :: error-1.txt
 # CHECK: *** TEST 'shtest-shell :: error-1.txt' FAILED ***
-# CHECK: shell parser error on: 'echo "missing quote'
+# CHECK: shell parser error on: ': \'RUN: at line 3\'; echo "missing quote'
 # CHECK: ***
 
 # CHECK: FAIL: shtest-shell :: error-2.txt
 # CHECK: *** TEST 'shtest-shell :: error-2.txt' FAILED ***
 # CHECK: Unsupported redirect:
 # CHECK: ***
 
 # CHECK: FAIL: shtest-shell :: mkdir-error-0.txt

 # CHECK: FAIL: shtest-shell :: rm-error-3.txt
 # CHECK: *** TEST 'shtest-shell :: rm-error-3.txt' FAILED ***
 # CHECK: Exit Code: 1
 # CHECK: ***
 
 # CHECK: PASS: shtest-shell :: sequencing-0.txt
 # CHECK: XFAIL: shtest-shell :: sequencing-1.txt
 # CHECK: PASS: shtest-shell :: valid-shell.txt
-# CHECK: Failing Tests (26)
+# CHECK: Failing Tests (27)

utils/lit/tests/unit/TestRunner.py

def load_keyword_parser_lit_tests():
         assert os.path.isdir(inputs[0])
         run = lit.run.Run(lit_config,
                           lit.discovery.find_tests_for_inputs(lit_config, inputs))
         assert len(run.tests) == 1 and "there should only be one test"
         TestIntegratedTestKeywordParser.inputTestCase = run.tests[0]
 
     @staticmethod
     def make_parsers():
-        def custom_parse(line_number, line, output):
+        def custom_parse(line_number, line, output, keyword):
             if output is None:
                 output = []
             output += [part for part in line.split(' ') if part.strip()]
             return output
 
         return [
             IntegratedTestKeywordParser("MY_TAG.", ParserKind.TAG),
             IntegratedTestKeywordParser("MY_DNE_TAG.", ParserKind.TAG),
def test_lists(self):
                               ['one', 'two', 'three', 'four'])
 
     def test_commands(self):
         parsers = self.make_parsers()
         self.parse_test(parsers)
         cmd_parser = self.get_parser(parsers, 'MY_RUN:')
         value = cmd_parser.getValue()
         self.assertEqual(len(value), 2)  # there are only two run lines
-        self.assertEqual(value[0].strip(), 'baz')
-        self.assertEqual(value[1].strip(), 'foo  bar')
+        self.assertEqual(value[0].strip(), ': \'MY_RUN: at line 4\';  baz')
+        self.assertEqual(value[1].strip(), ': \'MY_RUN: at line 7\';  foo  bar')
 
     def test_custom(self):
         parsers = self.make_parsers()
         self.parse_test(parsers)
         custom_parser = self.get_parser(parsers, 'MY_CUSTOM:')
         value = custom_parser.getValue()
         self.assertEqual(value, ['a', 'b', 'c'])