[lit] Report line number for failed RUN command

[jdenny]

Summary

When debugging test failures with -vv (or -v in the case of the
internal shell), this makes it easier to locate the RUN line that
failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN
lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines
after concatenation for line continuations.

When reading the generated shell script, this also makes it easier to
locate the RUN line that produced each command.

To support reporting RUN line numbers in the case of the internal
shell, this patch extends the internal shell to support the null
command, ":", except pipelines are not supported.

To support reporting RUN line numbers in the case of windows cmd.exe
as the external shell, this patch extends -vv to set "echo on" instead
of "echo off" in bat files.

[delcypher]

Your description of needing to pass -vv doesn't look accurate, doesn't the test case just pass -v?

[jdenny]

In D44598#1041038, @delcypher wrote:

Your description of needing to pass -vv doesn't look accurate, doesn't the test case just pass -v?

Hi Dan,

Thanks for your question, and sorry for the confusion. The existing test case isn't a good example as it exercises a test parse error rather than a normal failed RUN command. When using the external shell, -vv is indeed needed. When using the internal shell, -v would be sufficient... except I have a bug in that case.

I'm about to update the diff with new tests that should hopefully demonstrate the new feature better. It will also fix this feature for the internal shell.

[jdenny]

This update makes several improvements:

1. Add new tests that actually check the feature this patch introduces.

2. Fix internal shell to have that feature (only the external shell had it before).

3. Add new test that the internal shell ":" command cannot be part of a pipeline.

[jdenny]

Sorry, one more minor fix: use END. instead of trying to escape RUN commands in CHECK lines.

[jdenny]

A few more improvements:

1. Quote entire "RUN: at line N" argument so that output from the internal shell looks better.

2. Add tests for the case of line continuations.

3. Adjust tests not to depend on the external shell's exact "set -x" output format.

4. Rebase to a more recent master.

[jdenny]

utils/lit/tests/shtest-shell.py

6 ↗

(On Diff #139011)

This change is unintentional. Will revert later.

[jdenny]

This update does the following:

1. Adds a line about this feature to the lit documentation.
2. Adds more motivation to the patch summary.
3. Removes the unintended change I commented on before.
4. Rebases onto a more recent master.

[jdenny]

Rebased onto a more recent master.

[jdenny]

Rebased onto a more recent master.

Added more reviewers who have recently modified lit.

[jdenny]

Brief discussion about this commit on llvm-dev:

http://lists.llvm.org/pipermail/llvm-dev/2018-April/122308.html

[asmith]

My team has some Windows 10 and Ubuntu 14 Azure VM's that build llvm and run the clang, llvm, lld, and lldb unit tests. I kicked off a test run with this patch and can report everything looks normal. It's cool to see the run line showing up on failing tests. LGTM.

2018-04-11T19:03:16.5491363Z FAIL: lldb :: Breakpoint/case-insensitive.test (335 of 343)
2018-04-11T19:03:16.5491683Z **** TEST 'lldb :: Breakpoint/case-insensitive.test' FAILED ****
2018-04-11T19:03:16.5492308Z Script:
2018-04-11T19:03:16.5492996Z --
2018-04-11T19:03:16.5493638Z : 'RUN: at line 3'; C:/agent/_work/62/b/release/bin/clang.exe C:\agent\_work\62\s\llvm\tools\lldb\lit\Breakpoint/Inputs/case-sensitive.c -g -o C:\agent\_work\62\b\tools\lldb\lit\Breakpoint\Output\case-insensitive.test.tmp
2018-04-11T19:03:16.5494360Z : 'RUN: at line 4'; C:\agent\_work\62\b\release\bin\lldb-test.EXE breakpoints C:\agent\_work\62\b\tools\lldb\lit\Breakpoint\Output\case-insensitive.test.tmp

[jdenny]

In D44598#1064835, @asmith wrote:

My team has some Windows 10 and Ubuntu 14 Azure VM's that build llvm and run the clang, llvm, lld, and lldb unit tests. I kicked off a test run with this patch and can report everything looks normal. It's cool to see the run line showing up on failing tests. LGTM.

Hi Aaron. That's great to hear. Thanks.

Are you accepting the patch, or should we wait for more eyes?

[asmith]

Sure, I will accept it :) Good luck with the buildbots!

[jdenny]

In D44598#1064874, @asmith wrote:

Sure, I will accept it :)

Thanks!

Good luck with the buildbots!

Good point. I'll wait until morning so I can deal with any trouble.

[delcypher]

utils/lit/lit/TestRunner.py

792 ↗

(On Diff #141681)

The null command is pretty esoteric but occasionally useful in normal shell script. What's the reason it's not supported?
I guess we don't normally need it and it's a waste of time to implement it for the internal shell?

1367 ↗

(On Diff #141681)

I think this would be more readable as

line = ": '{keyword} at line {line}'; {real_command}".format(
  keyword=keyword,
  line=line,
  real_command=line)

Or something similar. There's no reason for the \' if you use " quotes instead.

utils/lit/tests/unit/TestRunner.py

103 ↗

(On Diff #141681)

Again if you use " quotes you can avoid having to escape the ' quotes.

[jdenny]

utils/lit/lit/TestRunner.py

792 ↗

(On Diff #141681)

That would be my guess. Do you object to implementing it for this purpose?

1367 ↗

(On Diff #141681)

Sure, I'll change that and the other occurrence you pointed out.

[delcypher]

utils/lit/lit/TestRunner.py

792 ↗

(On Diff #141681)

I'm fine with you not implementing the null command. I was just curious. You could perhaps leave a TODO or FIXME commenting that we don't need to support it right now.

1367 ↗

(On Diff #141681)

Great. I also much prefer use of python's string format function as in the example I gave but I'll leave that up to you.

[jdenny]

utils/lit/lit/TestRunner.py

792 ↗

(On Diff #141681)

It seems we're miscommunicating. This does implement the null command. It just rejects the null command in a pipeline because I don't see a use for that right now.

1367 ↗

(On Diff #141681)

Yeah, that looks nice too.

[jdenny]

1. Applied suggested changes to string quoting and formatting.

2. Rebased onto a more recent master.

[jdenny]

utils/lit/lit/TestRunner.py

792 ↗

(On Diff #141681)

In case it helps to make this thread intelligible: I made my initial response here under the assumption that you were asking about the existing state of lit without my patch. There, the null command indeed is not implemented.

The null command implementation is needed by this patch so that the internal shell reports run lines as well.

Any objection?

[delcypher]

utils/lit/lit/TestRunner.py

792 ↗

(On Diff #141681)

Ah sorry. I misread the code. I see now that your are adding support for the null command to the external shell in the case that there is no pipeline. This isn't mentioned in the commit message so perhaps you should mention it?

I have no objection to the code here.

[jdenny]

utils/lit/lit/TestRunner.py

792 ↗

(On Diff #141681)

Good point. I've added that.

[jdenny]

Rebased onto a more recent master.

Fixed some indentation.

Improved a test case.

[jdenny]

Hi Dan. It seems as if you reviewed the whole patch, and I've made the changes you suggested. Is this ready to land?

[jdenny]

Ping.

[delcypher]

In D44598#1075801, @jdenny wrote:

Ping.

Sorry for the delay. LGTM. I've tested the patch on upstream LLVM and the tests seem to pass. Are you able to commit this?

[jdenny]

In D44598#1076891, @delcypher wrote:

In D44598#1075801, @jdenny wrote:

Ping.

Sorry for the delay. LGTM. I've tested the patch on upstream LLVM and the tests seem to pass.

No problem. Thanks!

Are you able to commit this?

Yes. Will do so shortly.

[jdenny]

This broke windows and has been reverted. Discussion is here:

https://bugs.llvm.org/show_bug.cgi?id=37239

[jdenny]

I've attempted to replace the ":" command with "echo > null" in the case of windows cmd.exe. I don't build on windows, so someone else will have to try it, or we can let the buildbots tell us if windows is happy now.

I cleaned up the patch not to pass the "RUN:" keyword (or a customized version of it) around to parsers that don't need it.

[stella.stamenova]

In D44598#1084281, @jdenny wrote:

I've attempted to replace the ":" command with "echo > null" in the case of windows cmd.exe. I don't build on windows, so someone else will have to try it, or we can let the buildbots tell us if windows is happy now.

The tests work correctly on our setup (Windows 10/Python 3).

[jdenny]

In D44598#1084413, @stella.stamenova wrote:

In D44598#1084281, @jdenny wrote:

I've attempted to replace the ":" command with "echo > null" in the case of windows cmd.exe. I don't build on windows, so someone else will have to try it, or we can let the buildbots tell us if windows is happy now.

I meant "echo > nul", but the code had it right.

The tests work correctly on our setup (Windows 10/Python 3).

Good to know! Thanks!

Before committing, I'll wait a bit for a code review as I made some small but non-trivial changes.

[jmorse]

Hi, this broadly works on the Sony setup (Windows 10, python 2, batch) but I get a failing test case for "shtest-run-at-line.py", see inline comments. It seems to just be minor output differences.

For full clarity, here's the output of running lit on shtest-run-at-line.py, followed by the output that that test is testing: https://pastebin.com/XLfw4k8D

[jmorse]

Aaannnddd, I didn't hit save in phabricator sorry, so here are those inline comments.

utils/lit/tests/shtest-run-at-line.py

21 ↗

(On Diff #144748)

On Windows, the output I get here is:

Script:

echo "RUN: at line 1" > nul && true
echo "RUN: at line 2" > nul && false
echo "RUN: at line 3" > nul && true

And no subsequent trailing stderr

31 ↗

(On Diff #144748)

Similar to above, on Windows I get

echo "RUN: at line 1" > nul && : first line continued to second line

Here, as well as no stderr.

[jdenny]

In D44598#1085092, @jmorse wrote:

Aaannnddd, I didn't hit save in phabricator sorry, so here are those inline comments.

Ah, I forgot those tests would need to be updated. Thanks! I'll work on it.

[jdenny]

In D44598#1084413, @stella.stamenova wrote:

In D44598#1084281, @jdenny wrote:

I've attempted to replace the ":" command with "echo > null" in the case of windows cmd.exe. I don't build on windows, so someone else will have to try it, or we can let the buildbots tell us if windows is happy now.

The tests work correctly on our setup (Windows 10/Python 3).

Hi Stella. Can you confirm that your setup uses bash in Windows? That would explain why you didn't see the errors Jeremy saw.

[jdenny]

Fixed test failures for windows cmd.exe, hopefully....

I'm setting up a windows build so that others don't have to keep trying out my failed attempts at this patch. In the meantime, please try it if you like.

[jdenny]

This fixes windows 10 with cmd.exe for me. If anyone else wants to try it out, that would be great, but it should be ready for a patch review.

[jmorse]

Works perfectly for me on Win10 / cmd.exe, many thanks for getting that working!

[jdenny]

In D44598#1091098, @jmorse wrote:

Works perfectly for me on Win10 / cmd.exe, many thanks for getting that working!

Great. Thanks for trying it out!

[jdenny]

Ping.

[jdenny]

Ping.

[jdenny]

Ping.

[lebedev.ri]

In D44598#1115221, @jdenny wrote:

Ping.

What seems to be the problem?
This was accepted before, reverted due to windows bot breakage,
https://reviews.llvm.org/D44598#1091098 suggests that breakage is fixed.
Just commit again?

[jdenny]

In D44598#1115229, @lebedev.ri wrote:

In D44598#1115221, @jdenny wrote:

Ping.

What seems to be the problem?
This was accepted before, reverted due to windows bot breakage,
https://reviews.llvm.org/D44598#1091098 suggests that breakage is fixed.
Just commit again?

There was enough of a code change that I thought the reviewer would want to take another look. Is that not the right approach?

[delcypher]

In D44598#1115233, @jdenny wrote:

In D44598#1115229, @lebedev.ri wrote:

In D44598#1115221, @jdenny wrote:

Ping.

What seems to be the problem?
This was accepted before, reverted due to windows bot breakage,
https://reviews.llvm.org/D44598#1091098 suggests that breakage is fixed.
Just commit again?

There was enough of a code change that I thought the reviewer would want to take another look. Is that not the right approach?

A high level question. Why are we even testing using cmd.exe with an external shell? IIRC that's not a use case LLVM uses for Windows. Instead
the internal shell is used on Windows. Is this (cmd.exe as an external shell) really something we want to support? I don't see anything desirable about supporting cmd.exe as an
external shell because that means it can be very easy to write non-portable shell tests.

[jdenny]

In D44598#1115242, @delcypher wrote:

In D44598#1115233, @jdenny wrote:

In D44598#1115229, @lebedev.ri wrote:

In D44598#1115221, @jdenny wrote:

Ping.

What seems to be the problem?
This was accepted before, reverted due to windows bot breakage,
https://reviews.llvm.org/D44598#1091098 suggests that breakage is fixed.
Just commit again?

There was enough of a code change that I thought the reviewer would want to take another look. Is that not the right approach?

A high level question. Why are we even testing using cmd.exe with an external shell? IIRC that's not a use case LLVM uses for Windows. Instead
the internal shell is used on Windows. Is this (cmd.exe as an external shell) really something we want to support? I don't see anything desirable about supporting cmd.exe as an
external shell because that means it can be very easy to write non-portable shell tests.

I'm not a windows developer. I fixed that because others reverted my patch when it broke windows buildbots. Someone else will have to answer. Here's the bugzilla, in case that helps:

https://bugs.llvm.org/show_bug.cgi?id=37239

[jdenny]

In D44598#1115251, @jdenny wrote:

In D44598#1115242, @delcypher wrote:

A high level question. Why are we even testing using cmd.exe with an external shell? IIRC that's not a use case LLVM uses for Windows. Instead
the internal shell is used on Windows. Is this (cmd.exe as an external shell) really something we want to support? I don't see anything desirable about supporting cmd.exe as an
external shell because that means it can be very easy to write non-portable shell tests.

I'm not a windows developer. I fixed that because others reverted my patch when it broke windows buildbots. Someone else will have to answer. Here's the bugzilla, in case that helps:

https://bugs.llvm.org/show_bug.cgi?id=37239

Hmm. I'm now remembering that this patch broke only lit's own tests -- both in that bugzilla and when I eventually built it in windows myself. That supports your assessment. Then again, I didn't attempt to build all the LLVM test suites in windows.

I'll post the question in that bugzilla in case they're not following here.

[jdenny]

Would there be any harm in committing this patch as is? If we decide to remove all support for windows cmd.exe, that should be a separate patch anyway.

[rnk]

In D44598#1115606, @jdenny wrote:

Would there be any harm in committing this patch as is? If we decide to remove all support for windows cmd.exe, that should be a separate patch anyway.

Nope, go for it.

Sorry, there were many updates, and I think every reviewer assumed someone else would stamp it.

llvm/trunk/docs/CommandGuide/lit.rst

  Show more information on test failures, for example the entire test output
  instead of just the test result.
 
 .. option:: -vv, --echo-all-commands
 
  Echo all commands to stdout, as they are being executed.
  This can be valuable for debugging test failures, as the last echoed command
  will be the one which has failed.
+ :program:`lit` normally inserts a no-op command (``:`` in the case of bash)
+ with argument ``'RUN: at line N'`` before each command pipeline, and this
+ option also causes those no-op commands to be echoed to stdout to help you
+ locate the source line of the failed command.
  This option implies ``--verbose``.
 
 .. option:: -a, --show-all
 
  Show more information about all tests, for example the entire test
  commandline and output.
 
 .. option:: --no-progress-bar

llvm/trunk/utils/lit/lit/TestRunner.py

 
 # Don't use close_fds on Windows.
 kUseCloseFDs = not kIsWindows
 
 # Use temporary files to replace /dev/null on Windows.
 kAvoidDevNull = kIsWindows
 kDevNull = "/dev/null"
 
+# A regex that matches %dbg(ARG), which lit inserts at the beginning of each
+# run command pipeline such that ARG specifies the pipeline's source line
+# number.  lit later expands each %dbg(ARG) to a command that behaves as a null
+# command in the target shell so that the line number is seen in lit's verbose
+# mode.
+#
+# This regex captures ARG.  ARG must not contain a right parenthesis, which
+# terminates %dbg.  ARG must not contain quotes, in which ARG might be enclosed
+# during expansion.
+kPdbgRegex = '%dbg\(([^)\'"]*)\)'
+
 class ShellEnvironment(object):
 
     """Mutable shell environment containing things like CWD and env vars.
 
     Environment variables are not implemented, but cwd tracking is.
     """
 
     def __init__(self, cwd, env):
def _executeShCmd(cmd, shenv, results, timeoutHelper):
     if cmd.commands[0].args[0] == 'rm':
         if len(cmd.commands) != 1:
             raise InternalShellError(cmd.commands[0], "Unsupported: 'rm' "
                                      "cannot be part of a pipeline")
         cmdResult = executeBuiltinRm(cmd.commands[0], shenv)
         results.append(cmdResult)
         return cmdResult.exitCode
 
+    if cmd.commands[0].args[0] == ':':
+        if len(cmd.commands) != 1:
+            raise InternalShellError(cmd.commands[0], "Unsupported: ':' "
+                                     "cannot be part of a pipeline")
+        results.append(ShellCommandResult(cmd.commands[0], '', '', 0, False))
+        return 0;
+
     procs = []
     default_stdin = subprocess.PIPE
     stderrTempFiles = []
     opened_files = []
     named_temp_files = []
     builtin_commands = set(['cat'])
     builtin_commands_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "builtin_commands")
     # To avoid deadlock, we use a single stderr stream for piped
for f in named_temp_files:
             pass
 
     if cmd.negate:
         exitCode = not exitCode
 
     return exitCode
 
 def executeScriptInternal(test, litConfig, tmpBase, commands, cwd):
+    commands = applySubstitutions(commands, [(kPdbgRegex, ": '\\1'")])
     cmds = []
     for ln in commands:
         try:
             cmds.append(ShUtil.ShParser(ln, litConfig.isWindows,
                                         test.config.pipefail).parse())
         except:
             return lit.Test.Result(Test.FAIL, "shell parser error on: %r" % ln)
 
for i,result in enumerate(results):
             else:
                 codeStr = str(result.exitCode)
             out += "error: command failed with exit status: %s\n" % (
                 codeStr,)
         if litConfig.maxIndividualTestTime > 0:
             out += 'error: command reached timeout: %s\n' % (
                 str(result.timeoutReached),)
 
-    return out, err, exitCode, timeoutInfo
+    return out, err, exitCode, timeoutInfo, commands
 
 def executeScript(test, litConfig, tmpBase, commands, cwd):
     bashPath = litConfig.getBashPath()
     isWin32CMDEXE = (litConfig.isWindows and not bashPath)
     script = tmpBase + '.script'
     if isWin32CMDEXE:
         script += '.bat'
 
     # Write script file
     mode = 'w'
     if litConfig.isWindows and not isWin32CMDEXE:
       mode += 'b'  # Avoid CRLFs when writing bash scripts.
     f = open(script, mode)
     if isWin32CMDEXE:
+        commands = applySubstitutions(commands, [(kPdbgRegex,
+                                                  "echo '\\1' > nul")])
+        if litConfig.echo_all_commands:
+            f.write('@echo on\n')
+        else:
             f.write('@echo off\n')
-        f.write('\nif %ERRORLEVEL% NEQ 0 EXIT\n'.join(commands))
+        f.write('\n@if %ERRORLEVEL% NEQ 0 EXIT\n'.join(commands))
     else:
+        commands = applySubstitutions(commands, [(kPdbgRegex, ": '\\1'")])
         if test.config.pipefail:
             f.write('set -o pipefail;')
         if litConfig.echo_all_commands:
             f.write('set -x;')
         f.write('{ ' + '; } &&\n{ '.join(commands) + '; }')
     f.write('\n')
     f.close()
 
     if isWin32CMDEXE:
         command = ['cmd','/c', script]
     else:
         if bashPath:
             command = [bashPath, script]
         else:
             command = ['/bin/sh', script]
         if litConfig.useValgrind:
             # FIXME: Running valgrind on sh is overkill. We probably could just
             # run on clang with no real loss.
             command = litConfig.valgrindArgs + command
 
     try:
         out, err, exitCode = lit.util.executeCommand(command, cwd=cwd,
                                        env=test.config.environment,
                                        timeout=litConfig.maxIndividualTestTime)
-        return (out, err, exitCode, None)
+        return (out, err, exitCode, None, commands)
     except lit.util.ExecuteCommandTimeoutException as e:
-        return (e.out, e.err, e.exitCode, e.msg)
+        return (e.out, e.err, e.exitCode, e.msg, commands)
 
 def parseIntegratedTestScriptCommands(source_path, keywords):
     """
     parseIntegratedTestScriptCommands(source_path) -> commands
 
     Parse the commands in an integrated test script file into a list of
     (line_number, command_type, line).
     """
substitutions.extend([
             ('%:s', colonNormalizePath(sourcepath)),
             ('%:S', colonNormalizePath(sourcedir)),
             ('%:p', colonNormalizePath(sourcedir)),
             ('%:t', colonNormalizePath(tmpBase + '.tmp')),
             ('%:T', colonNormalizePath(tmpDir)),
             ])
     return substitutions
 
-def applySubstitutions(script, substitutions):
+def applySubstitutions(script, substitutions, escapeForWindows=False):
     """Apply substitutions to the script.  Allow full regular expression syntax.
     Replace each matching occurrence of regular expression pattern a with
     substitution b in line ln."""
     def processLine(ln):
         # Apply substitutions
         for a,b in substitutions:
-            if kIsWindows:
+            if kIsWindows and escapeForWindows:
                 b = b.replace("\\","\\\\")
             ln = re.sub(a, b, ln)
 
         # Strip the trailing newline and any extra whitespace.
         return ln.strip()
     # Note Python 3 map() gives an iterator rather than a list so explicitly
     # convert to list before returning.
     return list(map(processLine, script))
def __init__(self, keyword, kind, parser=None, initial_value=None):
                              "ParserKind.CUSTOM")
         self.keyword = keyword
         self.kind = kind
         self.parsed_lines = []
         self.value = initial_value
         self.parser = parser
 
         if kind == ParserKind.COMMAND:
-            self.parser = self._handleCommand
+            self.parser = lambda line_number, line, output: \
+                                 self._handleCommand(line_number, line, output,
+                                                     self.keyword)
         elif kind == ParserKind.LIST:
             self.parser = self._handleList
         elif kind == ParserKind.BOOLEAN_EXPR:
             self.parser = self._handleBooleanExpr
         elif kind == ParserKind.TAG:
             self.parser = self._handleTag
         elif kind == ParserKind.CUSTOM:
             if parser is None:
def getValue(self):
         return self.value
 
     @staticmethod
     def _handleTag(line_number, line, output):
         """A helper for parsing TAG type keywords"""
         return (not line.strip() or output)
 
     @staticmethod
-    def _handleCommand(line_number, line, output):
+    def _handleCommand(line_number, line, output, keyword):
         """A helper for parsing COMMAND type keywords"""
         # Trim trailing whitespace.
         line = line.rstrip()
         # Substitute line number expressions
         line = re.sub('%\(line\)', str(line_number), line)
 
         def replace_line_number(match):
             if match.group(1) == '+':
                 return str(line_number + int(match.group(2)))
             if match.group(1) == '-':
                 return str(line_number - int(match.group(2)))
         line = re.sub('%\(line *([\+-]) *(\d+)\)', replace_line_number, line)
         # Collapse lines with trailing '\\'.
         if output and output[-1][-1] == '\\':
             output[-1] = output[-1][:-1] + line
         else:
             if output is None:
                 output = []
+            pdbg = "%dbg({keyword} at line {line_number})".format(
+                keyword=keyword,
+                line_number=line_number)
+            assert re.match(kPdbgRegex + "$", pdbg), \
+                   "kPdbgRegex expected to match actual %dbg usage"
+            line = "{pdbg} && {real_command}".format(
+                pdbg=pdbg,
+                real_command=line)
             output.append(line)
         return output
 
     @staticmethod
     def _handleList(line_number, line, output):
         """A parser for LIST type keywords"""
         if output is None:
             output = []
def _runShTest(test, litConfig, useExternalSh, script, tmpBase):
     execdir = os.path.dirname(test.getExecPath())
     if useExternalSh:
         res = executeScript(test, litConfig, tmpBase, script, execdir)
     else:
         res = executeScriptInternal(test, litConfig, tmpBase, script, execdir)
     if isinstance(res, lit.Test.Result):
         return res
 
-    out,err,exitCode,timeoutInfo = res
+    out,err,exitCode,timeoutInfo,script = res
     if exitCode == 0:
         status = Test.PASS
     else:
         if timeoutInfo is None:
             status = Test.FAIL
         else:
             status = Test.TIMEOUT
 
if isinstance(script, lit.Test.Result):
         return script
     if litConfig.noExecute:
         return lit.Test.Result(Test.PASS)
 
     tmpDir, tmpBase = getTempPaths(test)
     substitutions = list(extra_substitutions)
     substitutions += getDefaultSubstitutions(test, tmpDir, tmpBase,
                                              normalize_slashes=useExternalSh)
-    script = applySubstitutions(script, substitutions)
+    script = applySubstitutions(script, substitutions, True)
 
     # Re-run failed tests up to test_retry_attempts times.
     attempts = 1
     if hasattr(test.config, 'test_retry_attempts'):
         attempts += test.config.test_retry_attempts
     for i in range(attempts):
         res = _runShTest(test, litConfig, useExternalSh, script, tmpBase)
         if res.code != Test.FAIL:
             break
     # If we had to run the test more than once, count it as a flaky pass. These
     # will be printed separately in the test summary.
     if i > 0 and res.code == Test.PASS:
         res.code = Test.FLAKYPASS
     return res

llvm/trunk/utils/lit/tests/Inputs/shtest-run-at-line/external-shell/basic.txt

+# These commands must run under both bash and windows cmd.exe (with GnuWin32
+# tools).
+
+# RUN: true
+# RUN: false
+# RUN: true

llvm/trunk/utils/lit/tests/Inputs/shtest-run-at-line/external-shell/line-continuation.txt

+# These commands must run under both bash and windows cmd.exe (with GnuWin32
+# tools).
+
+# RUN: echo 'foo bar' \
+# RUN: | FileCheck %s
+# RUN: echo \
+# RUN: 'foo baz' \
+# RUN: | FileCheck %s
+# RUN: echo 'foo bar' \
+# RUN: | FileCheck %s
+
+# CHECK: foo bar

llvm/trunk/utils/lit/tests/Inputs/shtest-run-at-line/external-shell/lit.local.cfg

+import lit.formats
+config.test_format = lit.formats.ShTest(execute_external=True)

llvm/trunk/utils/lit/tests/Inputs/shtest-run-at-line/internal-shell/basic.txt

+# RUN: true
+# RUN: false
+# RUN: true

llvm/trunk/utils/lit/tests/Inputs/shtest-run-at-line/internal-shell/line-continuation.txt

+# RUN: : first line continued \
+# RUN:   to second line
+# RUN: echo 'foo bar' \
+# RUN: | FileCheck %s
+# RUN: echo \
+# RUN: 'foo baz' \
+# RUN: | FileCheck %s
+# RUN: echo 'foo bar' \
+# RUN: | FileCheck %s
+
+# CHECK: foo bar

llvm/trunk/utils/lit/tests/Inputs/shtest-run-at-line/internal-shell/lit.local.cfg

+import lit.formats
+config.test_format = lit.formats.ShTest(execute_external=False)

llvm/trunk/utils/lit/tests/Inputs/shtest-run-at-line/lit.cfg

+config.name = 'shtest-run-at-line'
+config.suffixes = ['.txt']

llvm/trunk/utils/lit/tests/Inputs/shtest-shell/colon-error.txt

+# Check error on an unsupported ":". (cannot be part of a pipeline)
+#
+# RUN: : | echo "hello"

llvm/trunk/utils/lit/tests/lit.cfg

 
 # Add llvm and lit tools directories if this config is being loaded indirectly.
 path = config.environment['PATH']
 for attribute in ('llvm_tools_dir', 'lit_tools_dir'):
     directory = getattr(config, attribute, None)
     if directory:
         path = os.path.pathsep.join((directory, path))
 config.environment['PATH'] = path
+
+# These substitutions are needed only in tests where the external shell is used
+# and could be either bash or windows cmd.exe.  Substitutions are expected to
+# be expanded in double quotes.
+isWin32CMDEXE = lit_config.isWindows and not lit_config.getBashPath()
+if isWin32CMDEXE:
+    config.substitutions.append(('%{pdbg0}', "echo '"))
+    config.substitutions.append(('%{pdbg1}', "' > nul"))
+else:
+    config.substitutions.append(('%{pdbg0}', ": '"))
+    config.substitutions.append(('%{pdbg1}', "'"))

llvm/trunk/utils/lit/tests/max-failures.py

 # Check the behavior of --max-failures option.
 #
 # RUN: not %{lit} -j 1 -v %{inputs}/max-failures > %t.out
 # RUN: not %{lit} --max-failures=1 -j 1 -v %{inputs}/max-failures >> %t.out
 # RUN: not %{lit} --max-failures=2 -j 1 -v %{inputs}/max-failures >> %t.out
 # RUN: not %{lit} --max-failures=0 -j 1 -v %{inputs}/max-failures 2>> %t.out
 # RUN: FileCheck < %t.out %s
 #
 # END.
 
-# CHECK: Failing Tests (26)
+# CHECK: Failing Tests (27)
 # CHECK: Failing Tests (1)
 # CHECK: Failing Tests (2)
 # CHECK: error: Setting --max-failures to 0 does not have any effect.

llvm/trunk/utils/lit/tests/shtest-format.py

 # CHECK-NEXT: --
 # CHECK-NEXT: printf "line 1
 # CHECK-NEXT: false
 # CHECK-NEXT: --
 # CHECK-NEXT: Exit Code: 1
 #
 # CHECK: Command Output (stdout):
 # CHECK-NEXT: --
+# CHECK-NEXT: $ ":" "RUN: at line 1"
 # CHECK-NEXT: $ "printf"
 # CHECK-NEXT: # command output:
 # CHECK-NEXT: line 1: failed test output on stdout
 # CHECK-NEXT: line 2: failed test output on stdout
 
 # CHECK: UNRESOLVED: shtest-format :: no-test-line.txt
 # CHECK: PASS: shtest-format :: pass.txt
 # CHECK: UNSUPPORTED: shtest-format :: requires-any-missing.txt

llvm/trunk/utils/lit/tests/shtest-output-printing.py

 # CHECK-NEXT: *** TEST 'shtest-output-printing :: basic.txt' FAILED ***
 # CHECK-NEXT: Script:
 # CHECK-NEXT: --
 # CHECK:      --
 # CHECK-NEXT: Exit Code: 1
 #
 # CHECK:      Command Output
 # CHECK-NEXT: --
+# CHECK-NEXT: $ ":" "RUN: at line 1"
 # CHECK-NEXT: $ "true"
+# CHECK-NEXT: $ ":" "RUN: at line 2"
 # CHECK-NEXT: $ "echo" "hi"
 # CHECK-NEXT: # command output:
 # CHECK-NEXT: hi
 #
-# CHECK:      $ "wc" "missing-file"
+# CHECK:      $ ":" "RUN: at line 3"
+# CHECK-NEXT: $ "wc" "missing-file"
 # CHECK-NEXT: # redirected output from '{{.*(/|\\\\)}}basic.txt.tmp.out':
 # CHECK-NEXT: missing-file{{.*}} No such file or directory
 # CHECK:      note: command had no output on stdout or stderr
 # CHECK-NEXT: error: command failed with exit status: 1

llvm/trunk/utils/lit/tests/shtest-run-at-line.py

+# Check that -vv makes the line number of the failing RUN command clear.
+# (-v is actually sufficient in the case of the internal shell.)
+#
+# RUN: not %{lit} -j 1 -vv %{inputs}/shtest-run-at-line > %t.out
+# RUN: FileCheck --input-file %t.out -Dpdbg0="%{pdbg0}" -Dpdbg1="%{pdbg1}" %s
+#
+# END.
+
+
+# CHECK: Testing: 4 tests
+
+
+# In the case of the external shell, we check for only RUN lines in stderr in
+# case some shell implementations format "set -x" output differently.
+
+# CHECK-LABEL: FAIL: shtest-run-at-line :: external-shell/basic.txt
+
+# CHECK:      Script:
+# CHECK:      [[pdbg0]]RUN: at line 4[[pdbg1]] &&  true
+# CHECK-NEXT: [[pdbg0]]RUN: at line 5[[pdbg1]] &&  false
+# CHECK-NEXT: [[pdbg0]]RUN: at line 6[[pdbg1]] &&  true
+
+# CHECK:     RUN: at line 4
+# CHECK:     RUN: at line 5
+# CHECK-NOT: RUN
+
+# CHECK-LABEL: FAIL: shtest-run-at-line :: external-shell/line-continuation.txt
+
+# CHECK:      Script:
+# CHECK:      [[pdbg0]]RUN: at line 4[[pdbg1]] &&  echo 'foo bar'  | FileCheck
+# CHECK-NEXT: [[pdbg0]]RUN: at line 6[[pdbg1]] &&  echo  'foo baz'  | FileCheck
+# CHECK-NEXT: [[pdbg0]]RUN: at line 9[[pdbg1]] &&  echo 'foo bar'  | FileCheck
+
+# CHECK:     RUN: at line 4
+# CHECK:     RUN: at line 6
+# CHECK-NOT: RUN
+
+
+# CHECK-LABEL: FAIL: shtest-run-at-line :: internal-shell/basic.txt
+
+# CHECK:      Script:
+# CHECK:      : 'RUN: at line 1' &&  true
+# CHECK-NEXT: : 'RUN: at line 2' &&  false
+# CHECK-NEXT: : 'RUN: at line 3' &&  true
+
+# CHECK:      Command Output (stdout)
+# CHECK:      $ ":" "RUN: at line 1"
+# CHECK-NEXT: $ "true"
+# CHECK-NEXT: $ ":" "RUN: at line 2"
+# CHECK-NEXT: $ "false"
+# CHECK-NOT:  RUN
+
+# CHECK-LABEL: FAIL: shtest-run-at-line :: internal-shell/line-continuation.txt
+
+# CHECK:      Script:
+# CHECK:      : 'RUN: at line 1' &&  : first line continued to second line
+# CHECK-NEXT: : 'RUN: at line 3' &&  echo 'foo bar'  | FileCheck
+# CHECK-NEXT: : 'RUN: at line 5' &&  echo  'foo baz'  | FileCheck
+# CHECK-NEXT: : 'RUN: at line 8' &&  echo 'foo bar'  | FileCheck
+
+# CHECK:      Command Output (stdout)
+# CHECK:      $ ":" "RUN: at line 1"
+# CHECK-NEXT: $ ":" "first" "line" "continued" "to" "second" "line"
+# CHECK-NEXT: $ ":" "RUN: at line 3"
+# CHECK-NEXT: $ "echo" "foo bar"
+# CHECK-NEXT: $ "FileCheck" "{{.*}}"
+# CHECK-NEXT: $ ":" "RUN: at line 5"
+# CHECK-NEXT: $ "echo" "foo baz"
+# CHECK-NEXT: $ "FileCheck" "{{.*}}"
+# CHECK-NOT:  RUN

llvm/trunk/utils/lit/tests/shtest-shell.py

 # CHECK: FAIL: shtest-shell :: cat-error-1.txt
 # CHECK: *** TEST 'shtest-shell :: cat-error-1.txt' FAILED ***
 # CHECK: $ "cat" "temp1.txt"
 # CHECK: # command stderr:
 # CHECK: [Errno 2] No such file or directory: 'temp1.txt'
 # CHECK: error: command failed with exit status: 1
 # CHECK: ***
 
+# CHECK: FAIL: shtest-shell :: colon-error.txt
+# CHECK: *** TEST 'shtest-shell :: colon-error.txt' FAILED ***
+# CHECK: $ ":"
+# CHECK: # command stderr:
+# CHECK: Unsupported: ':' cannot be part of a pipeline
+# CHECK: error: command failed with exit status: 127
+# CHECK: ***
+
 # CHECK: FAIL: shtest-shell :: diff-error-0.txt
 # CHECK: *** TEST 'shtest-shell :: diff-error-0.txt' FAILED ***
 # CHECK: $ "diff" "diff-error-0.txt" "diff-error-0.txt"
 # CHECK: # command stderr:
 # CHECK: Unsupported: 'diff' cannot be part of a pipeline
 # CHECK: error: command failed with exit status: 127
 # CHECK: ***
 

 # CHECK: 'not-a-real-command': command not found
 # CHECK: error: command failed with exit status: 127
 # CHECK: ***
 
 # FIXME: The output here sucks.
 #
 # CHECK: FAIL: shtest-shell :: error-1.txt
 # CHECK: *** TEST 'shtest-shell :: error-1.txt' FAILED ***
-# CHECK: shell parser error on: 'echo "missing quote'
+# CHECK: shell parser error on: ': \'RUN: at line 3\' && echo "missing quote'
 # CHECK: ***
 
 # CHECK: FAIL: shtest-shell :: error-2.txt
 # CHECK: *** TEST 'shtest-shell :: error-2.txt' FAILED ***
 # CHECK: Unsupported redirect:
 # CHECK: ***
 
 # CHECK: FAIL: shtest-shell :: mkdir-error-0.txt

 # CHECK: FAIL: shtest-shell :: rm-error-3.txt
 # CHECK: *** TEST 'shtest-shell :: rm-error-3.txt' FAILED ***
 # CHECK: Exit Code: 1
 # CHECK: ***
 
 # CHECK: PASS: shtest-shell :: sequencing-0.txt
 # CHECK: XFAIL: shtest-shell :: sequencing-1.txt
 # CHECK: PASS: shtest-shell :: valid-shell.txt
-# CHECK: Failing Tests (26)
+# CHECK: Failing Tests (27)

llvm/trunk/utils/lit/tests/unit/TestRunner.py

def test_lists(self):
                               ['one', 'two', 'three', 'four'])
 
     def test_commands(self):
         parsers = self.make_parsers()
         self.parse_test(parsers)
         cmd_parser = self.get_parser(parsers, 'MY_RUN:')
         value = cmd_parser.getValue()
         self.assertEqual(len(value), 2)  # there are only two run lines
-        self.assertEqual(value[0].strip(), 'baz')
-        self.assertEqual(value[1].strip(), 'foo  bar')
+        self.assertEqual(value[0].strip(), "%dbg(MY_RUN: at line 4) &&  baz")
+        self.assertEqual(value[1].strip(), "%dbg(MY_RUN: at line 7) &&  foo  bar")
 
     def test_custom(self):
         parsers = self.make_parsers()
         self.parse_test(parsers)
         custom_parser = self.get_parser(parsers, 'MY_CUSTOM:')
         value = custom_parser.getValue()
         self.assertEqual(value, ['a', 'b', 'c'])