[lit] Implement non-pipelined echo commands internally

[rnk]

Summary

This speeds up the LLD test suite on Windows by 3x. Most of the time is
spent on lld/test/ELF/linkerscript/diagnostics.s, which repeatedly
constructs linker scripts with appending echo commands.

[dlj]

Looks pretty good... just a few nits and one suggestion.

llvm/utils/lit/lit/TestRunner.py

7 ↗

(On Diff #105546)

Minor nit: you should usually have access to cStringIO, which might be faster. The common pattern to use it goes like this:

try:
  import cStringIO as StringIO
except ImportError:
  import StringIO

245 ↗

(On Diff #105546)

Would it make sense to use while instead of if? (That would handle the case of 'echo -e -n blah').

262 ↗

(On Diff #105546)

(Syntax nit) It looks like args will always be a list, even if it's 0-length, right? You can omit the if-check in that case (for-loop over an empty list is effectively a no-op).

421 ↗

(On Diff #105546)

(Minor nit) For consistency, you might want to drop the outer parens. (Or leave them, I just wanted to double-check.)

[rnk]

llvm/utils/lit/lit/TestRunner.py

7 ↗

(On Diff #105546)

Done. It's kind of a micro-optimization, so I wasn't sure if it was worth it.

262 ↗

(On Diff #105546)

I need to check this for stdout.write(maybeUnescape(args[-1])), though. It seemed simpler to put the for loop inside the conditional as well.

421 ↗

(On Diff #105546)

Oops, this was a two-line conditional that got shortened.

[rnk]

• comments

llvm/trunk/utils/lit/lit/TestRunner.py

 from __future__ import absolute_import
 import os, signal, subprocess, sys
 import re
 import platform
 import tempfile
 import threading
 
+try:
+    import cStringIO as StringIO
+except ImportError:
+    import StringIO
+
 from lit.ShCommands import GlobItem
 import lit.ShUtil as ShUtil
 import lit.Test as Test
 import lit.util
 from lit.util import to_bytes, to_string
 from lit.BooleanExpression import BooleanExpression
 
 class InternalShellError(Exception):
for arg_idx, arg in enumerate(cmd.args[1:]):
         # Partition the string into KEY=VALUE.
         key, eq, val = arg.partition('=')
         # Stop if there was no equals.
         if eq == '':
             break
         env.env[key] = val
     cmd.args = cmd.args[arg_idx+1:]
 
+def executeBuiltinEcho(cmd, shenv):
+    """Interpret a redirected echo command"""
+    opened_files = []
+    stdin, stdout, stderr = processRedirects(cmd, subprocess.PIPE, shenv,
+                                             opened_files)
+    if stdin != subprocess.PIPE or stderr != subprocess.PIPE:
+        raise InternalShellError(
+                cmd, "stdin and stderr redirects not supported for echo")
+
+    # Some tests have un-redirected echo commands to help debug test failures.
+    # Buffer our output and return it to the caller.
+    is_redirected = True
+    if stdout == subprocess.PIPE:
+        is_redirected = False
+        stdout = StringIO.StringIO()
+    elif kIsWindows:
+        # Reopen stdout in binary mode to avoid CRLF translation. The versions
+        # of echo we are replacing on Windows all emit plain LF, and the LLVM
+        # tests now depend on this.
+        stdout = open(stdout.name, stdout.mode + 'b')
+        opened_files.append((None, None, stdout, None))
+
+    # Implement echo flags. We only support -e and -n, and not yet in
+    # combination. We have to ignore unknown flags, because `echo "-D FOO"`
+    # prints the dash.
+    args = cmd.args[1:]
+    interpret_escapes = False
+    write_newline = True
+    while len(args) >= 1 and args[0] in ('-e', '-n'):
+        flag = args[0]
+        args = args[1:]
+        if flag == '-e':
+            interpret_escapes = True
+        elif flag == '-n':
+            write_newline = False
+
+    def maybeUnescape(arg):
+        if not interpret_escapes:
+            return arg
+        # Python string escapes and "echo" escapes are obviously different, but
+        # this should be enough for the LLVM test suite.
+        return arg.decode('string_escape')
+
+    if args:
+        for arg in args[:-1]:
+            stdout.write(maybeUnescape(arg))
+            stdout.write(' ')
+        stdout.write(maybeUnescape(args[-1]))
+    if write_newline:
+        stdout.write('\n')
+
+    for (name, mode, f, path) in opened_files:
+        f.close()
+
+    if not is_redirected:
+        return stdout.getvalue()
+    return ""
+
 def processRedirects(cmd, stdin_source, cmd_shenv, opened_files):
     """Return the standard fds for cmd after applying redirects
 
     Returns the three standard file descriptors for the new child process.  Each
     fd may be an open, writable file object or a sentinel value from the
     subprocess module.
     """
 
if cmd.commands[0].args[0] == 'cd':
         if os.path.isabs(newdir):
             shenv.cwd = newdir
         else:
             shenv.cwd = os.path.realpath(os.path.join(shenv.cwd, newdir))
         # The cd builtin always succeeds. If the directory does not exist, the
         # following Popen calls will fail instead.
         return 0
 
+    # Handle "echo" as a builtin if it is not part of a pipeline. This greatly
+    # speeds up tests that construct input files by repeatedly echo-appending to
+    # a file.
+    # FIXME: Standardize on the builtin echo implementation. We can use a
+    # temporary file to sidestep blocking pipe write issues.
+    if cmd.commands[0].args[0] == 'echo' and len(cmd.commands) == 1:
+        output = executeBuiltinEcho(cmd.commands[0], shenv)
+        results.append(ShellCommandResult(cmd.commands[0], output, "", 0,
+                                          False))
+        return 0
+
     if cmd.commands[0].args[0] == 'export':
         if len(cmd.commands) != 1:
             raise ValueError("'export' cannot be part of a pipeline")
         if len(cmd.commands[0].args) != 2:
             raise ValueError("'export' supports only one argument")
         updateEnv(shenv, cmd.commands[0])
         return 0