[lit] Make internal diff work in pipelines

[jdenny]

Summary

When using lit's internal shell, RUN lines like the following
accidentally execute an external diff instead of lit's internal
diff:

# RUN: program | diff file -
# RUN: not diff file1 file2 | FileCheck %s

Such cases exist now, in clang/test/Analysis for example. We are
preparing patches to ensure lit's internal diff is called in such
cases, which will then fail because lit's internal diff cannot
currently be used in pipelines and doesn't recognize - as a
command-line option.

To enable pipelines, this patch moves lit's diff implementation into
an out-of-process script, similar to lit's cat implementation. A
follow-up patch will implement - to mean stdin.

Also, when lit's diff prints differences to stdout in Windows, this
patch ensures it always terminate lines with \n not \r\n. That
way, strict FileCheck directives checking the diff output succeed in
both Linux and Windows. This wasn't an issue when diff was internal
to lit because diff didn't then write to the true stdout, which is
where the \n -> \r\n conversion happened in Python.

[probinson]

+Maggie who did the original diff implementation, AFAICT.

As before, someone with actual Python chops should review this. I've made some functionality remarks.

llvm/utils/lit/lit/builtin_commands/diff.py

197

I've also seen "a" and "U1" in my searches.
Does this correctly handle combined short options? e.g. "-aub" which I see used a fair amount.
I didn't notice --strip-trailing-cr anywhere but presumably the original author needed it, so probably should keep it.
I was going to say we don't need to bother with "r" but then I found a case that used it!

llvm/utils/lit/tests/shtest-shell.py

7

Is this for your temporary benefit or did you intend to make this change permanent? "-color" in particular seems odd here.

[jdenny]

In D66574#1641050, @probinson wrote:

+Maggie who did the original diff implementation, AFAICT.

Good idea.

Thanks for the review.

llvm/utils/lit/lit/builtin_commands/diff.py

197

I've also seen "a" and "U1" in my searches.

Looks like those will be easy to implement. difflib appears to provide functionality for the latter. I'll work on them.

Does this correctly handle combined short options? e.g. "-aub" which I see used a fair amount.

I believe so, but I'll add some tests to be sure.

Thanks for researching the current usage!

llvm/utils/lit/tests/shtest-shell.py

7

Thanks for catching that. Sorry, I meant to remove it.

[jdenny]

I made several changes:

• Added missing functools

[jdenny]

In D66574#1643790, @jdenny wrote:

I made several changes:

• Added missing functools

• ... import to diff.py.
• Extracted - implementation. I'll put that in a separate patch to make it easier to review.
• Fixed some missing newlines in diagnostics (bugs I introduced when converting from exceptions).
• Cleaned up some tests.

[jdenny]

llvm/utils/lit/lit/builtin_commands/diff.py

197

I'll address these in separate patches.

[stella.stamenova]

LGTM as long as all the tests still pass

[jdenny]

In D66574#1666005, @stella.stamenova wrote:

LGTM as long as all the tests still pass

Thanks. Yes, they pass for me. I'm busy now, but I'll try to find some time soon to push this and propose the next patch (support for - to mean stdin).

[jdenny]

I had to revert this because it broke a Windows bot. I didn't consider that this patch not only enables internal diff to function in pipelines but also causes it to be used instead of external diffs when in pipelines. Thus, it cannot be pushed without additional patches that add features like -, which are sometimes used in the test suite when diff appears in pipelines.

[jdenny]

Rebased onto D68839 (recently re-landed). As a result, the diff.py created here has to import lit.util for to_string. The only way I found to make that possible was to set PYTHONPATH in TestRunner.py when calling diff.py.

[jdenny]

Rebased this onto D69207.

Changed diff so that, when printing differences to stdout in Windows, it always terminate lines with \n not \r\n. That way, strict FileCheck directives checking the diff output succeed in both Linux and Windows. This somehow wasn't an issue when diff
was internal to lit.

llvm/utils/lit/lit/TestRunner.py

 from __future__ import absolute_import
-import difflib
 import errno
-import functools
 import io
 import itertools
 import getopt
-import locale
 import os, signal, subprocess, sys
 import re
 import stat
 import platform
 import shutil
 import tempfile
 import threading
 
for dir in args:
         else:
             try:
                 lit.util.mkdir(dir)
             except OSError as err:
                 stderr.write("Error: 'mkdir' command failed, %s\n" % str(err))
                 exitCode = 1
     return ShellCommandResult(cmd, "", stderr.getvalue(), exitCode, False)
 
-def executeBuiltinDiff(cmd, cmd_shenv):
-    """executeBuiltinDiff - Compare files line by line."""
-    args = expand_glob_expressions(cmd.args, cmd_shenv.cwd)[1:]
-    try:
-        opts, args = getopt.gnu_getopt(args, "wbuU:r", ["strip-trailing-cr"])
-    except getopt.GetoptError as err:
-        raise InternalShellError(cmd, "Unsupported: 'diff':  %s" % str(err))
-
-    filelines, filepaths, dir_trees = ([] for i in range(3))
-    ignore_all_space = False
-    ignore_space_change = False
-    unified_diff = False
-    num_context_lines = 3
-    recursive_diff = False
-    strip_trailing_cr = False
-    for o, a in opts:
-        if o == "-w":
-            ignore_all_space = True
-        elif o == "-b":
-            ignore_space_change = True
-        elif o == "-u":
-            unified_diff = True
-        elif o.startswith("-U"):
-            unified_diff = True
-            try:
-                num_context_lines = int(a)
-                if num_context_lines < 0:
-                    raise ValueException
-            except:
-                raise InternalShellError(cmd,
-                                         "Error: invalid '-U' argument: {}\n"
-                                         .format(a))
-        elif o == "-r":
-            recursive_diff = True
-        elif o == "--strip-trailing-cr":
-            strip_trailing_cr = True
-        else:
-            assert False, "unhandled option"
-
-    if len(args) != 2:
-        raise InternalShellError(cmd, "Error:  missing or extra operand")
-
-    def getDirTree(path, basedir=""):
-        # Tree is a tuple of form (dirname, child_trees).
-        # An empty dir has child_trees = [], a file has child_trees = None.
-        child_trees = []
-        for dirname, child_dirs, files in os.walk(os.path.join(basedir, path)):
-            for child_dir in child_dirs:
-                child_trees.append(getDirTree(child_dir, dirname))
-            for filename in files:
-                child_trees.append((filename, None))
-            return path, sorted(child_trees)
-
-    def compareTwoFiles(filepaths):
-        filelines = []
-        for file in filepaths:
-            with open(file, 'rb') as file_bin:
-                filelines.append(file_bin.readlines())
-
-        try:
-            return compareTwoTextFiles(filepaths, filelines,
-                                       locale.getpreferredencoding(False))
-        except UnicodeDecodeError:
-            try:
-                return compareTwoTextFiles(filepaths, filelines, "utf-8")
-            except:
-                return compareTwoBinaryFiles(filepaths, filelines)
-
-    def compareTwoBinaryFiles(filepaths, filelines):
-        exitCode = 0
-        if hasattr(difflib, 'diff_bytes'):
-            # python 3.5 or newer
-            diffs = difflib.diff_bytes(difflib.unified_diff, filelines[0],
-                                       filelines[1], filepaths[0].encode(),
-                                       filepaths[1].encode(),
-                                       n = num_context_lines)
-            diffs = [diff.decode(errors="backslashreplace") for diff in diffs]
-        else:
-            # python 2.7
-            func = difflib.unified_diff if unified_diff else difflib.context_diff
-            diffs = func(filelines[0], filelines[1], filepaths[0], filepaths[1],
-                         n = num_context_lines)
-
-        for diff in diffs:
-            stdout.write(to_string(diff))
-            exitCode = 1
-        return exitCode
-
-    def compareTwoTextFiles(filepaths, filelines_bin, encoding):
-        filelines = []
-        for lines_bin in filelines_bin:
-            lines = []
-            for line_bin in lines_bin:
-                line = line_bin.decode(encoding=encoding)
-                lines.append(line)
-            filelines.append(lines)
-
-        exitCode = 0
-        def compose2(f, g):
-            return lambda x: f(g(x))
-
-        f = lambda x: x
-        if strip_trailing_cr:
-            f = compose2(lambda line: line.replace('\r\n', '\n'), f)
-        if ignore_all_space or ignore_space_change:
-            ignoreSpace = lambda line, separator: separator.join(line.split())
-            ignoreAllSpaceOrSpaceChange = functools.partial(ignoreSpace, separator='' if ignore_all_space else ' ')
-            f = compose2(ignoreAllSpaceOrSpaceChange, f)
-
-        for idx, lines in enumerate(filelines):
-            filelines[idx]= [f(line) for line in lines]
-
-        func = difflib.unified_diff if unified_diff else difflib.context_diff
-        for diff in func(filelines[0], filelines[1], filepaths[0], filepaths[1],
-                         n = num_context_lines):
-            stdout.write(to_string(diff))
-            exitCode = 1
-        return exitCode
-
-    def printDirVsFile(dir_path, file_path):
-        if os.path.getsize(file_path):
-            msg = "File %s is a directory while file %s is a regular file"
-        else:
-            msg = "File %s is a directory while file %s is a regular empty file"
-        stdout.write(msg % (dir_path, file_path) + "\n")
-
-    def printFileVsDir(file_path, dir_path):
-        if os.path.getsize(file_path):
-            msg = "File %s is a regular file while file %s is a directory"
-        else:
-            msg = "File %s is a regular empty file while file %s is a directory"
-        stdout.write(msg % (file_path, dir_path) + "\n")
-
-    def printOnlyIn(basedir, path, name):
-        stdout.write("Only in %s: %s\n" % (os.path.join(basedir, path), name))
-
-    def compareDirTrees(dir_trees, base_paths=["", ""]):
-        # Dirnames of the trees are not checked, it's caller's responsibility,
-        # as top-level dirnames are always different. Base paths are important
-        # for doing os.walk, but we don't put it into tree's dirname in order
-        # to speed up string comparison below and while sorting in getDirTree.
-        left_tree, right_tree = dir_trees[0], dir_trees[1]
-        left_base, right_base = base_paths[0], base_paths[1]
-
-        # Compare two files or report file vs. directory mismatch.
-        if left_tree[1] is None and right_tree[1] is None:
-            return compareTwoFiles([os.path.join(left_base, left_tree[0]),
-                                    os.path.join(right_base, right_tree[0])])
-
-        if left_tree[1] is None and right_tree[1] is not None:
-            printFileVsDir(os.path.join(left_base, left_tree[0]),
-                           os.path.join(right_base, right_tree[0]))
-            return 1
-
-        if left_tree[1] is not None and right_tree[1] is None:
-            printDirVsFile(os.path.join(left_base, left_tree[0]),
-                           os.path.join(right_base, right_tree[0]))
-            return 1
-
-        # Compare two directories via recursive use of compareDirTrees.
-        exitCode = 0
-        left_names = [node[0] for node in left_tree[1]]
-        right_names = [node[0] for node in right_tree[1]]
-        l, r = 0, 0
-        while l < len(left_names) and r < len(right_names):
-            # Names are sorted in getDirTree, rely on that order.
-            if left_names[l] < right_names[r]:
-                exitCode = 1
-                printOnlyIn(left_base, left_tree[0], left_names[l])
-                l += 1
-            elif left_names[l] > right_names[r]:
-                exitCode = 1
-                printOnlyIn(right_base, right_tree[0], right_names[r])
-                r += 1
-            else:
-                exitCode |= compareDirTrees([left_tree[1][l], right_tree[1][r]],
-                                            [os.path.join(left_base, left_tree[0]),
-                                            os.path.join(right_base, right_tree[0])])
-                l += 1
-                r += 1
-
-        # At least one of the trees has ended. Report names from the other tree.
-        while l < len(left_names):
-            exitCode = 1
-            printOnlyIn(left_base, left_tree[0], left_names[l])
-            l += 1
-        while r < len(right_names):
-            exitCode = 1
-            printOnlyIn(right_base, right_tree[0], right_names[r])
-            r += 1
-        return exitCode
-
-    stderr = StringIO()
-    stdout = StringIO()
-    exitCode = 0
-    try:
-        for file in args:
-            if not os.path.isabs(file):
-                file = os.path.realpath(os.path.join(cmd_shenv.cwd, file))
-    
-            if recursive_diff:
-                dir_trees.append(getDirTree(file))
-            else:
-                filepaths.append(file)
-
-        if not recursive_diff:
-            exitCode = compareTwoFiles(filepaths)
-        else:
-            exitCode = compareDirTrees(dir_trees)
-
-    except IOError as err:
-        stderr.write("Error: 'diff' command failed, %s\n" % str(err))
-        exitCode = 1
-
-    return ShellCommandResult(cmd, stdout.getvalue(), stderr.getvalue(), exitCode, False)
-
 def executeBuiltinRm(cmd, cmd_shenv):
     """executeBuiltinRm - Removes (deletes) files or directories."""
     args = expand_glob_expressions(cmd.args, cmd_shenv.cwd)[1:]
     try:
         opts, args = getopt.gnu_getopt(args, "frR", ["--recursive"])
     except getopt.GetoptError as err:
         raise InternalShellError(cmd, "Unsupported: 'rm':  %s" % str(err))
 
def _executeShCmd(cmd, shenv, results, timeoutHelper):
     if cmd.commands[0].args[0] == 'mkdir':
         if len(cmd.commands) != 1:
             raise InternalShellError(cmd.commands[0], "Unsupported: 'mkdir' "
                                      "cannot be part of a pipeline")
         cmdResult = executeBuiltinMkdir(cmd.commands[0], shenv)
         results.append(cmdResult)
         return cmdResult.exitCode
 
-    if cmd.commands[0].args[0] == 'diff':
-        if len(cmd.commands) != 1:
-            raise InternalShellError(cmd.commands[0], "Unsupported: 'diff' "
-                                     "cannot be part of a pipeline")
-        cmdResult = executeBuiltinDiff(cmd.commands[0], shenv)
-        results.append(cmdResult)
-        return cmdResult.exitCode
-
     if cmd.commands[0].args[0] == 'rm':
         if len(cmd.commands) != 1:
             raise InternalShellError(cmd.commands[0], "Unsupported: 'rm' "
                                      "cannot be part of a pipeline")
         cmdResult = executeBuiltinRm(cmd.commands[0], shenv)
         results.append(cmdResult)
         return cmdResult.exitCode
 
     if cmd.commands[0].args[0] == ':':
         if len(cmd.commands) != 1:
             raise InternalShellError(cmd.commands[0], "Unsupported: ':' "
                                      "cannot be part of a pipeline")
         results.append(ShellCommandResult(cmd.commands[0], '', '', 0, False))
         return 0;
 
     procs = []
     default_stdin = subprocess.PIPE
     stderrTempFiles = []
     opened_files = []
     named_temp_files = []
-    builtin_commands = set(['cat'])
+    builtin_commands = set(['cat', 'diff'])
     builtin_commands_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "builtin_commands")
     # To avoid deadlock, we use a single stderr stream for piped
     # output. This is null until we have seen some output using
     # stderr.
     for i,j in enumerate(cmd.commands):
         # Reference the global environment by default.
         cmd_shenv = shenv
         args = list(j.args)
for i,j in enumerate(cmd.commands):
                     f.close()
                     named_temp_files.append(f.name)
                     args[i] = arg.replace(kDevNull, f.name)
 
         # Expand all glob expressions
         args = expand_glob_expressions(args, cmd_shenv.cwd)
         if is_builtin_cmd:
             args.insert(0, sys.executable)
+            cmd_shenv.env['PYTHONPATH'] = \
+                os.path.dirname(os.path.abspath(__file__))
             args[1] = os.path.join(builtin_commands_dir ,args[1] + ".py")
 
         # On Windows, do our own command line quoting for better compatibility
         # with some core utility distributions.
         if kIsWindows:
             args = quote_windows_command(args)
 
         try:

llvm/utils/lit/lit/builtin_commands/diff.py

+import difflib
+import functools
+import getopt
+import locale
+import os
+import sys
+
+import util
+from util import to_string
+
+class DiffFlags():
+    def __init__(self):
+        self.ignore_all_space = False
+        self.ignore_space_change = False
+        self.unified_diff = False
+        self.num_context_lines = 3
+        self.recursive_diff = False
+        self.strip_trailing_cr = False
+
+def getDirTree(path, basedir=""):
+    # Tree is a tuple of form (dirname, child_trees).
+    # An empty dir has child_trees = [], a file has child_trees = None.
+    child_trees = []
+    for dirname, child_dirs, files in os.walk(os.path.join(basedir, path)):
+        for child_dir in child_dirs:
+            child_trees.append(getDirTree(child_dir, dirname))
+        for filename in files:
+            child_trees.append((filename, None))
+        return path, sorted(child_trees)
+
+def compareTwoFiles(flags, filepaths):
+    filelines = []
+    for file in filepaths:
+        with open(file, 'rb') as file_bin:
+            filelines.append(file_bin.readlines())
+
+    try:
+        return compareTwoTextFiles(flags, filepaths, filelines,
+                                   locale.getpreferredencoding(False))
+    except UnicodeDecodeError:
+        try:
+            return compareTwoTextFiles(flags, filepaths, filelines, "utf-8")
+        except:
+            return compareTwoBinaryFiles(flags, filepaths, filelines)
+
+def compareTwoBinaryFiles(flags, filepaths, filelines):
+    exitCode = 0
+    if hasattr(difflib, 'diff_bytes'):
+        # python 3.5 or newer
+        diffs = difflib.diff_bytes(difflib.unified_diff, filelines[0],
+                                   filelines[1], filepaths[0].encode(),
+                                   filepaths[1].encode(),
+                                   n = flags.num_context_lines)
+        diffs = [diff.decode(errors="backslashreplace") for diff in diffs]
+    else:
+        # python 2.7
+        if flags.unified_diff:
+            func = difflib.unified_diff
+        else:
+            func = difflib.context_diff
+        diffs = func(filelines[0], filelines[1], filepaths[0], filepaths[1],
+                     n = flags.num_context_lines)
+
+    for diff in diffs:
+        sys.stdout.write(to_string(diff))
+        exitCode = 1
+    return exitCode
+
+def compareTwoTextFiles(flags, filepaths, filelines_bin, encoding):
+    filelines = []
+    for lines_bin in filelines_bin:
+        lines = []
+        for line_bin in lines_bin:
+            line = line_bin.decode(encoding=encoding)
+            lines.append(line)
+        filelines.append(lines)
+
+    exitCode = 0
+    def compose2(f, g):
+        return lambda x: f(g(x))
+
+    f = lambda x: x
+    if flags.strip_trailing_cr:
+        f = compose2(lambda line: line.replace('\r\n', '\n'), f)
+    if flags.ignore_all_space or flags.ignore_space_change:
+        ignoreSpace = lambda line, separator: separator.join(line.split())
+        ignoreAllSpaceOrSpaceChange = functools.partial(ignoreSpace, separator='' if flags.ignore_all_space else ' ')
+        f = compose2(ignoreAllSpaceOrSpaceChange, f)
+
+    for idx, lines in enumerate(filelines):
+        filelines[idx]= [f(line) for line in lines]
+
+    func = difflib.unified_diff if flags.unified_diff else difflib.context_diff
+    for diff in func(filelines[0], filelines[1], filepaths[0], filepaths[1],
+                     n = flags.num_context_lines):
+        sys.stdout.write(to_string(diff))
+        exitCode = 1
+    return exitCode
+
+def printDirVsFile(dir_path, file_path):
+    if os.path.getsize(file_path):
+        msg = "File %s is a directory while file %s is a regular file"
+    else:
+        msg = "File %s is a directory while file %s is a regular empty file"
+    sys.stdout.write(msg % (dir_path, file_path) + "\n")
+
+def printFileVsDir(file_path, dir_path):
+    if os.path.getsize(file_path):
+        msg = "File %s is a regular file while file %s is a directory"
+    else:
+        msg = "File %s is a regular empty file while file %s is a directory"
+    sys.stdout.write(msg % (file_path, dir_path) + "\n")
+
+def printOnlyIn(basedir, path, name):
+    sys.stdout.write("Only in %s: %s\n" % (os.path.join(basedir, path), name))
+
+def compareDirTrees(flags, dir_trees, base_paths=["", ""]):
+    # Dirnames of the trees are not checked, it's caller's responsibility,
+    # as top-level dirnames are always different. Base paths are important
+    # for doing os.walk, but we don't put it into tree's dirname in order
+    # to speed up string comparison below and while sorting in getDirTree.
+    left_tree, right_tree = dir_trees[0], dir_trees[1]
+    left_base, right_base = base_paths[0], base_paths[1]
+
+    # Compare two files or report file vs. directory mismatch.
+    if left_tree[1] is None and right_tree[1] is None:
+        return compareTwoFiles(flags,
+                               [os.path.join(left_base, left_tree[0]),
+                                os.path.join(right_base, right_tree[0])])
+
+    if left_tree[1] is None and right_tree[1] is not None:
+        printFileVsDir(os.path.join(left_base, left_tree[0]),
+                       os.path.join(right_base, right_tree[0]))
+        return 1
+
+    if left_tree[1] is not None and right_tree[1] is None:
+        printDirVsFile(os.path.join(left_base, left_tree[0]),
+                       os.path.join(right_base, right_tree[0]))
+        return 1
+
+    # Compare two directories via recursive use of compareDirTrees.
+    exitCode = 0
+    left_names = [node[0] for node in left_tree[1]]
+    right_names = [node[0] for node in right_tree[1]]
+    l, r = 0, 0
+    while l < len(left_names) and r < len(right_names):
+        # Names are sorted in getDirTree, rely on that order.
+        if left_names[l] < right_names[r]:
+            exitCode = 1
+            printOnlyIn(left_base, left_tree[0], left_names[l])
+            l += 1
+        elif left_names[l] > right_names[r]:
+            exitCode = 1
+            printOnlyIn(right_base, right_tree[0], right_names[r])
+            r += 1
+        else:
+            exitCode |= compareDirTrees(flags,
+                                        [left_tree[1][l], right_tree[1][r]],
+                                        [os.path.join(left_base, left_tree[0]),
+                                        os.path.join(right_base, right_tree[0])])
+            l += 1
+            r += 1
+
+    # At least one of the trees has ended. Report names from the other tree.
+    while l < len(left_names):
+        exitCode = 1
+        printOnlyIn(left_base, left_tree[0], left_names[l])
+        l += 1
+    while r < len(right_names):
+        exitCode = 1
+        printOnlyIn(right_base, right_tree[0], right_names[r])
+        r += 1
+    return exitCode
+
+def main(argv):
+    args = argv[1:]
+    try:
+        opts, args = getopt.gnu_getopt(args, "wbuU:r", ["strip-trailing-cr"])
+    except getopt.GetoptError as err:
+        sys.stderr.write("Unsupported: 'diff': %s\n" % str(err))
+        sys.exit(1)
+
+    flags = DiffFlags()
+    filelines, filepaths, dir_trees = ([] for i in range(3))
+    for o, a in opts:
+        if o == "-w":
+            flags.ignore_all_space = True
+        elif o == "-b":
+            flags.ignore_space_change = True
+        elif o == "-u":
+            flags.unified_diff = True
+        elif o.startswith("-U"):
+            flags.unified_diff = True
+            try:
+                flags.num_context_lines = int(a)
+                if flags.num_context_lines < 0:
+                    raise ValueException
+            except:
+                sys.stderr.write("Error: invalid '-U' argument: {}\n"
+                                 .format(a))
+                sys.exit(1)
+        elif o == "-r":
+            flags.recursive_diff = True
+        elif o == "--strip-trailing-cr":
+            flags.strip_trailing_cr = True
+        else:
+            assert False, "unhandled option"
+
+    if len(args) != 2:
+        sys.stderr.write("Error: missing or extra operand\n")
+        sys.exit(1)
+
+    exitCode = 0
+    try:
+        for file in args:
+            if not os.path.isabs(file):
+                file = os.path.realpath(os.path.join(os.getcwd(), file))
+
+            if flags.recursive_diff:
+                dir_trees.append(getDirTree(file))
+            else:
+                filepaths.append(file)
+
+        if not flags.recursive_diff:
+            exitCode = compareTwoFiles(flags, filepaths)
+        else:
+            exitCode = compareDirTrees(flags, dir_trees)
+
+    except IOError as err:
+        sys.stderr.write("Error: 'diff' command failed, %s\n" % str(err))
+        exitCode = 1
+
+    sys.exit(exitCode)
+
+if __name__ == "__main__":
+    main(sys.argv)

llvm/utils/lit/tests/Inputs/shtest-shell/diff-error-0.txt

-# Check error on a unsupported diff (cannot be part of a pipeline).
-#
-# RUN: diff diff-error-0.txt diff-error-0.txt | echo Output

llvm/utils/lit/tests/Inputs/shtest-shell/diff-pipes.txt

+# RUN: echo foo > %t.foo
+# RUN: echo bar > %t.bar
+
+# Check output pipe.
+# RUN: diff %t.foo %t.foo | FileCheck -allow-empty -check-prefix=EMPTY %s
+# RUN: diff -u %t.foo %t.bar | FileCheck %s && false || true
+
+# Fail so lit will print output.
+# RUN: false
+
+# CHECK:      @@
+# CHECK-NEXT: -foo
+# CHECK-NEXT: +bar
+
+# EMPTY-NOT: {{.}}

llvm/utils/lit/tests/Inputs/shtest-shell/diff-unified-error-0.txt

-# Check bad -U argument.
-# RUN: echo foo > %t
-# RUN: diff -U 30.1 %t %t

llvm/utils/lit/tests/Inputs/shtest-shell/diff-unified-error-1.txt

-# Check bad -U argument.
-# RUN: echo foo > %t
-# RUN: diff -U-1 %t %t

llvm/utils/lit/tests/Inputs/shtest-shell/diff-unified.txt

 # Default is 3 lines of context.
 # RUN: diff -u %t.foo %t.bar && false || true
 
 # Override default of 3 lines of context.
 # RUN: diff -U 2 %t.foo %t.bar && false || true
 # RUN: diff -U4 %t.foo %t.bar && false || true
 # RUN: diff -U0 %t.foo %t.bar && false || true
 
+# Check bad -U argument.
+# RUN: diff -U 30.1 %t.foo %t.foo && false || true
+# RUN: diff -U-1 %t.foo %t.foo && false || true
+
 # Fail so lit will print output.
 # RUN: false

llvm/utils/lit/tests/max-failures.py

 # Check the behavior of --max-failures option.
 #
 # RUN: not %{lit} -j 1 -v %{inputs}/max-failures > %t.out
 # RUN: not %{lit} --max-failures=1 -j 1 -v %{inputs}/max-failures >> %t.out
 # RUN: not %{lit} --max-failures=2 -j 1 -v %{inputs}/max-failures >> %t.out
 # RUN: not %{lit} --max-failures=0 -j 1 -v %{inputs}/max-failures 2>> %t.out
 # RUN: FileCheck < %t.out %s
 #
 # END.
 
-# CHECK: Failing Tests (32)
+# CHECK: Failing Tests (30)
 # CHECK: Failing Tests (1)
 # CHECK: Failing Tests (2)
 # CHECK: error: argument --max-failures: requires positive integer, but found '0'

llvm/utils/lit/tests/shtest-shell.py

 # Check the internal shell handling component of the ShTest format.
 #
 # RUN: not %{lit} -j 1 -v %{inputs}/shtest-shell > %t.out
 # FIXME: Temporarily dump test output so we can debug failing tests on
 # buildbots.
 # RUN: cat %t.out
 # RUN: FileCheck --input-file %t.out %s
 #
 # END.
 
 # CHECK: -- Testing:
 
 # CHECK: FAIL: shtest-shell :: cat-error-0.txt
 # CHECK: *** TEST 'shtest-shell :: cat-error-0.txt' FAILED ***
 # CHECK: $ "cat" "-b" "temp1.txt"

 # CHECK: error: command failed with exit status: 1
 # CHECK: $ "true"
 
 # CHECK: $ "false"
 
 # CHECK: ***
 
 
-# CHECK: FAIL: shtest-shell :: diff-error-0.txt
-# CHECK: *** TEST 'shtest-shell :: diff-error-0.txt' FAILED ***
-# CHECK: $ "diff" "diff-error-0.txt" "diff-error-0.txt"
-# CHECK: # command stderr:
-# CHECK: Unsupported: 'diff' cannot be part of a pipeline
-# CHECK: error: command failed with exit status: 127
-# CHECK: ***
-
 # CHECK: FAIL: shtest-shell :: diff-error-1.txt
 # CHECK: *** TEST 'shtest-shell :: diff-error-1.txt' FAILED ***
 # CHECK: $ "diff" "-B" "temp1.txt" "temp2.txt"
 # CHECK: # command stderr:
 # CHECK: Unsupported: 'diff': option -B not recognized
-# CHECK: error: command failed with exit status: 127
+# CHECK: error: command failed with exit status: 1
 # CHECK: ***
 
 # CHECK: FAIL: shtest-shell :: diff-error-2.txt
 # CHECK: *** TEST 'shtest-shell :: diff-error-2.txt' FAILED ***
 # CHECK: $ "diff" "temp.txt"
 # CHECK: # command stderr:
-# CHECK: Error:  missing or extra operand
-# CHECK: error: command failed with exit status: 127
+# CHECK: Error: missing or extra operand
+# CHECK: error: command failed with exit status: 1
 # CHECK: ***
 
 # CHECK: FAIL: shtest-shell :: diff-error-3.txt
 # CHECK: *** TEST 'shtest-shell :: diff-error-3.txt' FAILED ***
 # CHECK: $ "diff" "temp.txt" "temp1.txt"
 # CHECK: # command stderr:
 # CHECK: Error: 'diff' command failed
 # CHECK: error: command failed with exit status: 1

 # CHECK: --- 1 ----
 # CHECK: ! hello-second
 # CHECK: ***
 
 # CHECK: FAIL: shtest-shell :: diff-error-5.txt
 # CHECK: *** TEST 'shtest-shell :: diff-error-5.txt' FAILED ***
 # CHECK: $ "diff"
 # CHECK: # command stderr:
-# CHECK: Error:  missing or extra operand
-# CHECK: error: command failed with exit status: 127
+# CHECK: Error: missing or extra operand
+# CHECK: error: command failed with exit status: 1
 # CHECK: ***
 
 # CHECK: FAIL: shtest-shell :: diff-error-6.txt
 # CHECK: *** TEST 'shtest-shell :: diff-error-6.txt' FAILED ***
 # CHECK: $ "diff"
 # CHECK: # command stderr:
-# CHECK: Error:  missing or extra operand
-# CHECK: error: command failed with exit status: 127
+# CHECK: Error: missing or extra operand
+# CHECK: error: command failed with exit status: 1
+# CHECK: ***
+
+
+# CHECK: FAIL: shtest-shell :: diff-pipes.txt
+
+# CHECK: *** TEST 'shtest-shell :: diff-pipes.txt' FAILED ***
+
+# CHECK: $ "diff" "{{[^"]*}}.foo" "{{[^"]*}}.foo"
+# CHECK-NOT: note
+# CHECK-NOT: error
+# CHECK: $ "FileCheck"
+# CHECK-NOT: note
+# CHECK-NOT: error
+
+# CHECK: $ "diff" "-u" "{{[^"]*}}.foo" "{{[^"]*}}.bar"
+# CHECK: note: command had no output on stdout or stderr
+# CHECK: error: command failed with exit status: 1
+# CHECK: $ "FileCheck"
+# CHECK-NOT: note
+# CHECK-NOT: error
+# CHECK: $ "true"
+
+# CHECK: $ "false"
+
 # CHECK: ***
 
+
 # CHECK: FAIL: shtest-shell :: diff-r-error-0.txt
 # CHECK: *** TEST 'shtest-shell :: diff-r-error-0.txt' FAILED ***
 # CHECK: $ "diff" "-r" 
 # CHECK: # command output:
 # CHECK: Only in {{.*}}dir1: dir1unique
 # CHECK: Only in {{.*}}dir2: dir2unique
 # CHECK: error: command failed with exit status: 1
 

 # CHECK: error: command failed with exit status: 1
 # CHECK: $ "true"
 
 # CHECK: $ "false"
 
 # CHECK: ***
 
 
-# CHECK: FAIL: shtest-shell :: diff-unified-error-0.txt
-# CHECK: *** TEST 'shtest-shell :: diff-unified-error-0.txt' FAILED ***
-# CHECK: $ "diff" "-U" "30.1" "{{[^"]*}}" "{{[^"]*}}"
-# CHECK: # command stderr:
-# CHECK: Error: invalid '-U' argument: 30.1
-# CHECK: error: command failed with exit status: 127
-# CHECK: ***
-
-# CHECK: FAIL: shtest-shell :: diff-unified-error-1.txt
-# CHECK: *** TEST 'shtest-shell :: diff-unified-error-1.txt' FAILED ***
-# CHECK: $ "diff" "-U-1" "{{[^"]*}}" "{{[^"]*}}"
-# CHECK: # command stderr:
-# CHECK: Error: invalid '-U' argument: -1
-# CHECK: error: command failed with exit status: 127
-# CHECK: ***
-
-
 # CHECK: FAIL: shtest-shell :: diff-unified.txt
 
 # CHECK: *** TEST 'shtest-shell :: diff-unified.txt' FAILED ***
 
 # CHECK: $ "diff" "-u" "{{[^"]*}}.foo" "{{[^"]*}}.bar"
 # CHECK: # command output:
 # CHECK: @@ {{.*}} @@
 # CHECK-NEXT: 3

 # CHECK: # command output:
 # CHECK: @@ {{.*}} @@
 # CHECK-NEXT: -6 foo
 # CHECK-NEXT: +6 bar
 # CHECK-EMPTY:
 # CHECK-NEXT: error: command failed with exit status: 1
 # CHECK-NEXT: $ "true"
 
+# CHECK: $ "diff" "-U" "30.1" "{{[^"]*}}" "{{[^"]*}}"
+# CHECK: # command stderr:
+# CHECK: Error: invalid '-U' argument: 30.1
+# CHECK: error: command failed with exit status: 1
+# CHECK: $ "true"
+
+# CHECK: $ "diff" "-U-1" "{{[^"]*}}" "{{[^"]*}}"
+# CHECK: # command stderr:
+# CHECK: Error: invalid '-U' argument: -1
+# CHECK: error: command failed with exit status: 1
+# CHECK: $ "true"
+
 # CHECK: $ "false"
 
 # CHECK: ***
 
 
 # CHECK: FAIL: shtest-shell :: error-0.txt
 # CHECK: *** TEST 'shtest-shell :: error-0.txt' FAILED ***
 # CHECK: $ "not-a-real-command"

 # CHECK: *** TEST 'shtest-shell :: rm-error-3.txt' FAILED ***
 # CHECK: Exit Code: 1
 # CHECK: ***
 
 # CHECK: PASS: shtest-shell :: rm-unicode-0.txt
 # CHECK: PASS: shtest-shell :: sequencing-0.txt
 # CHECK: XFAIL: shtest-shell :: sequencing-1.txt
 # CHECK: PASS: shtest-shell :: valid-shell.txt
-# CHECK: Failing Tests (32)
+# CHECK: Failing Tests (30)