[lit] Make internal diff work in pipelines

[jdenny]

Summary

When using lit's internal shell, RUN lines like the following
accidentally execute an external diff instead of lit's internal
diff:

# RUN: program | diff file -
# RUN: not diff file1 file2 | FileCheck %s

Such cases exist now, in clang/test/Analysis for example. We are
preparing patches to ensure lit's internal diff is called in such
cases, which will then fail because lit's internal diff cannot
currently be used in pipelines and doesn't recognize - as a
command-line option.

To enable pipelines, this patch moves lit's diff implementation into
an out-of-process script, similar to lit's cat implementation. A
follow-up patch will implement - to mean stdin.

Also, when lit's diff prints differences to stdout in Windows, this
patch ensures it always terminate lines with \n not \r\n. That
way, strict FileCheck directives checking the diff output succeed in
both Linux and Windows. This wasn't an issue when diff was internal
to lit because diff didn't then write to the true stdout, which is
where the \n -> \r\n conversion happened in Python.

[probinson]

+Maggie who did the original diff implementation, AFAICT.

As before, someone with actual Python chops should review this. I've made some functionality remarks.

llvm/utils/lit/lit/builtin_commands/diff.py

196

I've also seen "a" and "U1" in my searches.
Does this correctly handle combined short options? e.g. "-aub" which I see used a fair amount.
I didn't notice --strip-trailing-cr anywhere but presumably the original author needed it, so probably should keep it.
I was going to say we don't need to bother with "r" but then I found a case that used it!

llvm/utils/lit/tests/shtest-shell.py

7

Is this for your temporary benefit or did you intend to make this change permanent? "-color" in particular seems odd here.

[jdenny]

In D66574#1641050, @probinson wrote:

+Maggie who did the original diff implementation, AFAICT.

Good idea.

Thanks for the review.

llvm/utils/lit/lit/builtin_commands/diff.py

196

I've also seen "a" and "U1" in my searches.

Looks like those will be easy to implement. difflib appears to provide functionality for the latter. I'll work on them.

Does this correctly handle combined short options? e.g. "-aub" which I see used a fair amount.

I believe so, but I'll add some tests to be sure.

Thanks for researching the current usage!

llvm/utils/lit/tests/shtest-shell.py

7

Thanks for catching that. Sorry, I meant to remove it.

[jdenny]

I made several changes:

• Added missing functools

[jdenny]

In D66574#1643790, @jdenny wrote:

I made several changes:

• Added missing functools

• ... import to diff.py.
• Extracted - implementation. I'll put that in a separate patch to make it easier to review.
• Fixed some missing newlines in diagnostics (bugs I introduced when converting from exceptions).
• Cleaned up some tests.

[jdenny]

llvm/utils/lit/lit/builtin_commands/diff.py

196

I'll address these in separate patches.

[stella.stamenova]

LGTM as long as all the tests still pass

[jdenny]

In D66574#1666005, @stella.stamenova wrote:

LGTM as long as all the tests still pass

Thanks. Yes, they pass for me. I'm busy now, but I'll try to find some time soon to push this and propose the next patch (support for - to mean stdin).

[jdenny]

I had to revert this because it broke a Windows bot. I didn't consider that this patch not only enables internal diff to function in pipelines but also causes it to be used instead of external diffs when in pipelines. Thus, it cannot be pushed without additional patches that add features like -, which are sometimes used in the test suite when diff appears in pipelines.

[jdenny]

Rebased onto D68839 (recently re-landed). As a result, the diff.py created here has to import lit.util for to_string. The only way I found to make that possible was to set PYTHONPATH in TestRunner.py when calling diff.py.

[jdenny]

Rebased this onto D69207.

Changed diff so that, when printing differences to stdout in Windows, it always terminate lines with \n not \r\n. That way, strict FileCheck directives checking the diff output succeed in both Linux and Windows. This somehow wasn't an issue when diff
was internal to lit.

llvm/utils/lit/lit/TestRunner.py

 from __future__ import absolute_import
-import difflib
 import errno
 import functools
 import io
 import itertools
 import getopt
 import os, signal, subprocess, sys
 import re
 import stat
for dir in args:
         else:
             try:
                 lit.util.mkdir(dir)
             except OSError as err:
                 stderr.write("Error: 'mkdir' command failed, %s\n" % str(err))
                 exitCode = 1
     return ShellCommandResult(cmd, "", stderr.getvalue(), exitCode, False)
 
-def executeBuiltinDiff(cmd, cmd_shenv):
-    """executeBuiltinDiff - Compare files line by line."""
-    args = expand_glob_expressions(cmd.args, cmd_shenv.cwd)[1:]
-    try:
-        opts, args = getopt.gnu_getopt(args, "wbur", ["strip-trailing-cr"])
-    except getopt.GetoptError as err:
-        raise InternalShellError(cmd, "Unsupported: 'diff':  %s" % str(err))
-
-    filelines, filepaths, dir_trees = ([] for i in range(3))
-    ignore_all_space = False
-    ignore_space_change = False
-    unified_diff = False
-    recursive_diff = False
-    strip_trailing_cr = False
-    for o, a in opts:
-        if o == "-w":
-            ignore_all_space = True
-        elif o == "-b":
-            ignore_space_change = True
-        elif o == "-u":
-            unified_diff = True
-        elif o == "-r":
-            recursive_diff = True
-        elif o == "--strip-trailing-cr":
-            strip_trailing_cr = True
-        else:
-            assert False, "unhandled option"
-
-    if len(args) != 2:
-        raise InternalShellError(cmd, "Error:  missing or extra operand")
-
-    def getDirTree(path, basedir=""):
-        # Tree is a tuple of form (dirname, child_trees).
-        # An empty dir has child_trees = [], a file has child_trees = None.
-        child_trees = []
-        for dirname, child_dirs, files in os.walk(os.path.join(basedir, path)):
-            for child_dir in child_dirs:
-                child_trees.append(getDirTree(child_dir, dirname))
-            for filename in files:
-                child_trees.append((filename, None))
-            return path, sorted(child_trees)
-
-    def compareTwoFiles(filepaths):
-        compare_bytes = False
-        encoding = None
-        filelines = []
-        for file in filepaths:
-            try:
-                with open(file, 'r') as f:
-                    filelines.append(f.readlines())
-            except UnicodeDecodeError:
-                try:
-                    with io.open(file, 'r', encoding="utf-8") as f:
-                        filelines.append(f.readlines())
-                    encoding = "utf-8"
-                except:
-                    compare_bytes = True
-
-        if compare_bytes:
-            return compareTwoBinaryFiles(filepaths)
-        else:
-            return compareTwoTextFiles(filepaths, encoding)
-
-    def compareTwoBinaryFiles(filepaths):
-        filelines = []
-        for file in filepaths:
-            with open(file, 'rb') as f:
-                filelines.append(f.readlines())
-
-        exitCode = 0
-        if hasattr(difflib, 'diff_bytes'):
-            # python 3.5 or newer
-            diffs = difflib.diff_bytes(difflib.unified_diff, filelines[0], filelines[1], filepaths[0].encode(), filepaths[1].encode())
-            diffs = [diff.decode() for diff in diffs]
-        else:
-            # python 2.7
-            func = difflib.unified_diff if unified_diff else difflib.context_diff
-            diffs = func(filelines[0], filelines[1], filepaths[0], filepaths[1])
-
-        for diff in diffs:
-            stdout.write(diff)
-            exitCode = 1
-        return exitCode
-
-    def compareTwoTextFiles(filepaths, encoding):
-        filelines = []
-        for file in filepaths:
-            if encoding is None:
-                with open(file, 'r') as f:
-                    filelines.append(f.readlines())
-            else:
-                with io.open(file, 'r', encoding=encoding) as f:
-                    filelines.append(f.readlines())
-
-        exitCode = 0
-        def compose2(f, g):
-            return lambda x: f(g(x))
-
-        f = lambda x: x
-        if strip_trailing_cr:
-            f = compose2(lambda line: line.rstrip('\r'), f)
-        if ignore_all_space or ignore_space_change:
-            ignoreSpace = lambda line, separator: separator.join(line.split())
-            ignoreAllSpaceOrSpaceChange = functools.partial(ignoreSpace, separator='' if ignore_all_space else ' ')
-            f = compose2(ignoreAllSpaceOrSpaceChange, f)
-
-        for idx, lines in enumerate(filelines):
-            filelines[idx]= [f(line) for line in lines]
-
-        func = difflib.unified_diff if unified_diff else difflib.context_diff
-        for diff in func(filelines[0], filelines[1], filepaths[0], filepaths[1]):
-            stdout.write(diff)
-            exitCode = 1
-        return exitCode
-
-    def printDirVsFile(dir_path, file_path):
-        if os.path.getsize(file_path):
-            msg = "File %s is a directory while file %s is a regular file"
-        else:
-            msg = "File %s is a directory while file %s is a regular empty file"
-        stdout.write(msg % (dir_path, file_path) + "\n")
-
-    def printFileVsDir(file_path, dir_path):
-        if os.path.getsize(file_path):
-            msg = "File %s is a regular file while file %s is a directory"
-        else:
-            msg = "File %s is a regular empty file while file %s is a directory"
-        stdout.write(msg % (file_path, dir_path) + "\n")
-
-    def printOnlyIn(basedir, path, name):
-        stdout.write("Only in %s: %s\n" % (os.path.join(basedir, path), name))
-
-    def compareDirTrees(dir_trees, base_paths=["", ""]):
-        # Dirnames of the trees are not checked, it's caller's responsibility,
-        # as top-level dirnames are always different. Base paths are important
-        # for doing os.walk, but we don't put it into tree's dirname in order
-        # to speed up string comparison below and while sorting in getDirTree.
-        left_tree, right_tree = dir_trees[0], dir_trees[1]
-        left_base, right_base = base_paths[0], base_paths[1]
-
-        # Compare two files or report file vs. directory mismatch.
-        if left_tree[1] is None and right_tree[1] is None:
-            return compareTwoFiles([os.path.join(left_base, left_tree[0]),
-                                    os.path.join(right_base, right_tree[0])])
-
-        if left_tree[1] is None and right_tree[1] is not None:
-            printFileVsDir(os.path.join(left_base, left_tree[0]),
-                           os.path.join(right_base, right_tree[0]))
-            return 1
-
-        if left_tree[1] is not None and right_tree[1] is None:
-            printDirVsFile(os.path.join(left_base, left_tree[0]),
-                           os.path.join(right_base, right_tree[0]))
-            return 1
-
-        # Compare two directories via recursive use of compareDirTrees.
-        exitCode = 0
-        left_names = [node[0] for node in left_tree[1]]
-        right_names = [node[0] for node in right_tree[1]]
-        l, r = 0, 0
-        while l < len(left_names) and r < len(right_names):
-            # Names are sorted in getDirTree, rely on that order.
-            if left_names[l] < right_names[r]:
-                exitCode = 1
-                printOnlyIn(left_base, left_tree[0], left_names[l])
-                l += 1
-            elif left_names[l] > right_names[r]:
-                exitCode = 1
-                printOnlyIn(right_base, right_tree[0], right_names[r])
-                r += 1
-            else:
-                exitCode |= compareDirTrees([left_tree[1][l], right_tree[1][r]],
-                                            [os.path.join(left_base, left_tree[0]),
-                                            os.path.join(right_base, right_tree[0])])
-                l += 1
-                r += 1
-
-        # At least one of the trees has ended. Report names from the other tree.
-        while l < len(left_names):
-            exitCode = 1
-            printOnlyIn(left_base, left_tree[0], left_names[l])
-            l += 1
-        while r < len(right_names):
-            exitCode = 1
-            printOnlyIn(right_base, right_tree[0], right_names[r])
-            r += 1
-        return exitCode
-
-    stderr = StringIO()
-    stdout = StringIO()
-    exitCode = 0
-    try:
-        for file in args:
-            if not os.path.isabs(file):
-                file = os.path.realpath(os.path.join(cmd_shenv.cwd, file))
-    
-            if recursive_diff:
-                dir_trees.append(getDirTree(file))
-            else:
-                filepaths.append(file)
-
-        if not recursive_diff:
-            exitCode = compareTwoFiles(filepaths)
-        else:
-            exitCode = compareDirTrees(dir_trees)
-
-    except IOError as err:
-        stderr.write("Error: 'diff' command failed, %s\n" % str(err))
-        exitCode = 1
-
-    return ShellCommandResult(cmd, stdout.getvalue(), stderr.getvalue(), exitCode, False)
-
 def executeBuiltinRm(cmd, cmd_shenv):
     """executeBuiltinRm - Removes (deletes) files or directories."""
     args = expand_glob_expressions(cmd.args, cmd_shenv.cwd)[1:]
     try:
         opts, args = getopt.gnu_getopt(args, "frR", ["--recursive"])
     except getopt.GetoptError as err:
         raise InternalShellError(cmd, "Unsupported: 'rm':  %s" % str(err))
 
def _executeShCmd(cmd, shenv, results, timeoutHelper):
     if cmd.commands[0].args[0] == 'mkdir':
         if len(cmd.commands) != 1:
             raise InternalShellError(cmd.commands[0], "Unsupported: 'mkdir' "
                                      "cannot be part of a pipeline")
         cmdResult = executeBuiltinMkdir(cmd.commands[0], shenv)
         results.append(cmdResult)
         return cmdResult.exitCode
 
-    if cmd.commands[0].args[0] == 'diff':
-        if len(cmd.commands) != 1:
-            raise InternalShellError(cmd.commands[0], "Unsupported: 'diff' "
-                                     "cannot be part of a pipeline")
-        cmdResult = executeBuiltinDiff(cmd.commands[0], shenv)
-        results.append(cmdResult)
-        return cmdResult.exitCode
-
     if cmd.commands[0].args[0] == 'rm':
         if len(cmd.commands) != 1:
             raise InternalShellError(cmd.commands[0], "Unsupported: 'rm' "
                                      "cannot be part of a pipeline")
         cmdResult = executeBuiltinRm(cmd.commands[0], shenv)
         results.append(cmdResult)
         return cmdResult.exitCode
 
     if cmd.commands[0].args[0] == ':':
         if len(cmd.commands) != 1:
             raise InternalShellError(cmd.commands[0], "Unsupported: ':' "
                                      "cannot be part of a pipeline")
         results.append(ShellCommandResult(cmd.commands[0], '', '', 0, False))
         return 0;
 
     procs = []
     default_stdin = subprocess.PIPE
     stderrTempFiles = []
     opened_files = []
     named_temp_files = []
-    builtin_commands = set(['cat'])
+    builtin_commands = set(['cat', 'diff'])
     builtin_commands_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "builtin_commands")
     # To avoid deadlock, we use a single stderr stream for piped
     # output. This is null until we have seen some output using
     # stderr.
     for i,j in enumerate(cmd.commands):
         # Reference the global environment by default.
         cmd_shenv = shenv
         args = list(j.args)

llvm/utils/lit/lit/builtin_commands/diff.py

+import difflib
+import getopt
+import os
+import sys
+
+class DiffFlags():
+    def __init__(self):
+        self.ignore_all_space = False
+        self.ignore_space_change = False
+        self.unified_diff = False
+        self.recursive_diff = False
+        self.strip_trailing_cr = False
+
+def getDirTree(path, basedir=""):
+    # Tree is a tuple of form (dirname, child_trees).
+    # An empty dir has child_trees = [], a file has child_trees = None.
+    child_trees = []
+    for dirname, child_dirs, files in os.walk(os.path.join(basedir, path)):
+        for child_dir in child_dirs:
+            child_trees.append(getDirTree(child_dir, dirname))
+        for filename in files:
+            child_trees.append((filename, None))
+        return path, sorted(child_trees)
+
+def compareTwoFiles(flags, filepaths):
+    # FIXME: How can we restart stdin if the encoding is wrong?  How can we
+    # read stdin with a different encoding or in binary mode in a way that's
+    # compatible with python 2 and 3?
+    compare_bytes = False
+    encoding = None
+    filelines = []
+    has_stdin = False
+    for file in filepaths:
+        if file == "-":
+            has_stdin = True
+            continue
+        try:
+            with open(file, 'r') as f:
+                filelines.append(f.readlines())
+        except UnicodeDecodeError:
+            try:
+                with io.open(file, 'r', encoding="utf-8") as f:
+                    filelines.append(f.readlines())
+                encoding = "utf-8"
+            except:
+                compare_bytes = True
+
+    if has_stdin and encoding:
+        sys.stderr.write("Unsupported: cannot compare '-' with encoding {}"
+                         .format(encoding))
+        sys.exit(1)
+
+    if compare_bytes:
+        if has_stdin:
+            sys.stderr.write("Unsupported: cannot compare '-' as binary")
+            sys.exit(1)
+        return compareTwoBinaryFiles(flags, filepaths)
+    else:
+        return compareTwoTextFiles(flags, filepaths, encoding)
+
+def compareTwoBinaryFiles(flags, filepaths):
+    filelines = []
+    for file in filepaths:
+        with open(file, 'rb') as f:
+            filelines.append(f.readlines())
+
+    exitCode = 0
+    if hasattr(difflib, 'diff_bytes'):
+        # python 3.5 or newer
+        diffs = difflib.diff_bytes(difflib.unified_diff, filelines[0], filelines[1], filepaths[0].encode(), filepaths[1].encode())
+        diffs = [diff.decode() for diff in diffs]
+    else:
+        # python 2.7
+        if flags.unified_diff:
+            func = difflib.unified_diff
+        else:
+            func = difflib.context_diff
+        diffs = func(filelines[0], filelines[1], filepaths[0], filepaths[1])
+
+    for diff in diffs:
+        sys.stdout.write(diff)
+        exitCode = 1
+    return exitCode
+
+def compareTwoTextFiles(flags, filepaths, encoding):
+    filelines = []
+    for file in filepaths:
+        if file == "-":
+            filelines.append(sys.stdin.readlines())
+        elif encoding is None:
+            with open(file, 'r') as f:
+                filelines.append(f.readlines())
+        else:
+            with io.open(file, 'r', encoding=encoding) as f:
+                filelines.append(f.readlines())
+
+    exitCode = 0
+    def compose2(f, g):
+        return lambda x: f(g(x))
+
+    f = lambda x: x
+    if flags.strip_trailing_cr:
+        f = compose2(lambda line: line.rstrip('\r'), f)
+    if flags.ignore_all_space or flags.ignore_space_change:
+        ignoreSpace = lambda line, separator: separator.join(line.split())
+        ignoreAllSpaceOrSpaceChange = functools.partial(ignoreSpace, separator='' if flags.ignore_all_space else ' ')
+        f = compose2(ignoreAllSpaceOrSpaceChange, f)
+
+    for idx, lines in enumerate(filelines):
+        filelines[idx]= [f(line) for line in lines]
+
+    func = difflib.unified_diff if flags.unified_diff else difflib.context_diff
+    for diff in func(filelines[0], filelines[1], filepaths[0], filepaths[1]):
+        sys.stdout.write(diff)
+        exitCode = 1
+    return exitCode
+
+def printDirVsFile(dir_path, file_path):
+    if os.path.getsize(file_path):
+        msg = "File %s is a directory while file %s is a regular file"
+    else:
+        msg = "File %s is a directory while file %s is a regular empty file"
+    sys.stdout.write(msg % (dir_path, file_path) + "\n")
+
+def printFileVsDir(file_path, dir_path):
+    if os.path.getsize(file_path):
+        msg = "File %s is a regular file while file %s is a directory"
+    else:
+        msg = "File %s is a regular empty file while file %s is a directory"
+    sys.stdout.write(msg % (file_path, dir_path) + "\n")
+
+def printOnlyIn(basedir, path, name):
+    sys.stdout.write("Only in %s: %s\n" % (os.path.join(basedir, path), name))
+
+def compareDirTrees(flags, dir_trees, base_paths=["", ""]):
+    # Dirnames of the trees are not checked, it's caller's responsibility,
+    # as top-level dirnames are always different. Base paths are important
+    # for doing os.walk, but we don't put it into tree's dirname in order
+    # to speed up string comparison below and while sorting in getDirTree.
+    left_tree, right_tree = dir_trees[0], dir_trees[1]
+    left_base, right_base = base_paths[0], base_paths[1]
+
+    # Compare two files or report file vs. directory mismatch.
+    if left_tree[1] is None and right_tree[1] is None:
+        return compareTwoFiles(flags,
+                               [os.path.join(left_base, left_tree[0]),
+                                os.path.join(right_base, right_tree[0])])
+
+    if left_tree[1] is None and right_tree[1] is not None:
+        printFileVsDir(os.path.join(left_base, left_tree[0]),
+                       os.path.join(right_base, right_tree[0]))
+        return 1
+
+    if left_tree[1] is not None and right_tree[1] is None:
+        printDirVsFile(os.path.join(left_base, left_tree[0]),
+                       os.path.join(right_base, right_tree[0]))
+        return 1
+
+    # Compare two directories via recursive use of compareDirTrees.
+    exitCode = 0
+    left_names = [node[0] for node in left_tree[1]]
+    right_names = [node[0] for node in right_tree[1]]
+    l, r = 0, 0
+    while l < len(left_names) and r < len(right_names):
+        # Names are sorted in getDirTree, rely on that order.
+        if left_names[l] < right_names[r]:
+            exitCode = 1
+            printOnlyIn(left_base, left_tree[0], left_names[l])
+            l += 1
+        elif left_names[l] > right_names[r]:
+            exitCode = 1
+            printOnlyIn(right_base, right_tree[0], right_names[r])
+            r += 1
+        else:
+            exitCode |= compareDirTrees(flags,
+                                        [left_tree[1][l], right_tree[1][r]],
+                                        [os.path.join(left_base, left_tree[0]),
+                                        os.path.join(right_base, right_tree[0])])
+            l += 1
+            r += 1
+
+    # At least one of the trees has ended. Report names from the other tree.
+    while l < len(left_names):
+        exitCode = 1
+        printOnlyIn(left_base, left_tree[0], left_names[l])
+        l += 1
+    while r < len(right_names):
+        exitCode = 1
+        printOnlyIn(right_base, right_tree[0], right_names[r])
+        r += 1
+    return exitCode
+
+def main(argv):
+    args = argv[1:]
+    try:
+        opts, args = getopt.gnu_getopt(args, "wbur", ["strip-trailing-cr"])
+    except getopt.GetoptError as err:
+        sys.stderr.write("Unsupported: 'diff':  %s\n" % str(err))
+        sys.exit(1)
+
+    flags = DiffFlags()
+    filelines, filepaths, dir_trees = ([] for i in range(3))
+    for o, a in opts:
+        if o == "-w":
+            flags.ignore_all_space = True
+        elif o == "-b":
+            flags.ignore_space_change = True
+        elif o == "-u":
+            flags.unified_diff = True
+        elif o == "-r":
+            flags.recursive_diff = True
+        elif o == "--strip-trailing-cr":
+            flags.strip_trailing_cr = True
+        else:
+            assert False, "unhandled option"
+
+    if len(args) != 2:
+        sys.stderr.write("Error: missing or extra operand")
+        sys.exit(1)
+
+    exitCode = 0
+    try:
+        for file in args:
+            if file != "-" and not os.path.isabs(file):
+                file = os.path.realpath(os.path.join(os.getcwd(), file))
+
+            if flags.recursive_diff:
+                if file == "-":
+                    sys.stderr.write("Error: cannot recursively compare '-'")
+                dir_trees.append(getDirTree(file))
+            else:
+                filepaths.append(file)
+
+        if not flags.recursive_diff:
+            exitCode = compareTwoFiles(flags, filepaths)
+        else:
+            exitCode = compareDirTrees(flags, dir_trees)
+
+    except IOError as err:
+        sys.stderr.write("Error: 'diff' command failed, %s\n" % str(err))
+        exitCode = 1
+
+    sys.exit(exitCode)
+
+if __name__ == "__main__":
+    main(sys.argv)

llvm/utils/lit/tests/Inputs/shtest-shell/diff-error-0.txt

-# Check error on a unsupported diff (cannot be part of a pipeline).
-#
-# RUN: diff diff-error-0.txt diff-error-0.txt | echo Output

llvm/utils/lit/tests/Inputs/shtest-shell/diff-pipes-0.txt

+# FIXME: When "not" is correctly supported on "diff", use it here so you can
+# merge diff-pipes-*.txt tests into one test.
+
+# RUN: echo foo > %t.foo
+# RUN: echo bar > %t.bar
+
+# Check output pipe.
+# RUN: diff %t.foo %t.foo | FileCheck -allow-empty -check-prefix=EMPTY %s
+# RUN: diff -u %t.foo %t.bar | FileCheck %s
+
+# CHECK:      @@
+# CHECK-NEXT: -foo
+# CHECK-NEXT: +bar
+# EMPTY-NOT: {{.}}

llvm/utils/lit/tests/Inputs/shtest-shell/diff-pipes-1.txt

+# FIXME: When "not" is correctly supported on "diff", use it here so you can
+# merge diff-pipes-*.txt tests into one test.
+
+# RUN: echo foo > %t.foo
+
+# Check input pipe.
+# RUN: echo foo | diff -u - %t.foo
+# RUN: echo foo | diff -u %t.foo -
+# RUN: echo bar | diff -u %t.foo -

llvm/utils/lit/tests/Inputs/shtest-shell/diff-pipes-2.txt

+# FIXME: When "not" is correctly supported on "diff", use it here so you can
+# merge diff-pipes-*.txt tests into one test.
+
+# RUN: echo foo > %t.foo
+
+# Check input pipe.
+# RUN: echo bar | diff -u - %t.foo

llvm/utils/lit/tests/Inputs/shtest-shell/diff-pipes-3.txt

+# FIXME: When "not" is correctly supported on "diff", use it here so you can
+# merge diff-pipes-*.txt tests into one test.
+
+# RUN: echo foo > %t.foo
+
+# Check output and input pipes at the same time.
+# RUN: echo foo | diff - %t.foo | FileCheck -allow-empty -check-prefix=EMPTY %s
+# RUN: echo bar | diff -u %t.foo - | FileCheck %s
+
+# CHECK:      @@
+# CHECK-NEXT: -foo
+# CHECK-NEXT: +bar
+# EMPTY-NOT: {{.}}

llvm/utils/lit/tests/Inputs/shtest-shell/diff-r-error-7.txt

+# diff -r currently cannot handle stdin.
+# RUN: diff -r - %t

llvm/utils/lit/tests/Inputs/shtest-shell/diff-r-error-8.txt

+# diff -r currently cannot handle stdin.
+# RUN: diff -r %t -

llvm/utils/lit/tests/max-failures.py

 # Check the behavior of --max-failures option.
 #
 # RUN: not %{lit} -j 1 -v %{inputs}/max-failures > %t.out
 # RUN: not %{lit} --max-failures=1 -j 1 -v %{inputs}/max-failures >> %t.out
 # RUN: not %{lit} --max-failures=2 -j 1 -v %{inputs}/max-failures >> %t.out
 # RUN: not %{lit} --max-failures=0 -j 1 -v %{inputs}/max-failures 2>> %t.out
 # RUN: FileCheck < %t.out %s
 #
 # END.
 
-# CHECK: Failing Tests (27)
+# CHECK: Failing Tests (32)
 # CHECK: Failing Tests (1)
 # CHECK: Failing Tests (2)
 # CHECK: error: Option '--max-failures' requires positive integer

llvm/utils/lit/tests/shtest-shell.py

 # Check the internal shell handling component of the ShTest format.
 #
 # RUN: not %{lit} -j 1 -v %{inputs}/shtest-shell > %t.out
 # FIXME: Temporarily dump test output so we can debug failing tests on
 # buildbots.
 # RUN: cat %t.out
-# RUN: FileCheck --input-file %t.out %s
+# RUN: FileCheck -dump-input=fail -vv -color --input-file %t.out %s
 #
 # END.
 
 # CHECK: -- Testing:
 
 # CHECK: FAIL: shtest-shell :: cat-error-0.txt
 # CHECK: *** TEST 'shtest-shell :: cat-error-0.txt' FAILED ***
 # CHECK: $ "cat" "-b" "temp1.txt"

 # CHECK: FAIL: shtest-shell :: colon-error.txt
 # CHECK: *** TEST 'shtest-shell :: colon-error.txt' FAILED ***
 # CHECK: $ ":"
 # CHECK: # command stderr:
 # CHECK: Unsupported: ':' cannot be part of a pipeline
 # CHECK: error: command failed with exit status: 127
 # CHECK: ***
 
-# CHECK: FAIL: shtest-shell :: diff-error-0.txt
-# CHECK: *** TEST 'shtest-shell :: diff-error-0.txt' FAILED ***
-# CHECK: $ "diff" "diff-error-0.txt" "diff-error-0.txt"
-# CHECK: # command stderr:
-# CHECK: Unsupported: 'diff' cannot be part of a pipeline
-# CHECK: error: command failed with exit status: 127
-# CHECK: ***
-
 # CHECK: FAIL: shtest-shell :: diff-error-1.txt
 # CHECK: *** TEST 'shtest-shell :: diff-error-1.txt' FAILED ***
 # CHECK: $ "diff" "-B" "temp1.txt" "temp2.txt"
 # CHECK: # command stderr:
 # CHECK: Unsupported: 'diff': option -B not recognized
-# CHECK: error: command failed with exit status: 127
+# CHECK: error: command failed with exit status: 1
 # CHECK: ***
 
 # CHECK: FAIL: shtest-shell :: diff-error-2.txt
 # CHECK: *** TEST 'shtest-shell :: diff-error-2.txt' FAILED ***
 # CHECK: $ "diff" "temp.txt"
 # CHECK: # command stderr:
-# CHECK: Error:  missing or extra operand
-# CHECK: error: command failed with exit status: 127
+# CHECK: Error: missing or extra operand
+# CHECK: error: command failed with exit status: 1
 # CHECK: ***
 
 # CHECK: FAIL: shtest-shell :: diff-error-3.txt
 # CHECK: *** TEST 'shtest-shell :: diff-error-3.txt' FAILED ***
 # CHECK: $ "diff" "temp.txt" "temp1.txt"
 # CHECK: # command stderr:
 # CHECK: Error: 'diff' command failed
 # CHECK: error: command failed with exit status: 1

 # CHECK: --- 1 ----
 # CHECK: ! hello-second
 # CHECK: ***
 
 # CHECK: FAIL: shtest-shell :: diff-error-5.txt
 # CHECK: *** TEST 'shtest-shell :: diff-error-5.txt' FAILED ***
 # CHECK: $ "diff"
 # CHECK: # command stderr:
-# CHECK: Error:  missing or extra operand
-# CHECK: error: command failed with exit status: 127
+# CHECK: Error: missing or extra operand
+# CHECK: error: command failed with exit status: 1
 # CHECK: ***
 
 # CHECK: FAIL: shtest-shell :: diff-error-6.txt
 # CHECK: *** TEST 'shtest-shell :: diff-error-6.txt' FAILED ***
 # CHECK: $ "diff"
 # CHECK: # command stderr:
-# CHECK: Error:  missing or extra operand
-# CHECK: error: command failed with exit status: 127
+# CHECK: Error: missing or extra operand
+# CHECK: error: command failed with exit status: 1
+# CHECK: ***
+
+# CHECK: FAIL: shtest-shell :: diff-pipes-0.txt
+# CHECK: *** TEST 'shtest-shell :: diff-pipes-0.txt' FAILED ***
+# CHECK: $ "diff" "{{[^"]*}}.foo" "{{[^"]*}}.foo"
+# CHECK-NOT: note
+# CHECK-NOT: error
+# CHECK: $ "FileCheck"
+# CHECK-NOT: note
+# CHECK-NOT: error
+# CHECK: $ "diff" "-u" "{{[^"]*}}.foo" "{{[^"]*}}.bar"
+# CHECK: note: command had no output on stdout or stderr
+# CHECK: error: command failed with exit status: 1
+# CHECK: $ "FileCheck"
+# CHECK-NOT: note
+# CHECK-NOT: error
+# CHECK: ***
+
+# CHECK: FAIL: shtest-shell :: diff-pipes-1.txt
+# CHECK: *** TEST 'shtest-shell :: diff-pipes-1.txt' FAILED ***
+# CHECK: $ "echo" "foo"
+# CHECK: $ "diff" "-u" "-" "{{[^"]*}}.foo"
+# CHECK-NOT: note
+# CHECK-NOT: error
+# CHECK: $ "echo" "foo"
+# CHECK: $ "diff" "-u" "{{[^"]*}}.foo" "-"
+# CHECK-NOT: note
+# CHECK-NOT: error
+# CHECK: $ "echo" "bar"
+# CHECK: $ "diff" "-u" "{{[^"]*}}.foo" "-"
+# CHECK: # command output:
+# CHECK:      @@
+# CHECK-NEXT: -foo
+# CHECK-NEXT: +bar
+# CHECK: error: command failed with exit status: 1
+# CHECK: ***
+
+# CHECK: FAIL: shtest-shell :: diff-pipes-2.txt
+# CHECK: *** TEST 'shtest-shell :: diff-pipes-2.txt' FAILED ***
+# CHECK: $ "echo" "bar"
+# CHECK: $ "diff" "-u" "-" "{{[^"]*}}.foo"
+# CHECK: # command output:
+# CHECK:      @@
+# CHECK-NEXT: -bar
+# CHECK-NEXT: +foo
+# CHECK: error: command failed with exit status: 1
+# CHECK: ***
+
+# CHECK: FAIL: shtest-shell :: diff-pipes-3.txt
+# CHECK: *** TEST 'shtest-shell :: diff-pipes-3.txt' FAILED ***
+# CHECK: $ "echo" "foo"
+# CHECK: $ "diff" "-" "{{[^"]*}}.foo"
+# CHECK-NOT: note
+# CHECK-NOT: error
+# CHECK: $ "FileCheck"
+# CHECK-NOT: note
+# CHECK-NOT: error
+# CHECK: $ "echo" "bar"
+# CHECK: $ "diff" "-u" "{{[^"]*}}.foo" "-"
+# CHECK: note: command had no output on stdout or stderr
+# CHECK: error: command failed with exit status: 1
+# CHECK: $ "FileCheck"
+# CHECK-NOT: note
+# CHECK-NOT: error
 # CHECK: ***
 
 # CHECK: FAIL: shtest-shell :: diff-r-error-0.txt
 # CHECK: *** TEST 'shtest-shell :: diff-r-error-0.txt' FAILED ***
 # CHECK: $ "diff" "-r" 
 # CHECK: # command output:
 # CHECK: Only in {{.*}}dir1: dir1unique
 # CHECK: Only in {{.*}}dir2: dir2unique

 
 # CHECK: FAIL: shtest-shell :: diff-r-error-6.txt
 # CHECK: *** TEST 'shtest-shell :: diff-r-error-6.txt' FAILED ***
 # CHECK: $ "diff" "-r" 
 # CHECK: # command output:
 # CHECK: File {{.*}}dir1{{.*}}extra_file is a regular empty file while file {{.*}}dir2{{.*}}extra_file is a directory
 # CHECK: error: command failed with exit status: 1
 
+# CHECK: FAIL: shtest-shell :: diff-r-error-7.txt
+# CHECK: *** TEST 'shtest-shell :: diff-r-error-7.txt' FAILED ***
+# CHECK: $ "diff" "-r" "-" "{{[^"]*}}"
+# CHECK: # command stderr:
+# CHECK: Error: cannot recursively compare '-'
+# CHECK: error: command failed with exit status: 1
+
+# CHECK: FAIL: shtest-shell :: diff-r-error-8.txt
+# CHECK: *** TEST 'shtest-shell :: diff-r-error-8.txt' FAILED ***
+# CHECK: $ "diff" "-r" "{{[^"]*}}" "-"
+# CHECK: # command stderr:
+# CHECK: Error: cannot recursively compare '-'
+# CHECK: error: command failed with exit status: 1
+
 # CHECK: PASS: shtest-shell :: diff-r.txt
 
 # CHECK: FAIL: shtest-shell :: error-0.txt
 # CHECK: *** TEST 'shtest-shell :: error-0.txt' FAILED ***
 # CHECK: $ "not-a-real-command"
 # CHECK: # command stderr:
 # CHECK: 'not-a-real-command': command not found
 # CHECK: error: command failed with exit status: 127

 # CHECK: *** TEST 'shtest-shell :: rm-error-3.txt' FAILED ***
 # CHECK: Exit Code: 1
 # CHECK: ***
 
 # CHECK: PASS: shtest-shell :: rm-unicode-0.txt
 # CHECK: PASS: shtest-shell :: sequencing-0.txt
 # CHECK: XFAIL: shtest-shell :: sequencing-1.txt
 # CHECK: PASS: shtest-shell :: valid-shell.txt
-# CHECK: Failing Tests (27)
+# CHECK: Failing Tests (32)