[lit] Improve lit's output with default settings and --verbose.

[varungandhi-apple]

Summary

• Before:
  • default (no flags): no script, no command output
  • -v: full script, no command output
  • -vv: full script, adds 'set +x', shows command output
• After:
  • default (no flags): no script, show only failing line in command output
  • -v: full script, adds 'set +x', shows command output
  • -vv: alias to -v (preserve backwards compatibility)

[varungandhi-apple]

Include missing commit separating verbose and showOutput.

[varungandhi-apple]

• [docs] [lit] Add a more helpful description for lit.py's -s flag.
• [NFC] [lit] Separate verbose and showOutput.
• [lit] Improve lit's output with default settings and --verbose.

[varungandhi-apple]

Rebase after changes in D82808.

[yln]

The changes here are for text printed for failing tests, right?
And showAllOutput is orthogonal and just shows everything even for non-failing tests?

llvm/utils/lit/lit/OutputSettings.py

35

I really like the "communicating intent" part of this infrastructure. However, this is a lot of code and ifs considering we really only have to distinguish two cases. From your commit message:

• default (no flags): no script, show only failing line in command output
• -v: full script, adds 'set +x', shows command output

Am I missing something or could everything be keyed off a single verbose (reading from the code below showAllOutput implies verbose) flag. Do we anticipate other combinations being useful? (In general I would argue for striving for the simplest implementation that gives us what we currently want and not try to anticipate future extensions.)

Have you considered (or started out with) just using a single verbose flag to base your decisions in the implementation functions?

llvm/utils/lit/lit/TestRunner.py

1499

I feel like this function is the most complex part of this patch. I don't fully follow it, but you experimented and believe this is the best approach and wrote a whole test suite so I am happy with it.

1503–1508

Why use a complicated parser-like return value? Our only caller below could just receive the potentially found RUN line.

1593

Why are we using two local functions here?

The whole thing could be (already assuming just one verbose flag):

def make_script_output(lit_config, script_lines, exit_code):
    if not lit_config.verbose:
        return ""
    return ...

1614–1617

This block should be pushed into the lit_config.command_output_style == OutputSettings.ONLY_FAILING_COMMAND case, otherwise we are always searching for the last line, even if we don't really use the result of the computation.

Also: if we fail to find the RUN line, we might print the note to use --verbose even if the user already specified it.

1624

Please try to simplify this a bit, maybe the following?

def make_command_output():
    if not lit_config.quiet:
        return ""
    
    verbose = lit_config.verbose
    output = "header {} ...".format(", truncated" if verbose else "")
    if verbose:
        output += cmd_output
    else:
        run_line = locate_last_run_line() # then deal with "not found" case
        output += ...
        output += "Note:  try to use --verbose"

     return output

llvm/utils/lit/lit/cl_arguments.py

56

I think aliases kwarg is something else (seems to be related to subparsers). You could just add "-vv", "--echo-all-commands" after "--verbose" to allow for additional names for the flag, but I think I prefer to have it separate (makes it easer to remove it if we ever decide to drop the alias in the future).

So long comment short: just drop this comment please.

177

Unconditionally overwritten below

194

opts.echo_all_commands = opts.command_output_style in {OutputSettings.UP_TO_AND_INCLUDING_FAILING_COMMAND, ...}

llvm/utils/lit/tests/shtest-run-at-line.py

70

I really like the extension to this test. Thanks!

llvm/utils/lit/tests/unittest-failing-locator.py

123

Thanks for being diligent and adding these tests. This will also make it easy to add additional tests for corner cases if we need them in the future!

[varungandhi-apple]

Thanks for the review, it's a big patch. 😅 I'm a bit busy at the moment, I will respond to the other comments later this week or sometime next week.

[varungandhi-apple]

Address review comments.

[varungandhi-apple]

Some of the changes probably don't make perfect sense in this revision because I initially started out with the whole change being one revision and then split things into commits, while trying to minimize the churn between revisions. Please also take a look at the follow-up revision D82811 and let me know if that's acceptable (less churn, less self-contained, things make sense when you look at the two together) or if you'd prefer that I make this revision more self-contained (that would create more churn in D82811).

llvm/utils/lit/lit/OutputSettings.py

35

Not sure what you mean by two cases, I think there are three cases:

1. Quiet -> Corresponds on NO_COMMAND
2. Default (no quiet, no verbose) -> Corresponds to ONLY_FAILING.
3. Verbose -> Corresponds to UP_TO_AND_INCLUDING_FAILING.

Since there is a 1-1 correspondence, we _could_ compare quiet vs (!quiet && !verbose) vs verbose in make_command_output. I chose not to do that because I figured this makes the intent clearer by distinguishing between user-specified options and derived options.

Does that make sense? Would you still prefer that I get rid of this?

llvm/utils/lit/lit/TestRunner.py

1503–1508

Sorry, it's not super clear because in this revision, there is only one caller, which could do with just using the index. However, the highlighting patch (which comes right after this logically) ends up making use of the substring. The line of reasoning is:

1. A substring by itself is not good enough (and hence make_command_output in this patch makes use of the first index return value), because we want to distinguish the case line_start == 0 from the case line_start > 0 by printing Command Output (stderr) instead of Command Output (stderr, truncated) when line_start == 0.
2. An index by itself could be "good enough", but we try to be smarter in the highlighting patch (https://reviews.llvm.org/D82811) by not highlighting the shell-specific prefix, and starting highlighting from the "RUN", which make_script_output makes use of.

Does that make sense? I decided against splitting changes to this function into two revisions and have the full functionality in the first iteration because I felt like that would create more churn with the tests and the implementation, but I see how it can be confusing as to why two related-but-slightly-different-values are being returned simultaneously when only one is really being used in this patch.

1593

It mimics the structure of the make_command_output function right below. Right now, this function is very simple, so it probably seems overkill. However, make_script_output becomes much more complicated in the highlighting patch after this: https://reviews.llvm.org/D82811, so I figured it would be helpful to have the two functions make_script_output and make_command_output follow a similar pattern.

1614–1617

This block should be pushed into the lit_config.command_output_style == OutputSettings.ONLY_FAILING_COMMAND case, otherwise we are always searching for the last line, even if we don't really use the result of the computation.

This part is actually used by the highlighting patch (https://reviews.llvm.org/D82811), where the last line of this function becomes:

failing_lines = highlight_failure_lines(cmd_output[line_start:])
return make_output(cmd_output[:line_start] + failing_lines,
                   is_truncated=False)

That's why I didn't push it into the conditional branch.

Also: if we fail to find the RUN line, we might print the note to use --verbose even if the user already specified it.

Good catch, I've fixed make_output to check for is_truncated properly.

[yln]

llvm/docs/CommandGuide/lit.rst

102

blank line

llvm/utils/lit/lit/OutputSettings.py

35

Okay, I agree now that having an enum-like thing is really nice because we can give it a good explanatory name.

Can we roll the following options into one enum-like OutputStyle.XXX corresponding to our 3 cases: "quiet, default/only_failing, verbose/up_to_first_failing"; since the user's can't configure them individually.

I imagine the following things
opts.show_output_on_failure, opts.script_output_style, opts.command_output_style, and opts.quiet being replaced with opts.output_style

llvm/utils/lit/lit/TestRunner.py

1503–1508

Yes, please implement it in the "simplest way possible" for the current functionality without anticipating future requirements. We can always make it more complex if required for a future feature, but's it's harder to "see" that there is opportunity for simplification once the code is there.

Also, another reviewer might suggest a solution you haven't thought about.

For example, could highlighting be implemented via (without worrying about indexes)?

s = get_important_bits(full_output)
s_highlighted = '<highlight>{}</highlight>'.format(s)
highlighted_output = full_output.replace(s, s_highlighted)

1593

I like symmetry in code, but please simplify this as much as possible.

llvm/utils/lit/lit/cl_arguments.py

194

Or even simpler (the 1 out of 3 cases that doesn't set it to true)?

opts.echo_all_commands = opts.output_style != OutputStyle.QUIET

llvm/docs/CommandGuide/lit.rst

 
 Each specified test will be executed (potentially in parallel) and once all
 tests have been run :program:`lit` will print summary information on the number
 of tests which passed or failed (see :ref:`test-status-results`).  The
 :program:`lit` program will execute with a non-zero exit code if any tests
 fail.
 
 By default :program:`lit` will use a succinct progress display and will only
-print summary information for test failures.  See :ref:`output-options` for
-options controlling the :program:`lit` progress display and output.
+print summary information for test failures, such as which exact line failed.
+See :ref:`output-options` for options controlling the :program:`lit`
+progress display and output.
 
 :program:`lit` also includes a number of options for controlling how tests are
 executed (specific features may depend on the particular test format).  See
 :ref:`execution-options` for more information.
 
 Finally, :program:`lit` also supports additional options for only running a
 subset of the options specified on the command line, see
 :ref:`selection-options` for more information.

  Show less output, for example don't show information on tests that pass.
  Also show a progress bar, unless ``--no-progress-bar`` is specificed.
 
 .. option:: -v, --verbose
 
  Show more information on test failures, for example the entire test output
  instead of just the test result.
 
-.. option:: -vv, --echo-all-commands
-
  Echo all commands to stdout, as they are being executed.
  This can be valuable for debugging test failures, as the last echoed command
  will be the one which has failed.
  :program:`lit` normally inserts a no-op command (``:`` in the case of bash)
  with argument ``'RUN: at line N'`` before each command pipeline, and this
  option also causes those no-op commands to be echoed to stdout to help you
  locate the source line of the failed command.
- This option implies ``--verbose``.
 
+.. option:: -vv, --echo-all-commands
+
+ Alias for ``-v``/``--verbose`` (for backwards compatibility).
 .. option:: -a, --show-all
 
  Show more information about all tests, for example the entire test
  commandline and output.
 
 .. option:: --no-progress-bar
 
  Do not use curses based progress bar.

llvm/utils/lit/lit/LitConfig.py

 from __future__ import absolute_import
 import inspect
 import os
 import platform
 import sys
 
 import lit.Test
 import lit.formats
+import lit.OutputSettings as OutputSettings
 import lit.TestingConfig
 import lit.util
 
 # LitConfig must be a new style class for properties to work
 class LitConfig(object):
     """LitConfig - Configuration data for a 'lit' test runner instance, shared
     across all tests.
 
     The LitConfig object is also used to communicate with client configuration
     files, it is always passed in as the global variable 'lit' so that
     configuration files can access common functionality and internal components
     easily.
     """
 
     def __init__(self, progname, path, quiet,
                  useValgrind, valgrindLeakCheck, valgrindArgs,
                  noExecute, debug, isWindows,
                  params, config_prefix = None,
                  maxIndividualTestTime = 0,
                  parallelism_groups = {},
-                 echo_all_commands = False):
+                 echo_all_commands=False,
+                 script_output_style=OutputSettings.NO_SCRIPT,
+                 command_output_style=OutputSettings.ONLY_FAILING_COMMAND):
         # The name of the test runner.
         self.progname = progname
         # The items to add to the PATH environment variable.
         self.path = [str(p) for p in path]
         self.quiet = bool(quiet)
         self.useValgrind = bool(useValgrind)
         self.valgrindLeakCheck = bool(valgrindLeakCheck)
         self.valgrindUserArgs = list(valgrindArgs)
def __init__(self, progname, path, quiet,
             else:
                 # The default is 'summary'.
                 self.valgrindArgs.append('--leak-check=no')
             self.valgrindArgs.extend(self.valgrindUserArgs)
 
         self.maxIndividualTestTime = maxIndividualTestTime
         self.parallelism_groups = parallelism_groups
         self.echo_all_commands = echo_all_commands
+        self.script_output_style = script_output_style
+        self.command_output_style = command_output_style
 
     @property
     def maxIndividualTestTime(self):
         """
             Interface for getting maximum time to spend executing
             a single test
         """
         return self._maxIndividualTestTime

llvm/utils/lit/lit/OutputSettings.py

+# TODO(python3): Could be an enum
+class ScriptOutputStyle:
+    def __init__(self, name):
+        self.name = name
+
+    # For print-debugging.
+    def __str__(self):
+        return '%s(%r)' % (self.__class__.__name__, self.name)
+
+NO_SCRIPT = ScriptOutputStyle("None")
+FULL_SCRIPT = ScriptOutputStyle("Full")
+
+# Consider a script
+#
+# ```
+# RUN: true
+# RUN: false
+# RUN: true
+# ```
+#
+# FULL_SCRIPT will show the whole script.
+
+# TODO(python3): Could be an enum
+class CommandOutputStyle:
+    def __init__(self, name):
+        self.name = name
+
+    # For print-debugging.
+    def __str__(self):
+        return '%s(%r)' % (self.__class__.__name__, self.name)
+
+NO_COMMAND = CommandOutputStyle("None")
+ONLY_FAILING_COMMAND = CommandOutputStyle("OnlyFailing")
+UP_TO_AND_INCLUDING_FAILING_COMMAND = CommandOutputStyle("UpToAndIncluding")
+
+# Consider a script
+#
+# ```
+# RUN: true
+# RUN: false
+# RUN: true
+# ```
+#
+# ONLY_FAILING_COMMAND will show something like
+#
+# ```
+# Command output (stderr, truncated)
+# --
+# + : 'RUN: at line 2'
+# + false
+# --
+# NOTE: The failure might depend on preceding RUN lines.
+#       Use --verbose to see preceding RUN lines and outputs.
+# ```
+#
+# UP_TO_AND_INCLUDING_FAILING_COMMAND will show something like
+#
+# ```
+# Command Output (stderr):
+# --
+# + : 'RUN: at line 1'
+# + true
+# + : 'RUN: at line 2'
+# + false
+# --
+# ```

llvm/utils/lit/lit/TestRunner.py

 import threading
 
 import io
 try:
     from StringIO import StringIO
 except ImportError:
     from io import StringIO
 
+import lit.OutputSettings as OutputSettings
 from lit.ShCommands import GlobItem, Command
 import lit.ShUtil as ShUtil
 import lit.Test as Test
 import lit.util
 from lit.util import to_bytes, to_string, to_unicode
 from lit.BooleanExpression import BooleanExpression
 
 class InternalShellError(Exception):
def parseIntegratedTestScript(test, additional_parsers=[],
     if not test.isWithinFeatureLimits():
         msg = ', '.join(test.config.limit_to_features)
         return lit.Test.Result(Test.UNSUPPORTED,
                                "Test does not require any of the features "
                                "specified in limit_to_features: %s" % msg)
 
     return script
 
+def locate_last_run_line(output_str):
+    """
+    Try to locate the last run line in ``output_str`` using heuristics.
+
+    Returns a pair of:
+    - The index in ``output_str`` pointing to immediately after the preceding
+      newline, i.e. the start of the RUN line, before any shell-specific
+      prefix.
+    - The matched substring itself, including the number at the end,
+      starting with 'RUN', skipping the shell-specific prefix.
+
+    For example, if ``output_str`` is ``"+ HI\n+ RUN: at line 10"``,
+    the index will be 5 and the substring will be ``"RUN: at line 10"``.
+
+    Returns (-1, None) on failure.
+    """
+    # We try to be clever; rather than directly finding the last
+    # 'RUN: at line', we try to search from the beginning, create a regex,
+    # which hopefully takes into account any shell-specific prefix,
+    # and then uses that regex to search from the end.
+    # Searching backwards directly can lead to erroneous reporting
+    # when the command output itself has a 'RUN: at line' due to something
+    # else (possibly because we are testing ``lit`` itself).
+    #
+    # However, this is not perfect either :(. It will fail if a shell emits
+    # varying prefixes for commands on 'set +x', such as line numbers or
+    # timestamps. So we try to fallback to the simpler technique if applicable.
+
+    error = (-1, None)
+    run_str = "RUN: at line"
+    first_run_str_start = output_str.find(run_str)
+    if first_run_str_start == -1:
+        return error
+    # Use the first RUN line to create a regex for searching.
+    newline_index = output_str[:first_run_str_start].rfind('\n')
+    first_run_line_start = 0 if newline_index == -1 else newline_index + 1
+    first_run_str_end = first_run_str_start + len(run_str)
+    run_line_regex = (
+        '('
+        + re.escape(output_str[first_run_line_start:first_run_str_end])
+        + ' [0-9]+)'
+    )
+
+    run_lines = list(re.finditer(run_line_regex, output_str))
+    if len(run_lines) == 0:
+        # impossible; we should've at least gotten the first run line...
+        return error
+
+    if len(run_lines) == 1:
+        # We must've gotten our original string. That's not helpful. :-/
+        # Try using the simple strategy to make things work with shells that
+        # use a varying prefix for 'set +x'.
+        # (N.B. I don't know of any shells that actually do this, but it's
+        # not inconceivable that a CI machine would, say, print out timestamps
+        # with 'set +x')
+        simple_run_line_regex = '(RUN: at line [0-9]+)'
+        simple_run_lines = list(re.finditer(simple_run_line_regex, output_str))
+        if len(simple_run_lines) == 0:
+            # impossible; we should've at least gotten the first run line...
+            return error
+        elif len(simple_run_lines) == 1:
+            # we must've hit the line we found earlier, don't try anything
+            pass
+        else:
+            # Assume that we have a shell with a varying prefix and use
+            # the results of the simple matching.
+            # However, this creates a problem when:
+            # 1. The very first RUN line failed and it has 'RUN: at line'.
+            # 2. A shell has varying prefixes and the output of the last command
+            #    has 'RUN: at line'.
+            # Can't really help these situations. :(
+            last_run_line = simple_run_lines[-1]
+            newline_index = output_str[:last_run_line.start()].rfind('\n')
+            start = 0 if newline_index == -1 else (newline_index + 1)
+            return (start, last_run_line.group(0))
+
+    last_run_line = run_lines[-1]
+    # trim the shell-specific prefix
+    delta = first_run_str_start - first_run_line_start
+    last_run_line_str = last_run_line.group(0)[delta:]
+    return (last_run_line.start(), last_run_line_str)
+
+def make_script_output(lit_config, script_lines, exit_code):
+    def make_output(script_display_lines):
+        return ("""Script:\n--\n%s\n--\nExit Code: %d\n"""
+                % ('\n'.join(script_display_lines), exit_code))
+
+    def default_output():
+        return make_output(script_lines)
+
+    if lit_config.script_output_style == OutputSettings.NO_SCRIPT:
+        return ""
+
+    assert(lit_config.script_output_style == OutputSettings.FULL_SCRIPT)
+    return default_output()
+
+def make_command_output(lit_config, cmd_output, stream_name, test_status):
+    def make_output(output_str, is_truncated=False):
+        if is_truncated:
+            format_str = """Command Output (%s, truncated):\n--\n%s\n--\n"""
+            format_str += (
+                "(NOTE: The failure may depend on previous RUN lines.\n"
+                "       Use --verbose to see previous RUN lines and outputs.)\n"
+            )
+        else:
+            format_str = """Command Output (%s):\n--\n%s\n--\n"""
+        return format_str % (stream_name, output_str)
+
+    def default_output():
+        return make_output(cmd_output)
+
+    if lit_config.command_output_style == OutputSettings.NO_COMMAND:
+        return ""
+    elif not test_status.isFailure:
+        return default_output()
+
+    line_start, run_line_str = locate_last_run_line(cmd_output)
+    # maybe there was an error, or maybe we are not truncating anything
+    if run_line_str is None or line_start == 0:
+        return default_output()
+
+    if lit_config.command_output_style == OutputSettings.ONLY_FAILING_COMMAND:
+        return make_output(cmd_output[line_start:], is_truncated=True)
+
+    assert(lit_config.command_output_style
+           == OutputSettings.UP_TO_AND_INCLUDING_FAILING_COMMAND)
+    return make_output(cmd_output, is_truncated=False)
 
 def _runShTest(test, litConfig, useExternalSh, script, tmpBase):
     def runOnce(execdir):
         if useExternalSh:
             res = executeScript(test, litConfig, tmpBase, script, execdir)
         else:
             res = executeScriptInternal(test, litConfig, tmpBase, script, execdir)
         if isinstance(res, lit.Test.Result):
for i in range(attempts):
             break
 
     # If we had to run the test more than once, count it as a flaky pass. These
     # will be printed separately in the test summary.
     if i > 0 and status == Test.PASS:
         status = Test.FLAKYPASS
 
     # Form the output log.
-    output = """Script:\n--\n%s\n--\nExit Code: %d\n""" % (
-        '\n'.join(script), exitCode)
+    output = make_script_output(litConfig, script, exitCode)
 
     if timeoutInfo is not None:
         output += """Timeout: %s\n""" % (timeoutInfo,)
     output += "\n"
 
     # Append the outputs, if present.
     if out:
-        output += """Command Output (stdout):\n--\n%s\n--\n""" % (out,)
+        output += make_command_output(litConfig, out, "stdout", status)
     if err:
-        output += """Command Output (stderr):\n--\n%s\n--\n""" % (err,)
+        output += make_command_output(litConfig, err, "stderr", status)
 
     return lit.Test.Result(status, output)
 
 
 def executeShTest(test, litConfig, useExternalSh,
                   extra_substitutions=[],
                   preamble_commands=[]):
     if test.config.unsupported:

llvm/utils/lit/lit/cl_arguments.py

 import argparse
 import os
 import shlex
 import sys
 
+import lit.OutputSettings as OutputSettings
 import lit.reports
 import lit.util
 
 
 def parse_args():
     parser = argparse.ArgumentParser(prog='lit')
     parser.add_argument('test_paths',
             nargs='+',
format_group.add_argument("-q", "--quiet",
             help="Suppress no error output",
             action="store_true")
     format_group.add_argument("-s", "--succinct",
             help="Reduce amount of output."
                  " Additionally, show a progress bar,"
                  " unless --no-progress-bar is specified.",
             action="store_true")
     format_group.add_argument("-v", "--verbose",
-            dest="showOutput",
-            help="Show test output for failures",
+            help="Show test output for failures."
+                 " The last command under 'Output' is the failing one."
+                 " All substituted commands are shown under 'Script',"
+                 " including any commands after the failing command.",
             action="store_true")
     format_group.add_argument("-vv", "--echo-all-commands",
-            dest="echoAllCommands",
-            action="store_true",
-            help="Echo all commands as they are executed to stdout. In case of "
-                 "failure, last command shown will be the failing one.")
+            help="Alias for --verbose.",
+            dest="verbose",
+            action="store_true")
     format_group.add_argument("-a", "--show-all",
             dest="showAllOutput",
             help="Display all commandlines and output",
             action="store_true")
     format_group.add_argument("-o", "--output",
             type=lit.reports.JsonReport,
             help="Write test results to the provided path",
             metavar="PATH")
debug_group.add_argument("--show-used-features",
             help="Show all features used in the test suite (in XFAIL, UNSUPPORTED and REQUIRES) and exit",
             action="store_true")
 
     # LIT is special: environment variables override command line arguments.
     env_args = shlex.split(os.environ.get("LIT_OPTS", ""))
     args = sys.argv[1:] + env_args
     opts = parser.parse_args(args)
 
-    # Validate command line options
-    if opts.echoAllCommands:
-        opts.showOutput = True
+    # default output settings
+    opts.show_output_on_failure = True
+    opts.script_output_style = OutputSettings.NO_SCRIPT
+    opts.command_output_style = OutputSettings.ONLY_FAILING_COMMAND
+
+    if opts.quiet:
+        opts.show_output_on_failure = False
+        opts.command_output_style = OutputSettings.NO_COMMAND
+    elif opts.verbose or opts.showAllOutput:
+        opts.script_output_style = OutputSettings.FULL_SCRIPT
+        opts.command_output_style = (
+            OutputSettings.UP_TO_AND_INCLUDING_FAILING_COMMAND
+        )
+
+    # If we call out to an external shell, we ask for all output,
+    # regardless of whether we're going to display all commands that were run,
+    # or only the last command which failed.
+    cmd_output_style = opts.command_output_style
+    opts.echo_all_commands = (
+        cmd_output_style in {
+            OutputSettings.UP_TO_AND_INCLUDING_FAILING_COMMAND,
+            OutputSettings.ONLY_FAILING_COMMAND
+        }
+    )
 
     # TODO(python3): Could be enum
     if opts.shuffle:
         opts.order = 'random'
     elif opts.incremental:
         opts.order = 'failing-first'
     else:
         opts.order = 'default'

llvm/utils/lit/lit/display.py

class Display(object):
 
     def print_result(self, test):
         # Show the test result line.
         test_name = test.getFullName()
         print('%s: %s (%d of %d)' % (test.result.code.name, test_name,
                                      self.completed, self.tests))
 
         # Show the test failure output, if requested.
-        if (test.isFailure() and self.opts.showOutput) or \
+        if (test.isFailure() and self.opts.show_output_on_failure) or \
            self.opts.showAllOutput:
             if test.isFailure():
                 print("%s TEST '%s' FAILED %s" % ('*'*20, test.getFullName(),
                                                   '*'*20))
             out = test.result.output
             # Encode/decode so that, when using Python 3.6.5 in Windows 10,
             # print(out) doesn't raise UnicodeEncodeError if out contains
             # special characters.  However, Python 2 might try to decode

llvm/utils/lit/lit/main.py

lit_config = lit.LitConfig.LitConfig(
         useValgrind=opts.useValgrind,
         valgrindLeakCheck=opts.valgrindLeakCheck,
         valgrindArgs=opts.valgrindArgs,
         noExecute=opts.noExecute,
         debug=opts.debug,
         isWindows=is_windows,
         params=params,
         config_prefix=opts.configPrefix,
-        echo_all_commands=opts.echoAllCommands)
+        echo_all_commands=opts.echo_all_commands,
+        script_output_style=opts.script_output_style,
+        command_output_style=opts.command_output_style)
 
     discovered_tests = lit.discovery.find_tests_for_inputs(lit_config, opts.test_paths)
     if not discovered_tests:
         sys.stderr.write('error: did not discover any tests for provided path(s)\n')
         sys.exit(2)
 
     if opts.show_suites or opts.show_tests:
         print_discovered(discovered_tests, opts.show_suites, opts.show_tests)

llvm/utils/lit/tests/shtest-format.py

 # CHECK: FAIL: shtest-format :: external_shell/fail.txt
 # CHECK-NEXT: *** TEST 'shtest-format :: external_shell/fail.txt' FAILED ***
 # CHECK: Command Output (stdout):
 # CHECK-NEXT: --
 # CHECK-NEXT: line 1: failed test output on stdout
 # CHECK-NEXT: line 2: failed test output on stdout
 # CHECK: Command Output (stderr):
 # CHECK-NEXT: --
-# CHECK-NEXT: cat{{(\.exe)?}}: {{cannot open does-not-exist|does-not-exist: No such file or directory}}
+# CHECK-NEXT: 'RUN: at line 3'
+#
+# Skipping some lines...
+#
+# CHECK:      'RUN: at line 5'
+# CHECK:      cat{{(\.exe)?}}: {{cannot open does-not-exist|does-not-exist: No such file or directory}}
 # CHECK: --
 
 # CHECK: FAIL: shtest-format :: external_shell/fail_with_bad_encoding.txt
 # CHECK-NEXT: *** TEST 'shtest-format :: external_shell/fail_with_bad_encoding.txt' FAILED ***
 # CHECK: Command Output (stdout):
 # CHECK-NEXT: --
 # CHECK-NEXT: a line with bad encoding:
 # CHECK: --

llvm/utils/lit/tests/shtest-run-at-line.py

-# Check that -vv makes the line number of the failing RUN command clear.
-# (-v is actually sufficient in the case of the internal shell.)
+# Check that the default terse output shows only the failing
+# line in the output, and doesn't show the script.
 #
-# RUN: not %{lit} -j 1 -vv %{inputs}/shtest-run-at-line > %t.out
-# RUN: FileCheck --input-file %t.out %s
+# RUN: not %{lit} -j 1 %{inputs}/shtest-run-at-line > %t-terse.out
+# RUN: FileCheck --input-file %t-terse.out %s --check-prefix=TERSE
+
+# Check that --verbose outputs both Script and Command Output.
+#
+# RUN: not %{lit} -j 1 --verbose %{inputs}/shtest-run-at-line > %t-verbose.out
+# RUN: FileCheck --input-file %t-verbose.out %s --check-prefix=VERBOSE
 #
 # END.
 
-
-# CHECK: Testing: 4 tests
-
-
 # In the case of the external shell, we check for only RUN lines in stderr in
 # case some shell implementations format "set -x" output differently.
 
-# CHECK-LABEL: FAIL: shtest-run-at-line :: external-shell/basic.txt
+################################################################################
+# Checking lines for terse output
+
+# TERSE-LABEL: FAIL: shtest-run-at-line :: external-shell/basic.txt
+
+# TERSE-NOT:      Script:
 
-# CHECK:      Script:
-# CHECK:      RUN: at line 4{{.*}}  true
-# CHECK-NEXT: RUN: at line 5{{.*}}  false
-# CHECK-NEXT: RUN: at line 6{{.*}}  true
-
-# CHECK:     RUN: at line 4
-# CHECK:     RUN: at line 5
-# CHECK-NOT: RUN
-
-# CHECK-LABEL: FAIL: shtest-run-at-line :: external-shell/line-continuation.txt
-
-# CHECK:      Script:
-# CHECK:      RUN: at line 4{{.*}}  echo 'foo bar'  | FileCheck
-# CHECK-NEXT: RUN: at line 6{{.*}}  echo 'foo baz'  | FileCheck
-# CHECK-NEXT: RUN: at line 9{{.*}}  echo 'foo bar'  | FileCheck
-
-# CHECK:     RUN: at line 4
-# CHECK:     RUN: at line 6
-# CHECK-NOT: RUN
-
-
-# CHECK-LABEL: FAIL: shtest-run-at-line :: internal-shell/basic.txt
-
-# CHECK:      Script:
-# CHECK:      : 'RUN: at line 1';  true
-# CHECK-NEXT: : 'RUN: at line 2';  false
-# CHECK-NEXT: : 'RUN: at line 3';  true
-
-# CHECK:      Command Output (stdout)
-# CHECK:      $ ":" "RUN: at line 1"
-# CHECK-NEXT: $ "true"
-# CHECK-NEXT: $ ":" "RUN: at line 2"
-# CHECK-NEXT: $ "false"
-# CHECK-NOT:  RUN
-
-# CHECK-LABEL: FAIL: shtest-run-at-line :: internal-shell/line-continuation.txt
-
-# CHECK:      Script:
-# CHECK:      : 'RUN: at line 1';  : first line continued to second line
-# CHECK-NEXT: : 'RUN: at line 3';  echo 'foo bar'  | FileCheck
-# CHECK-NEXT: : 'RUN: at line 5';  echo  'foo baz'  | FileCheck
-# CHECK-NEXT: : 'RUN: at line 8';  echo 'foo bar'  | FileCheck
-
-# CHECK:      Command Output (stdout)
-# CHECK:      $ ":" "RUN: at line 1"
-# CHECK-NEXT: $ ":" "first" "line" "continued" "to" "second" "line"
-# CHECK-NEXT: $ ":" "RUN: at line 3"
-# CHECK-NEXT: $ "echo" "foo bar"
-# CHECK-NEXT: $ "FileCheck" "{{.*}}"
-# CHECK-NEXT: $ ":" "RUN: at line 5"
-# CHECK-NEXT: $ "echo" "foo baz"
-# CHECK-NEXT: $ "FileCheck" "{{.*}}"
-# CHECK-NOT:  RUN
+# TERSE:      Command Output (stderr, truncated)
+# TERSE:      --
+# TERSE-NOT:  RUN: at line 4
+# TERSE-NOT:  true
+# TERSE:      RUN: at line 5
+# TERSE:      false
+# TERSE-NOT:  RUN
+# TERSE:      NOTE: The failure may depend on previous RUN lines.
+
+# TERSE-LABEL: FAIL: shtest-run-at-line :: external-shell/line-continuation.txt
+
+# TERSE-NOT:  Script:
+
+# TERSE:      Command Output (stderr, truncated)
+# TERSE-NOT:  RUN: at line 4
+# TERSE:      RUN: at line 6
+# TERSE:      echo 'foo baz'
+# TERSE:      FileCheck
+# TERSE-NOT:  RUN
+# TERSE:      NOTE: The failure may depend on previous RUN lines.
+
+# TERSE-LABEL: FAIL: shtest-run-at-line :: internal-shell/basic.txt
+
+# TERSE-NOT:      Script:
+
+# TERSE:      Command Output (stdout, truncated)
+# TERSE-NOT:  $ ":" "RUN: at line 1"
+# TERSE:      $ ":" "RUN: at line 2"
+# TERSE-NEXT: $ "false"
+# TERSE-NOT:  RUN
+# TERSE:      NOTE: The failure may depend on previous RUN lines.
+
+# TERSE-LABEL: FAIL: shtest-run-at-line :: internal-shell/line-continuation.txt
+
+# TERSE-NOT:  Script:
+
+# TERSE:      Command Output (stdout, truncated)
+# TERSE-NOT:  $ ":" "RUN: at line 1"
+# TERSE-NOT:  $ ":" "RUN: at line 3"
+# TERSE:      $ ":" "RUN: at line 5"
+# TERSE-NEXT: $ "echo" "foo baz"
+# TERSE-NEXT: $ "FileCheck" "{{.*}}"
+# TERSE-NOT:  RUN
+# TERSE:      (NOTE: The failure may depend on previous RUN lines.
+# TERSE-NEXT:        Use --verbose to see previous RUN lines and outputs.)
+
+################################################################################
+# Checking lines for verbose output
+
+# VERBOSE: Testing: 4 tests
+
+# VERBOSE-LABEL: FAIL: shtest-run-at-line :: external-shell/basic.txt
+
+# VERBOSE:      Script:
+# VERBOSE:      RUN: at line 4{{.*}}  true
+# VERBOSE-NEXT: RUN: at line 5{{.*}}  false
+# VERBOSE-NEXT: RUN: at line 6{{.*}}  true
+
+# VERBOSE:     RUN: at line 4
+# VERBOSE:     RUN: at line 5
+# VERBOSE-NOT: RUN: at line 6
+# VERBOSE-NOT: (NOTE: The failure may depend on previous RUN lines.
+
+# VERBOSE-LABEL: FAIL: shtest-run-at-line :: external-shell/line-continuation.txt
+
+# VERBOSE:      Script:
+# VERBOSE:      RUN: at line 4{{.*}}  echo 'foo bar'  | FileCheck
+# VERBOSE-NEXT: RUN: at line 6{{.*}}  echo 'foo baz'  | FileCheck
+# VERBOSE-NEXT: RUN: at line 9{{.*}}  echo 'foo bar'  | FileCheck
+
+# VERBOSE:     RUN: at line 4
+# VERBOSE:     RUN: at line 6
+# VERBOSE-NOT: RUN
+# VERBOSE-NOT: (NOTE: The failure may depend on previous RUN lines.
+
+# VERBOSE-LABEL: FAIL: shtest-run-at-line :: internal-shell/basic.txt
+
+# VERBOSE:      Script:
+# VERBOSE:      : 'RUN: at line 1';  true
+# VERBOSE-NEXT: : 'RUN: at line 2';  false
+# VERBOSE-NEXT: : 'RUN: at line 3';  true
+
+# VERBOSE:      Command Output (stdout)
+# VERBOSE:      $ ":" "RUN: at line 1"
+# VERBOSE-NEXT: $ "true"
+# VERBOSE-NEXT: $ ":" "RUN: at line 2"
+# VERBOSE-NEXT: $ "false"
+# VERBOSE-NOT:  RUN
+# VERBOSE-NOT: (NOTE: The failure may depend on previous RUN lines.
+
+# VERBOSE-LABEL: FAIL: shtest-run-at-line :: internal-shell/line-continuation.txt
+
+# VERBOSE:      Script:
+# VERBOSE:      : 'RUN: at line 1';  : first line continued to second line
+# VERBOSE-NEXT: : 'RUN: at line 3';  echo 'foo bar'  | FileCheck
+# VERBOSE-NEXT: : 'RUN: at line 5';  echo  'foo baz'  | FileCheck
+# VERBOSE-NEXT: : 'RUN: at line 8';  echo 'foo bar'  | FileCheck
+
+# VERBOSE:      Command Output (stdout)
+# VERBOSE:      $ ":" "RUN: at line 1"
+# VERBOSE-NEXT: $ ":" "first" "line" "continued" "to" "second" "line"
+# VERBOSE-NEXT: $ ":" "RUN: at line 3"
+# VERBOSE-NEXT: $ "echo" "foo bar"
+# VERBOSE-NEXT: $ "FileCheck" "{{.*}}"
+# VERBOSE-NEXT: $ ":" "RUN: at line 5"
+# VERBOSE-NEXT: $ "echo" "foo baz"
+# VERBOSE-NEXT: $ "FileCheck" "{{.*}}"
+# VERBOSE-NOT:  RUN
+# VERBOSE-NOT: (NOTE: The failure may depend on previous RUN lines.

llvm/utils/lit/tests/unittest-failing-locator.py

+# Check that the locate_last_failing_run_line function works as expected.
+
+# RUN: %{python} %s
+# END.
+
+import unittest
+
+from lit.TestRunner import locate_last_run_line
+
+class TestRunLineLocatorHeuristics(unittest.TestCase):
+
+    def test_basic(self):
+        # from a script like
+        # RUN: echo Hello 1>&2
+        basic = (
+            "+ : 'RUN: at line 1'\n"
+            "+ echo Hello 1>&2\n"
+            "+ Hello\n"
+        )
+        line_start, substr = locate_last_run_line(basic)
+        self.assertEqual(line_start, 0)
+        self.assertEqual(substr, "RUN: at line 1")
+
+    def test_multiple(self):
+        # from a script like
+        # RUN: echo Hello 1>&2
+        # RUN: false
+        multiple = (
+            "+ : 'RUN: at line 1'\n"
+            "+ echo Hello 1>&2\n"
+            "+ Hello\n"
+            "+ : 'RUN: at line 2'\n"
+            "+ false\n"
+        )
+        line_start, substr = locate_last_run_line(multiple)
+        self.assertEqual(line_start, multiple.rfind("+ :"))
+        self.assertEqual(substr, "RUN: at line 2")
+
+    def test_varying_prefix(self):
+        # from a script like
+        # RUN: echo Hello 1>&2
+        # RUN: false
+        #
+        # in a hypothetical shell which prints line-numbers on 'set +x'
+        # (as an example of something that varies)
+        varying_prefix = (
+            "+ 1 : 'RUN: at line 1'\n"
+            "+ 2 echo Hello 1>&2\n"
+            "+ Hello\n"
+            "+ 3 : 'RUN: at line 2'\n"
+            "+ 4 false\n"
+        )
+        line_start, substr = locate_last_run_line(varying_prefix)
+        self.assertEqual(line_start, varying_prefix.rfind("+ 3"))
+        self.assertEqual(substr, "RUN: at line 2")
+
+    def test_confusing_basic(self):
+        # from a script like
+        # RUN: echo 'RUN: at line 10' 1>&2
+        confusing_basic = (
+            "+ : 'RUN: at line 1'\n"
+            "+ echo 'RUN: at line 10'\n"
+            "RUN: at line 10\n"
+        )
+        line_start, substr = locate_last_run_line(confusing_basic)
+        # FIXME: These should both be equal ideally.
+        self.assertNotEqual(line_start, 0)
+        self.assertNotEqual(substr, "RUN: at line 1")
+
+    def test_confusing_multiple(self):
+        # from a script like
+        # RUN: echo 'RUN: at line 10' 1>&2
+        # RUN: false
+        confusing_multiple = (
+            "+ : 'RUN: at line 1'\n"
+            "+ echo 'RUN: at line 10'\n"
+            "RUN: at line 10\n"
+            "+ : 'RUN: at line 2'\n"
+            "+ false\n"
+        )
+        line_start, substr = locate_last_run_line(confusing_multiple)
+        self.assertEqual(line_start, confusing_multiple.rfind("+ :"))
+        self.assertEqual(substr, "RUN: at line 2")
+
+    def test_confusing_varying_prefix_1(self):
+        # from a script like
+        # RUN: echo 'RUN: at line 10' 1>&2
+        # RUN: false
+        #
+        # in a hypothetical shell which prints line-numbers on 'set +x'
+        # (as an example of something that varies)
+        confusing_varying_prefix = (
+            "+ 1 : 'RUN: at line 1'\n"
+            "+ 2 echo 'RUN: at line 10'\n"
+            "RUN: at line 10\n"
+            "+ 3 : 'RUN: at line 2'\n"
+            "+ 4 false\n"
+        )
+        line_start, substr = locate_last_run_line(confusing_varying_prefix)
+        self.assertEqual(line_start, confusing_varying_prefix.rfind("+ 3"))
+        self.assertEqual(substr, "RUN: at line 2")
+
+    def test_confusing_varying_prefix_2(self):
+        # from a script like
+        # RUN: true
+        # RUN: not echo 'RUN: at line 100'
+        #
+        # in a hypothetical shell which prints line-numbers on 'set +x'
+        # (as an example of something that varies)
+        confusing_varying_prefix = (
+            "+ 1 : 'RUN: at line 1'\n"
+            "+ 2 true\n"
+            "+ 3 : 'RUN: at line 2'\n"
+            "+ 4 not echo 'RUN: at line 100'\n"
+            "+ RUN: at line 100\n"
+        )
+        line_start, substr = locate_last_run_line(confusing_varying_prefix)
+        # FIXME: These should both be equal ideally.
+        self.assertNotEqual(line_start, confusing_varying_prefix.rfind("+ 3"))
+        self.assertNotEqual(substr, "RUN: at line 2")
+
+unittest.main()