[FileCheck] Implement -dump-input-context

[jdenny]

Summary

This patch is motivated by discussions at each of:

• https://reviews.llvm.org/D81422
• http://lists.llvm.org/pipermail/llvm-dev/2020-June/142369.html

When input is dumped as specified by -dump-input=fail, this patch
filters the dump to show only input lines that are the starting lines
of error diagnostics plus the number of contextual lines specified
-dump-input-context (defaults to 5).

When -dump-input=always, there might be not be any errors, so all
input lines are printed, as without this patch.

Here's some sample output with -dump-input-context=3 -vv:

<<<<<<
           .
           .
           .
          13: foo
          14: foo
          15: hello world
check:1       ^~~~~~~~~~~
          16: foo
check:2'0     X~~ error: no match found
          17: foo
check:2'0     ~~~
          18: foo
check:2'0     ~~~
          19: foo
check:2'0     ~~~
           .
           .
           .
          27: foo
check:2'0     ~~~
          28: foo
check:2'0     ~~~
          29: foo
check:2'0     ~~~
          30: goodbye word
check:2'0     ~~~~~~~~~~~~
check:2'1     ?            possible intended match
          31: foo
check:2'0     ~~~
          32: foo
check:2'0     ~~~
          33: foo
check:2'0     ~~~
           .
           .
           .
>>>>>>

[jdenny]

llvm/utils/FileCheck/FileCheck.cpp

517

Oops, it's not always 10.

[jdenny]

I'm thinking there could also be a -dump-input-filter command-line option to describe what lines of input should be sought, and -dump-input-context would describe how much context around those lines should be printed. It might have values like:

• all: include all input lines. The effect is to disable -dump-input-context. This is probably best for CI.
• annotation: include only input lines with annotations.
• error: include only input lines with error diagnostics. This is very nearly what this patch implements now. It would probably be the default.
• input: include only raw input lines and no annotations so you can more easily try that exact input again with FileCheck. This also would disable -dump-input-context.

[mehdi_amini]

fantastic :)

[arsenm]

llvm/utils/FileCheck/FileCheck.cpp

136

Does the description need to explicitly call out the default? I would expect that to be part of the standard opt dump, and it would avoid it getting out of sync with the actual value

[jdenny]

llvm/utils/FileCheck/FileCheck.cpp

136

-help isn't printing it. Am I doing something wrong?

[jdenny]

With this patch in place, I tried to write some tests and quickly became frustrated with the missing context, so I implemented the -dump-input-filter idea I mentioned to make things more tunable. I also adjusted it so it doesn't elide lines if the ellipsis would occupy more lines.

[mehdi_amini]

Are you still playing with this or are you looking for reviews? Or do you need more general feedback? (just making sure you're not blocked waiting on anything)

[jdenny]

No, I'm not blocked. Feedback is welcome with the caveat that details might evolve as I write tests. I posted early in case anyone has high-level suggestions. Thanks for the support so far. :-)

[jhenderson]

Not looked at the code at all, but I like the idea of being able to limit the context. grep can do it, and FileCheck is basically a glorified grep, so it seems like a logical (and useful) extension!

[rsmith]

This direction looks great to me! The recent changes to FileCheck have made its output useless for many Clang tests (so much output is produced that the actual error message is not even in my scrollback!) and it looks like this would give a much better balance between terseness and including useful information, while still letting us produce the full output as part of CI.

[jdenny]

Thanks for the feedback. I'll try to invest some more time in this tomorrow.

[SjoerdMeijer]

Sorry, late reply, but many thanks for working on this! I think the example in the description looks great, and this change would make dumping the input a very useful feature.

[jdenny]

Thanks for the feedback so far. This is ready for a detailed review now. I've made the following changes:

• I've updated/extended FileCheck's test suite.
• -dump-input-filter still defaults to error when -dump-input=fail (the default for -dump-input). However, it now defaults to all when -dump-input=always. The reason is that -dump-input=always is specifically useful for producing an input dump when there's no error, but -dump-input-filter=error always elides the entire input in that case.
• -dump-input-filter and -dump-input-context can now be specified multiple times, and the value that requests the most informative/verbose input dump has precedence. This mimics the existing -dump-input behavior, which has proven to be useful when a test author specifies one of these options in a test case but a test runner specifies the same option with a different value in FILECHECK_OPTS.

I've added many reviewers to make sure people who've participated in this discussion previously have a chance to comment. I apologize if I'm spamming or missing anyone.

[jdenny]

llvm/utils/FileCheck/FileCheck.cpp

136

This comment should be moot now that we're accepting multiple instances of the option and selecting the most informative/verbose value.

[arsenm]

llvm/utils/FileCheck/FileCheck.cpp

774–778

I already disliked the extra noise this message adds to the failure, and this adds additional lines. Can we only print this kind of information in a help mode?

[jdenny]

llvm/utils/FileCheck/FileCheck.cpp

774–778

Sure, I can move the new lines into the -dump-input=help output.

[Meinersbur]

To reduce the amount the output, could the input dump be restricted to the range between two labels? The rationale is that those ranges are logically independent blocks (such as a function) and the mismatch should be contained within such a block.

[jdenny]

In D82203#2113046, @Meinersbur wrote:

To reduce the amount the output, could the input dump be restricted to the range between two labels? The rationale is that those ranges are logically independent blocks (such as a function) and the mismatch should be contained within such a block.

Do you mean this as an alternative to the patch under review? As others have pointed out, the range between two labels can be large, and sometimes there are no labels and the input is large. Either way, its seems that restricting the dump to the range between two labels would often not solve the verbosity problem people are reporting, so I don't think that should be the default behavior. The patch under review should do better at limiting the verbosity per error.

Do you instead mean this as an additional opt-in feature? I'm not opposed if people feel that would be helpful, but I think it should be a later review. It might be enabled by -dump-input-context=label.

[Meinersbur]

The LABEL can be large as well (or the test does not have labels), but I think it would a nice feature to have in addition to line limits since it is not as arbitrary as a number of lines. Both approaches can be combined, i.e. restrict emitted lines to within two labels, even if -dump-input-context alone would print lines before and after.

Just asking to consider this, it might be too difficult to implement.

[jdenny]

In D82203#2114309, @Meinersbur wrote:

The LABEL can be large as well (or the test does not have labels), but I think it would a nice feature to have in addition to line limits since it is not as arbitrary as a number of lines. Both approaches can be combined, i.e. restrict emitted lines to within two labels, even if -dump-input-context alone would print lines before and after.

A question would be which label section. Given the patch under review, I think -dump-input-filter=error is an intuitive way to answer that question, and -dump-input-context=label is an intuitive way to specify the full label section should be dumped as the context of the error. Something like -dump-input-context=label+10 could be a way to indicate additional context outside the label section, if that's useful.

Just asking to consider this, it might be too difficult to implement.

I think the main challenge would be that label information is normally not available to the input dump implementation unless -v is specified, but we could make -dump-input-context=label enable it.

[jdenny]

Rebased.

Applied reviewer suggestion to move input-dump-related command-line options to -dump-input=help output.

[lattner]

High level question - do we need additional command line flags here? Why not build this into the default actions that FileCheck takes? These detailed flags are unlikely to be used by people in practice, and this expands the surface area of FileCheck that needs to be maintained - it also expands the testing surface of FileCheck itself.

[jdenny]

In D82203#2118764, @lattner wrote:

High level question - do we need additional command line flags here? Why not build this into the default actions that FileCheck takes?

By default, we'd have -dump-input-filter=error and -dump-input-context=5. That will hopefully address the concern that the current default (without this patch) is too verbose for many people.

These detailed flags are unlikely to be used by people in practice, and this expands the surface area of FileCheck that needs to be maintained - it also expands the testing surface of FileCheck itself.

I'm already using the patch under review while writing tests for other work, and the defaults I mentioned above are sometimes just right but other times are not nearly verbose enough. I frequently need more context to understand what has gone wrong in a test, so I find myself adjusting these options via FILECHECK_OPTS. I hope CI configurations will set FILECHECK_OPTS=-dump-input-filter=all -vv (maybe -color too) so that no one will ever miss context they need to debug a failure.

A major lesson I've learned from this discussion is that different LLVM developers have significantly different needs here, depending on what they're testing and probably depending on personal preferences. Tolerable defaults plus configurability seems key to addressing the various use cases.

By the way, I'm working on splitting up this patch to make it easier to review. Maybe it won't seem as complex after that.

[lattner]

Ok, but instead of allowing "fine grained control", why not allow a "tasteful default" and "all" option? The "all" option seems perfectly reasonable for the weird case, and eliminates a big configuration matrix. This would just entail turning the command line option into a magic number / enum.

-Chris

[jdenny]

In D82203#2118886, @lattner wrote:

Ok, but instead of allowing "fine grained control", why not allow a "tasteful default" and "all" option? The "all" option seems perfectly reasonable for the weird case,

Hopefully we can reach a consensus on what's tasteful. I figured there would always be someone unhappy with the default, but then they'd have a way out via these options and FILECHECK_OPTS.

and eliminates a big configuration matrix. This would just entail turning the command line option into a magic number / enum.

If the configuration space would reduce to a bool that either selects the default or all, there would be two main changes in terms of implementation and testing:

1. The elimination of -dump-input-context would mean that DumpInputContext would be replaced by 5 (or whatever value people ultimately agree on). That doesn't seem like much of a win. I will also need to rewrite the test dump-input-context.txt, which checks that the right amount of context is shown vs. elided in various boundary cases. It currently tests this by varying -dump-input-context, but I'll need to change it to vary the amount of input instead. At first glance, that seems more complex to write and maintain than what's there now, especially if the magic 5 should ever change, but maybe I'm overlooking a trick.

2. The change to -dump-input-filter would mean that FindInputLineInFilter would handle only two cases instead of four. However, the infrastructure to support those two cases is the same infrastructure needed to support all four. The result is that only this one already simple function would shrink. Again, that doesn't seem like a win. The test dump-input-filter.txt will also get a little smaller, but not much less complex, in my opinion.

If the win is that the configuration space would be significantly easier to understand for a user, I might be able to buy that for #2 but not for #1.

A middle ground here might be to drop -dump-input-filter but keep -dump-input-context as is. People can specify -dump-input-context=99999 (or maybe some magic string like all) if they want all the context.

What do you think? Thanks for your feedback.

[jdenny]

Fixed a FileCheck test broken by my last update.

[jdenny]

I've split this review into three:

• D83091: [FileCheck] Improve -dump-input documentation: This extracts some documentation changes that stand on their own.
• D82203: [FileCheck] Implement -dump-input-context: This is the current review. I've removed -dump-input-filter so we can decide whether it's worthwhile separately. I've kept -dump-input-context. I think it is frequently useful, it adds very little complexity to the implementation and interface, and it actually seems to make testing the elision of the input easier.
• D83097: [FileCheck] Implement -dump-input-filter

For the first and third reviews, I've added only a subset of the reviewers here as this review is the focus, but obviously everyone is welcome on all three.

[jdenny]

In an effort to facilitate the review, I've extracted the logic that elides input lines only when the ellipsis is shorter. That's now in D83526.

I've also rebased.

[jdenny]

I have no more simplifications planned, and there are no outstanding dependencies. This patch is ready for a review.

[jdenny]

Thanks for the reviews!

llvm/test/FileCheck/dump-input-annotations.txt

 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk -v 2>&1 \
 ; RUN: | FileCheck -strict-whitespace -match-full-lines -check-prefix=ALIGN \
 ; RUN:             -implicit-check-not='remark:' %s
 
 ; Verbose diagnostics are suppressed but not errors.
 ; ALIGN:{{.*}}error:{{.*}}
 ; ALIGN:{{.*}}possible intended match here{{.*}}
 
-; ALIGN:Full input was:
+; ALIGN:Input was:
 ; ALIGN-NEXT:<<<<<<
 ; ALIGN-NEXT:           1: hello world
 ; ALIGN-NEXT:check:1       ^~~~~
 ; ALIGN-NEXT:check:2'0           X~~~~ error: no match found
 ; ALIGN-NEXT:           2: goodbye
 ; ALIGN-NEXT:check:2'0     ~~~~~~~
 ; ALIGN-NEXT:           3: world
 ; ALIGN-NEXT:check:2'0     ~~~~~

llvm/test/FileCheck/dump-input-context.txt

+;--------------------------------------------------
+; Input file, check file, and directives for checking the size of the context.
+;
+; These are designed to be used with -dump-input=fail -vv.
+;
+; In the resulting input dump, there are three potential ellipses:
+;
+; - S: At the start of the input.
+; - M: Between two input lines included by the filter.
+; - E: At the end of the input.
+;
+; They are all present at -dump-input-context=6.  One disappears each time
+; -dump-input-context is incremented beyond that because there are no lines
+; left to elide.
+;--------------------------------------------------
+
+; RUN: echo foo8       >  %t.in
+; RUN: echo foo7       >> %t.in
+; RUN: echo foo6       >> %t.in
+; RUN: echo foo5       >> %t.in
+; RUN: echo foo4       >> %t.in
+; RUN: echo foo3       >> %t.in
+; RUN: echo foo2       >> %t.in
+; RUN: echo foo1       >> %t.in
+; RUN: echo lab1 hello >> %t.in
+; RUN: echo foo1       >> %t.in
+; RUN: echo foo2       >> %t.in
+; RUN: echo foo3       >> %t.in
+; RUN: echo foo4       >> %t.in
+; RUN: echo foo5       >> %t.in
+; RUN: echo foo6       >> %t.in
+; RUN: echo foo7       >> %t.in
+; RUN: echo foo7       >> %t.in
+; RUN: echo foo6       >> %t.in
+; RUN: echo foo5       >> %t.in
+; RUN: echo foo4       >> %t.in
+; RUN: echo foo3       >> %t.in
+; RUN: echo foo2       >> %t.in
+; RUN: echo foo1       >> %t.in
+; RUN: echo lab2 world >> %t.in
+; RUN: echo foo1       >> %t.in
+; RUN: echo foo2       >> %t.in
+; RUN: echo foo3       >> %t.in
+; RUN: echo foo4       >> %t.in
+; RUN: echo foo5       >> %t.in
+; RUN: echo foo6       >> %t.in
+; RUN: echo foo7       >> %t.in
+; RUN: echo foo8       >> %t.in
+; RUN: echo foo9       >> %t.in
+
+; RUN: echo 'CHECK-LABEL: lab1'  >  %t.chk
+; RUN: echo ' CHECK-NEXT: hello' >> %t.chk
+; RUN: echo 'CHECK-LABEL: lab2'  >> %t.chk
+; RUN: echo ' CHECK-NEXT: world' >> %t.chk
+
+;      C0: <<<<<<
+; CS-NEXT:            .
+; CS-NEXT:            .
+; CS-NEXT:            .
+; C8-NEXT:            1: foo8
+; C7-NEXT:            2: foo7
+; C6-NEXT:            3: foo6
+; C5-NEXT:            4: foo5
+; C4-NEXT:            5: foo4
+; C3-NEXT:            6: foo3
+; C2-NEXT:            7: foo2
+; C1-NEXT:            8: foo1
+; C0-NEXT:            9: lab1 hello
+; C0-NEXT: label:1'0     ^~~~
+; C0-NEXT: label:1'1     ^~~~
+; C0-NEXT: next:2             !~~~~ error: match on wrong line
+; C1-NEXT:           10: foo1
+; C2-NEXT:           11: foo2
+; C3-NEXT:           12: foo3
+; C4-NEXT:           13: foo4
+; C5-NEXT:           14: foo5
+; C6-NEXT:           15: foo6
+; C7-NEXT:           16: foo7
+; CM-NEXT:            .
+; CM-NEXT:            .
+; CM-NEXT:            .
+; C7-NEXT:           17: foo7
+; C6-NEXT:           18: foo6
+; C5-NEXT:           19: foo5
+; C4-NEXT:           20: foo4
+; C3-NEXT:           21: foo3
+; C2-NEXT:           22: foo2
+; C1-NEXT:           23: foo1
+; C0-NEXT:           24: lab2 world
+; C0-NEXT: label:3       ^~~~
+; C0-NEXT: next:4             !~~~~ error: match on wrong line
+; C1-NEXT:           25: foo1
+; C2-NEXT:           26: foo2
+; C3-NEXT:           27: foo3
+; C4-NEXT:           28: foo4
+; C5-NEXT:           29: foo5
+; C6-NEXT:           30: foo6
+; C7-NEXT:           31: foo7
+; C8-NEXT:           32: foo8
+; C9-NEXT:           33: foo9
+; CE-NEXT:            .
+; CE-NEXT:            .
+; CE-NEXT:            .
+; C0-NEXT: >>>>>>
+
+;--------------------------------------------------
+; Check -dump-input-context=<bad value>.
+;--------------------------------------------------
+
+; RUN: %ProtectFileCheckOutput \
+; RUN: not FileCheck -dump-input=fail -vv %t.chk < %t.in 2>&1 \
+; RUN:               -dump-input-context=-1 \
+; RUN: | FileCheck %s -match-full-lines -check-prefix=BADVAL -DVAL=-1
+
+; RUN: %ProtectFileCheckOutput \
+; RUN: not FileCheck -dump-input=fail -vv %t.chk < %t.in 2>&1 \
+; RUN:               -dump-input-context=foobar \
+; RUN: | FileCheck %s -match-full-lines -check-prefix=BADVAL -DVAL=foobar
+
+BADVAL: {{F|f}}ile{{C|c}}heck{{.*}}: for the --dump-input-context option: '[[VAL]]' value invalid for uint argument!
+
+;--------------------------------------------------
+; Check -dump-input-context explicit values.
+;--------------------------------------------------
+
+; 0 is an important boundary case.
+; RUN: %ProtectFileCheckOutput \
+; RUN: not FileCheck -dump-input=fail -vv  %t.chk < %t.in 2>&1 \
+; RUN:               -dump-input-context=0 \
+; RUN: | FileCheck %s -match-full-lines -check-prefixes=C0,CS,CM,CE
+
+; 1 is an important boundary case.
+; RUN: %ProtectFileCheckOutput \
+; RUN: not FileCheck -dump-input=fail -vv %t.chk < %t.in 2>&1 \
+; RUN:               -dump-input-context=1 \
+; RUN: | FileCheck %s -match-full-lines -check-prefixes=C0,C1,CS,CM,CE
+
+; 6 is the boundary case at which all ellipses are present in our test.
+; RUN: %ProtectFileCheckOutput \
+; RUN: not FileCheck -dump-input=fail -vv %t.chk < %t.in 2>&1 \
+; RUN:               -dump-input-context=6 \
+; RUN: | FileCheck %s -match-full-lines \
+; RUN:                -check-prefixes=C0,C1,C2,C3,C4,C5,C6,CS,CM,CE
+
+; 7 is the boundary case at which the middle ellipsis disappears.
+; RUN: %ProtectFileCheckOutput \
+; RUN: not FileCheck -dump-input=fail -vv %t.chk < %t.in 2>&1 \
+; RUN:               -dump-input-context=7 \
+; RUN: | FileCheck %s -match-full-lines \
+; RUN:                -check-prefixes=C0,C1,C2,C3,C4,C5,C6,C7,CS,CE
+
+; 8 is the boundary case at which the start ellipsis disappears.
+; RUN: %ProtectFileCheckOutput \
+; RUN: not FileCheck -dump-input=fail -vv %t.chk < %t.in 2>&1 \
+; RUN:               -dump-input-context=8 \
+; RUN: | FileCheck %s -match-full-lines \
+; RUN:                -check-prefixes=C0,C1,C2,C3,C4,C5,C6,C7,C8,CE
+
+; 9 is the boundary case at which the end ellipsis disappears.
+; RUN: %ProtectFileCheckOutput \
+; RUN: not FileCheck -dump-input=fail -vv %t.chk < %t.in 2>&1 \
+; RUN:               -dump-input-context=9 \
+; RUN: | FileCheck %s -match-full-lines \
+; RUN:                -check-prefixes=C0,C1,C2,C3,C4,C5,C6,C7,C8,C9
+
+; Make sure all is fine when -dump-input-context is far larger than the input.
+; RUN: %ProtectFileCheckOutput \
+; RUN: not FileCheck -dump-input=fail -vv %t.chk < %t.in 2>&1 \
+; RUN:               -dump-input-context=200 \
+; RUN: | FileCheck %s -match-full-lines \
+; RUN:                -check-prefixes=C0,C1,C2,C3,C4,C5,C6,C7,C8,C9
+
+;--------------------------------------------------
+; Check that -dump-input-context default is 5.
+;--------------------------------------------------
+
+; RUN: %ProtectFileCheckOutput \
+; RUN: not FileCheck -dump-input=fail -vv %t.chk < %t.in 2>&1 \
+; RUN: | FileCheck %s -match-full-lines \
+; RUN:                -check-prefixes=C0,C1,C2,C3,C4,C5,CS,CM,CE
+
+;--------------------------------------------------
+; Check multiple -dump-input-context options.
+;
+; This might occur when a test author specifies -dump-input-context on a
+; specific FileCheck call while a test runner specifies -dump-input-context in
+; FILECHECK_OPTS, but check the behavior generally.
+;
+; The largest value wins because it provides the most information.
+;--------------------------------------------------
+
+;- - - - - - - - - - - - - - - - - - - - - - - - -
+; Check duplicate.
+;- - - - - - - - - - - - - - - - - - - - - - - - -
+
+; RUN: %ProtectFileCheckOutput \
+; RUN: not FileCheck -dump-input=fail -vv %t.chk < %t.in 2>&1 \
+; RUN:               -dump-input-context=1 -dump-input-context=1 \
+; RUN: | FileCheck %s -match-full-lines -check-prefixes=C0,C1,CS,CM,CE
+
+;- - - - - - - - - - - - - - - - - - - - - - - - -
+; Check precedence.
+;- - - - - - - - - - - - - - - - - - - - - - - - -
+
+; RUN: %ProtectFileCheckOutput \
+; RUN: not FileCheck -dump-input=fail -vv %t.chk < %t.in 2>&1 \
+; RUN:               -dump-input-context=0 -dump-input-context=1 \
+; RUN: | FileCheck %s -match-full-lines -check-prefixes=C0,C1,CS,CM,CE
+
+; RUN: %ProtectFileCheckOutput \
+; RUN: not FileCheck -dump-input=fail -vv %t.chk < %t.in 2>&1 \
+; RUN:               -dump-input-context=1 -dump-input-context=0 \
+; RUN: | FileCheck %s -match-full-lines -check-prefixes=C0,C1,CS,CM,CE
+
+;- - - - - - - - - - - - - - - - - - - - - - - - -
+; Check that FILECHECK_OPTS isn't handled differently.
+;- - - - - - - - - - - - - - - - - - - - - - - - -
+
+; RUN: %ProtectFileCheckOutput FILECHECK_OPTS=-dump-input-context=0 \
+; RUN: not FileCheck -dump-input=fail -vv %t.chk < %t.in 2>&1 \
+; RUN:               -dump-input-context=1 \
+; RUN: | FileCheck %s -match-full-lines -check-prefixes=C0,C1,CS,CM,CE
+
+; RUN: %ProtectFileCheckOutput FILECHECK_OPTS=-dump-input-context=1 \
+; RUN: not FileCheck -dump-input=fail -vv %t.chk < %t.in 2>&1 \
+; RUN:               -dump-input-context=0 \
+; RUN: | FileCheck %s -match-full-lines -check-prefixes=C0,C1,CS,CM,CE

llvm/test/FileCheck/dump-input-enable.txt

 ; TRACE:       {{.*}}remark:{{.*}}
 ; NOTRACE-NOT: remark:
 
 ; Error diagnostics are never suppressed.
 ; ERR: {{.*}}error:{{.*}}
 
 ; NODUMP-NOT: <<<<<<
 
-; DUMP-OK:      Full input was:
+; DUMP-OK:      Input was:
 ; DUMP-OK-NEXT: <<<<<<
 ; DUMP-OK-NEXT:          1: hello
 ; DUMP-OK-NEXT: check:1     ^~~~~
 ; DUMP-OK-NEXT:          2: world
 ; DUMP-OK-NEXT: next:2      ^~~~~
 ; DUMP-OK-NEXT: >>>>>>
 
-; DUMP-ERR:        Full input was:
+; DUMP-ERR:        Input was:
 ; DUMP-ERR-NEXT:   <<<<<<
 ; DUMP-ERR-NEXT:            1: hello
 ; DUMP-ERR-V-NEXT: check:1     ^~~~~
 ; DUMP-ERR-NEXT:            2: whirled
 ; DUMP-ERR-NEXT:   next:2      X~~~~~~ error: no match found
 ; DUMP-ERR-NEXT:   >>>>>>

llvm/test/FileCheck/dump-input-filter.txt

+; To keep this test maintainable, avoid depending on -dump-input-context's
+; default value, which is checked in dump-input-context.txt instead.
+
+;--------------------------------------------------
+; Create the input file and the check file.
+;--------------------------------------------------
+
+; line 1
+; RUN: echo start >  %t.in
+; RUN: echo foo0  >> %t.in
+; RUN: echo foo1  >> %t.in
+; RUN: echo foo2  >> %t.in
+; RUN: echo foo3  >> %t.in
+; RUN: echo foo4  >> %t.in
+; RUN: echo foo5  >> %t.in
+; RUN: echo foo6  >> %t.in
+; RUN: echo foo7  >> %t.in
+; RUN: echo foo8  >> %t.in
+; RUN: echo foo9  >> %t.in
+; line 12
+; RUN: echo hello >> %t.in
+; RUN: echo foo0  >> %t.in
+; RUN: echo foo1  >> %t.in
+; RUN: echo foo2  >> %t.in
+; RUN: echo foo3  >> %t.in
+; RUN: echo foo4  >> %t.in
+; RUN: echo foo5  >> %t.in
+; RUN: echo foo6  >> %t.in
+; RUN: echo foo7  >> %t.in
+; RUN: echo foo8  >> %t.in
+; RUN: echo foo9  >> %t.in
+; line 23
+; RUN: echo word  >> %t.in
+; RUN: echo foo0  >> %t.in
+; RUN: echo foo1  >> %t.in
+; RUN: echo foo2  >> %t.in
+; RUN: echo foo3  >> %t.in
+; RUN: echo foo4  >> %t.in
+; RUN: echo foo5  >> %t.in
+; RUN: echo foo6  >> %t.in
+; RUN: echo foo7  >> %t.in
+; RUN: echo foo8  >> %t.in
+; RUN: echo foo9  >> %t.in
+; line 34
+; RUN: echo end   >> %t.in
+
+; RUN: echo 'CHECK: start' >  %t.chk
+; RUN: echo 'CHECK: hello' >> %t.chk
+; RUN: echo 'CHECK: world' >> %t.chk
+; RUN: echo 'CHECK: end'   >> %t.chk
+
+;--------------------------------------------------
+; Directives for checking the dump.
+;--------------------------------------------------
+
+;      ALL: <<<<<<
+; ALL-NEXT:            1: start
+; ALL-NEXT: check:1       ^~~~~
+; ALL-NEXT:            2: foo0
+; ALL-NEXT:            3: foo1
+; ALL-NEXT:            4: foo2
+; ALL-NEXT:            5: foo3
+; ALL-NEXT:            6: foo4
+; ALL-NEXT:            7: foo5
+; ALL-NEXT:            8: foo6
+; ALL-NEXT:            9: foo7
+; ALL-NEXT:           10: foo8
+; ALL-NEXT:           11: foo9
+; ALL-NEXT:           12: hello
+; ALL-NEXT: check:2       ^~~~~
+; ALL-NEXT:           13: foo0
+; ALL-NEXT: check:3'0     X~~~ error: no match found
+; ALL-NEXT:           14: foo1
+; ALL-NEXT: check:3'0     ~~~~
+; ALL-NEXT:           15: foo2
+; ALL-NEXT: check:3'0     ~~~~
+; ALL-NEXT:           16: foo3
+; ALL-NEXT: check:3'0     ~~~~
+; ALL-NEXT:           17: foo4
+; ALL-NEXT: check:3'0     ~~~~
+; ALL-NEXT:           18: foo5
+; ALL-NEXT: check:3'0     ~~~~
+; ALL-NEXT:           19: foo6
+; ALL-NEXT: check:3'0     ~~~~
+; ALL-NEXT:           20: foo7
+; ALL-NEXT: check:3'0     ~~~~
+; ALL-NEXT:           21: foo8
+; ALL-NEXT: check:3'0     ~~~~
+; ALL-NEXT:           22: foo9
+; ALL-NEXT: check:3'0     ~~~~
+; ALL-NEXT:           23: word
+; ALL-NEXT: check:3'0     ~~~~
+; ALL-NEXT: check:3'1     ?     possible intended match
+; ALL-NEXT:           24: foo0
+; ALL-NEXT: check:3'0     ~~~~
+; ALL-NEXT:           25: foo1
+; ALL-NEXT: check:3'0     ~~~~
+; ALL-NEXT:           26: foo2
+; ALL-NEXT: check:3'0     ~~~~
+; ALL-NEXT:           27: foo3
+; ALL-NEXT: check:3'0     ~~~~
+; ALL-NEXT:           28: foo4
+; ALL-NEXT: check:3'0     ~~~~
+; ALL-NEXT:           29: foo5
+; ALL-NEXT: check:3'0     ~~~~
+; ALL-NEXT:           30: foo6
+; ALL-NEXT: check:3'0     ~~~~
+; ALL-NEXT:           31: foo7
+; ALL-NEXT: check:3'0     ~~~~
+; ALL-NEXT:           32: foo8
+; ALL-NEXT: check:3'0     ~~~~
+; ALL-NEXT:           33: foo9
+; ALL-NEXT: check:3'0     ~~~~
+; ALL-NEXT:           34: end
+; ALL-NEXT: check:3'0     ~~~
+; ALL-NEXT: >>>>>>
+
+;      ERROR: <<<<<<
+; ERROR-NEXT:            .
+; ERROR-NEXT:            .
+; ERROR-NEXT:            .
+; ERROR-NEXT:           11: foo9
+; ERROR-NEXT:           12: hello
+; ERROR-NEXT: check:2       ^~~~~
+; ERROR-NEXT:           13: foo0
+; ERROR-NEXT: check:3'0     X~~~ error: no match found
+; ERROR-NEXT:           14: foo1
+; ERROR-NEXT: check:3'0     ~~~~
+; ERROR-NEXT:           15: foo2
+; ERROR-NEXT: check:3'0     ~~~~
+; ERROR-NEXT:            .
+; ERROR-NEXT:            .
+; ERROR-NEXT:            .
+; ERROR-NEXT:           21: foo8
+; ERROR-NEXT: check:3'0     ~~~~
+; ERROR-NEXT:           22: foo9
+; ERROR-NEXT: check:3'0     ~~~~
+; ERROR-NEXT:           23: word
+; ERROR-NEXT: check:3'0     ~~~~
+; ERROR-NEXT: check:3'1     ?     possible intended match
+; ERROR-NEXT:           24: foo0
+; ERROR-NEXT: check:3'0     ~~~~
+; ERROR-NEXT:           25: foo1
+; ERROR-NEXT: check:3'0     ~~~~
+; ERROR-NEXT:            .
+; ERROR-NEXT:            .
+; ERROR-NEXT:            .
+; ERROR-NEXT: >>>>>>
+
+;--------------------------------------------------
+; Check how -dump-input affects filter.
+;--------------------------------------------------
+
+; no -dump-input => include errors.
+; RUN: %ProtectFileCheckOutput \
+; RUN: not FileCheck -dump-input-context=2 -vv %t.chk < %t.in 2>&1 \
+; RUN: | FileCheck %s -match-full-lines -check-prefixes=ERROR
+
+; -dump-input=fail => include errors.
+; RUN: %ProtectFileCheckOutput \
+; RUN: not FileCheck -dump-input-context=2 -vv %t.chk < %t.in 2>&1 \
+; RUN:               -dump-input=fail \
+; RUN: | FileCheck %s -match-full-lines -check-prefixes=ERROR
+
+; -dump-input=always => include all.
+; RUN: %ProtectFileCheckOutput \
+; RUN: not FileCheck -dump-input-context=2 -vv %t.chk < %t.in 2>&1 \
+; RUN:               -dump-input=always \
+; RUN: | FileCheck %s -match-full-lines -check-prefixes=ALL
+
+;--------------------------------------------------
+; Check that other kinds of errors are included by -dump-input=fail.
+;
+; "error: no match found" and "possible intended match" are checked above.
+;--------------------------------------------------
+
+;- - - - - - - - - - - - - - - - - - - - - - - - -
+; error: no match expected.
+;- - - - - - - - - - - - - - - - - - - - - - - - -
+
+; RUN: echo 'foo'            > %t.not-err.in
+; RUN: echo 'CHECK-NOT: foo' > %t.not-err.chk
+
+; RUN: %ProtectFileCheckOutput \
+; RUN: not FileCheck -dump-input-context=0 -dump-input=fail \
+; RUN:               %t.not-err.chk < %t.not-err.in 2>&1 \
+; RUN: | FileCheck %s -match-full-lines -check-prefixes=NOT-ERR
+
+;      NOT-ERR:        1: foo
+; NOT-ERR-NEXT: not:1     !~~ error: no match expected
+
+;- - - - - - - - - - - - - - - - - - - - - - - - -
+; error: match on wrong line.
+;- - - - - - - - - - - - - - - - - - - - - - - - -
+
+; RUN: echo 'foo'             >  %t.next-err.in
+; RUN: echo 'foo'             >> %t.next-err.in
+; RUN: echo 'bar'             >> %t.next-err.in
+; RUN: echo 'CHECK: foo'      >  %t.next-err.chk
+; RUN: echo 'CHECK-NEXT: bar' >> %t.next-err.chk
+
+; RUN: %ProtectFileCheckOutput \
+; RUN: not FileCheck -dump-input-context=0 -dump-input=fail \
+; RUN:               %t.next-err.chk < %t.next-err.in 2>&1 \
+; RUN: | FileCheck %s -match-full-lines -check-prefixes=NEXT-ERR
+
+;      NEXT-ERR:         3: bar
+; NEXT-ERR-NEXT: next:2     !~~ error: match on wrong line

llvm/utils/FileCheck/FileCheck.cpp

cl::desc("Dump input to stderr, adding annotations representing\n"
              "occurrences of this option, the <value> that appears earliest\n"
              "in the list below has precedence.  The default is 'fail'.\n"),
     cl::value_desc("mode"),
     cl::values(clEnumValN(DumpInputHelp, "help", "Explain input dump and quit"),
                clEnumValN(DumpInputAlways, "always", "Always dump input"),
                clEnumValN(DumpInputFail, "fail", "Dump input on failure"),
                clEnumValN(DumpInputNever, "never", "Never dump input")));
 
+static cl::list<unsigned> DumpInputContexts(
+    "dump-input-context", cl::value_desc("N"),
+    cl::desc("In the dump requested by -dump-input=fail, print <N> input\n"
+             "lines before and <N> input lines after the starting line of\n"
+             "any error diagnostic.  When there are multiple occurrences of\n"
+             "this option, the largest specified <N> has precedence.  The\n"
+             "default is 5.\n"));
+
 typedef cl::list<std::string>::const_iterator prefix_iterator;
 
 
 
 
 
 
 
 static void DumpCommandLine(int argc, char **argv) {
   errs() << "FileCheck command line: ";
   for (int I = 0; I < argc; I++)
     errs() << " " << argv[I];
   errs() << "\n";
 }
 
 struct MarkerStyle {
   /// The starting char (before tildes) for marking the line.
   char Lead;
   /// What color to use for this annotation.
   raw_ostream::Colors Color;
   /// A note to follow the marker, or empty string if none.
   std::string Note;
+  /// Does this marker indicate inclusion by the input filter implied by
+  /// -dump-input=fail?
+  bool FiltersAsError;
   MarkerStyle() {}
   MarkerStyle(char Lead, raw_ostream::Colors Color,
-              const std::string &Note = "")
-      : Lead(Lead), Color(Color), Note(Note) {}
+              const std::string &Note = "", bool FiltersAsError = false)
+      : Lead(Lead), Color(Color), Note(Note), FiltersAsError(FiltersAsError) {
+    assert((!FiltersAsError || !Note.empty()) &&
+           "expected error diagnostic to have note");
+  }
 };
 
 static MarkerStyle GetMarker(FileCheckDiag::MatchType MatchTy) {
   switch (MatchTy) {
   case FileCheckDiag::MatchFoundAndExpected:
     return MarkerStyle('^', raw_ostream::GREEN);
   case FileCheckDiag::MatchFoundButExcluded:
-    return MarkerStyle('!', raw_ostream::RED, "error: no match expected");
+    return MarkerStyle('!', raw_ostream::RED, "error: no match expected",
+                       /*FiltersAsError=*/true);
   case FileCheckDiag::MatchFoundButWrongLine:
-    return MarkerStyle('!', raw_ostream::RED, "error: match on wrong line");
+    return MarkerStyle('!', raw_ostream::RED, "error: match on wrong line",
+                       /*FiltersAsError=*/true);
   case FileCheckDiag::MatchFoundButDiscarded:
     return MarkerStyle('!', raw_ostream::CYAN,
                        "discard: overlaps earlier match");
   case FileCheckDiag::MatchNoneAndExcluded:
     return MarkerStyle('X', raw_ostream::GREEN);
   case FileCheckDiag::MatchNoneButExpected:
-    return MarkerStyle('X', raw_ostream::RED, "error: no match found");
+    return MarkerStyle('X', raw_ostream::RED, "error: no match found",
+                       /*FiltersAsError=*/true);
   case FileCheckDiag::MatchFuzzy:
-    return MarkerStyle('?', raw_ostream::MAGENTA, "possible intended match");
+    return MarkerStyle('?', raw_ostream::MAGENTA, "possible intended match",
+                       /*FiltersAsError=*/true);
   }
   llvm_unreachable_internal("unexpected match type");
 }
 
 static void DumpInputAnnotationHelp(raw_ostream &OS) {
   OS << "The following description was requested by -dump-input=help to\n"
      << "explain the input dump printed by FileCheck.\n"
      << "\n"
      << "Related command-line options:\n"
      << "  - -dump-input=<value> enables or disables the input dump\n"
+     << "  - -dump-input-context=<N> adjusts the context of errors\n"
      << "  - -v and -vv add more annotations\n"
      << "  - -color forces colors to be enabled both in the dump and below\n"
      << "  - -help documents the above options in more detail\n"
      << "\n"
      << "Input dump annotation format:\n";
 
   // Labels for input lines.
   OS << "  - ";
static void DumpInputAnnotationHelp(raw_ostream &OS) {
   OS << "    marks search range when no match is found, such as:\n"
      << "           - CHECK-NEXT not found (error)\n"
      << "           - CHECK-NOT not found (success, reported if -vv)\n"
      << "           - CHECK-DAG not found after discarded matches (error)\n"
      << "  - ";
   WithColor(OS, raw_ostream::SAVEDCOLOR, true) << "?";
   OS << "      marks fuzzy match when no match is found\n";
 
+  // Elided lines.
+  OS << "  - ";
+  WithColor(OS, raw_ostream::SAVEDCOLOR, true) << "...";
+  OS << "    indicates elided input lines and annotations, as specified by\n"
+     << "           -dump-input=fail and -dump-input-context\n";
+
   // Colors.
   OS << "  - colors ";
   WithColor(OS, raw_ostream::GREEN, true) << "success";
   OS << ", ";
   WithColor(OS, raw_ostream::RED, true) << "error";
   OS << ", ";
   WithColor(OS, raw_ostream::MAGENTA, true) << "fuzzy match";
   OS << ", ";
   WithColor(OS, raw_ostream::CYAN, true, false) << "discarded match";
   OS << ", ";
   WithColor(OS, raw_ostream::CYAN, true, true) << "unmatched input";
   OS << "\n";
 }
 
 /// An annotation for a single input line.
 struct InputAnnotation {
   /// The index of the match result across all checks
   unsigned DiagIndex;
   /// The label for this annotation.
   std::string Label;
+  /// Is this the initial fragment of a diagnostic that has been broken across
+  /// multiple lines?
+  bool IsFirstLine;
   /// What input line (one-origin indexing) this annotation marks.  This might
-  /// be different from the starting line of the original diagnostic if this is
-  /// a non-initial fragment of a diagnostic that has been broken across
-  /// multiple lines.
+  /// be different from the starting line of the original diagnostic if
+  /// !IsFirstLine.
   unsigned InputLine;
   /// The column range (one-origin indexing, open end) in which to mark the
   /// input line.  If InputEndCol is UINT_MAX, treat it as the last column
   /// before the newline.
   unsigned InputStartCol, InputEndCol;
   /// The marker to use.
   MarkerStyle Marker;
   /// Whether this annotation represents a good match for an expected pattern.
for (auto DiagItr = Diags.begin(), DiagEnd = Diags.end(); DiagItr != DiagEnd;
     LabelWidth = std::max((std::string::size_type)LabelWidth, A.Label.size());
 
     A.Marker = GetMarker(DiagItr->MatchTy);
     A.FoundAndExpectedMatch =
         DiagItr->MatchTy == FileCheckDiag::MatchFoundAndExpected;
 
     // Compute the mark location, and break annotation into multiple
     // annotations if it spans multiple lines.
+    A.IsFirstLine = true;
     A.InputLine = DiagItr->InputStartLine;
     A.InputStartCol = DiagItr->InputStartCol;
     if (DiagItr->InputStartLine == DiagItr->InputEndLine) {
       // Sometimes ranges are empty in order to indicate a specific point, but
       // that would mean nothing would be marked, so adjust the range to
       // include the following character.
       A.InputEndCol =
           std::max(DiagItr->InputStartCol + 1, DiagItr->InputEndCol);
       Annotations.push_back(A);
     } else {
       assert(DiagItr->InputStartLine < DiagItr->InputEndLine &&
              "expected input range not to be inverted");
       A.InputEndCol = UINT_MAX;
       Annotations.push_back(A);
       for (unsigned L = DiagItr->InputStartLine + 1, E = DiagItr->InputEndLine;
            L <= E; ++L) {
         // If a range ends before the first column on a line, then it has no
         // characters on that line, so there's nothing to render.
         if (DiagItr->InputEndCol == 1 && L == E)
           break;
         InputAnnotation B;
         B.DiagIndex = A.DiagIndex;
         B.Label = A.Label;
+        B.IsFirstLine = false;
         B.InputLine = L;
         B.Marker = A.Marker;
         B.Marker.Lead = '~';
         B.Marker.Note = "";
         B.InputStartCol = 1;
         if (L != E)
           B.InputEndCol = UINT_MAX;
         else
           B.InputEndCol = DiagItr->InputEndCol;
         B.FoundAndExpectedMatch = A.FoundAndExpectedMatch;
         Annotations.push_back(B);
       }
     }
   }
 }
 
+static unsigned FindInputLineInFilter(
+    bool FilterOnError, unsigned CurInputLine,
+    const std::vector<InputAnnotation>::iterator &AnnotationBeg,
+    const std::vector<InputAnnotation>::iterator &AnnotationEnd) {
+  if (!FilterOnError)
+    return CurInputLine;
+  for (auto AnnotationItr = AnnotationBeg; AnnotationItr != AnnotationEnd;
+       ++AnnotationItr) {
+    if (AnnotationItr->IsFirstLine && AnnotationItr->Marker.FiltersAsError)
+      return AnnotationItr->InputLine;
+  }
+  return UINT_MAX;
+}
+
 static void DumpAnnotatedInput(raw_ostream &OS, const FileCheckRequest &Req,
+                               bool DumpInputFilterOnError,
+                               unsigned DumpInputContext,
                                StringRef InputFileText,
                                std::vector<InputAnnotation> &Annotations,
                                unsigned LabelWidth) {
-  OS << "Full input was:\n<<<<<<\n";
+  OS << "Input was:\n<<<<<<\n";
 
   // Sort annotations.
   std::sort(Annotations.begin(), Annotations.end(),
             [](const InputAnnotation &A, const InputAnnotation &B) {
               // 1. Sort annotations in the order of the input lines.
               //
               // This makes it easier to find relevant annotations while
               // iterating input lines in the implementation below.  FileCheck
static void DumpAnnotatedInput(raw_ostream &OS, const FileCheckRequest &Req,
   // visually distinct.  For example, the spaces on the annotation lines ensure
   // that input line numbers and check directive line numbers never align
   // horizontally.  Those line numbers might not even be for the same file.
   // One space would be enough to achieve that, but more makes it even easier
   // to see.
   LabelWidth = std::max(LabelWidth, LineNoWidth) + 3;
 
   // Print annotated input lines.
+  unsigned PrevLineInFilter = 0; // 0 means none so far
+  unsigned NextLineInFilter = 0; // 0 means uncomputed, UINT_MAX means none
+  bool PrevLineElided = false;
   auto AnnotationItr = Annotations.begin(), AnnotationEnd = Annotations.end();
   for (unsigned Line = 1;
        InputFilePtr != InputFileEnd || AnnotationItr != AnnotationEnd;
        ++Line) {
     const unsigned char *InputFileLine = InputFilePtr;
 
+    // Compute the previous and next line included by the filter.
+    if (NextLineInFilter < Line)
+      NextLineInFilter = FindInputLineInFilter(DumpInputFilterOnError, Line,
+                                               AnnotationItr, AnnotationEnd);
+    assert(NextLineInFilter && "expected NextLineInFilter to be computed");
+    if (NextLineInFilter == Line)
+      PrevLineInFilter = Line;
+
+    // Elide this input line and its annotations if it's not within the
+    // context specified by -dump-input-context of an input line included by
+    // the dump filter.
+    if ((!PrevLineInFilter || PrevLineInFilter + DumpInputContext < Line) &&
+        (NextLineInFilter == UINT_MAX ||
+         Line + DumpInputContext < NextLineInFilter)) {
+      while (InputFilePtr != InputFileEnd && *InputFilePtr != '\n')
+        ++InputFilePtr;
+      if (InputFilePtr != InputFileEnd)
+        ++InputFilePtr;
+      while (AnnotationItr != AnnotationEnd && AnnotationItr->InputLine == Line)
+        ++AnnotationItr;
+      if (!PrevLineElided) {
+        for (unsigned i = 0; i < 3; ++i) {
+          WithColor(OS, raw_ostream::BLACK, /*Bold=*/true)
+              << right_justify(".", LabelWidth);
+          OS << '\n';
+        }
+        PrevLineElided = true;
+      }
+      continue;
+    }
+    PrevLineElided = false;
+
     // Print right-aligned line number.
     WithColor(OS, raw_ostream::BLACK, true)
         << format_decimal(Line, LabelWidth) << ": ";
 
     // For the case where -v and colors are enabled, find the annotations for
     // good matches for expected patterns in order to highlight everything
     // else in the line.  There are no such annotations if -v is disabled.
     std::vector<InputAnnotation> FoundAndExpectedMatches;

 int main(int argc, char **argv) {
   // Enable use of ANSI color codes because FileCheck is using them to
   // highlight text.
   llvm::sys::Process::UseANSIEscapeCodes(true);
 
   InitLLVM X(argc, argv);
   cl::ParseCommandLineOptions(argc, argv, /*Overview*/ "", /*Errs*/ nullptr,
                               "FILECHECK_OPTS");
+
+  // Select -dump-input* values.  The -help documentation specifies the default
+  // value and which value to choose if an option is specified multiple times.
+  // In the latter case, the general rule of thumb is to choose the value that
+  // provides the most information.
   DumpInputValue DumpInput =
       DumpInputs.empty()
           ? DumpInputFail
           : *std::max_element(DumpInputs.begin(), DumpInputs.end());
+  bool DumpInputFilterOnError = DumpInput == DumpInputFail;
+  unsigned DumpInputContext = DumpInputContexts.empty()
+                                  ? 5
+                                  : *std::max_element(DumpInputContexts.begin(),
+                                                      DumpInputContexts.end());
+
   if (DumpInput == DumpInputHelp) {
     DumpInputAnnotationHelp(outs());
     return 0;
   }
   if (CheckFilename.empty()) {
     errs() << "<check-file> not specified\n";
     return 2;
   }
int main(int argc, char **argv) {
   std::vector<FileCheckDiag> Diags;
   int ExitCode = FC.checkInput(SM, InputFileText,
                                DumpInput == DumpInputNever ? nullptr : &Diags)
                      ? EXIT_SUCCESS
                      : 1;
   if (DumpInput == DumpInputAlways ||
       (ExitCode == 1 && DumpInput == DumpInputFail)) {
     errs() << "\n"
            << "Input file: " << InputFilename << "\n"
            << "Check file: " << CheckFilename << "\n"
            << "\n"
            << "-dump-input=help explains the following input dump.\n"
            << "\n";
     std::vector<InputAnnotation> Annotations;
     unsigned LabelWidth;
     BuildInputAnnotations(SM, CheckFileBufferID, ImpPatBufferIDRange, Diags,
                           Annotations, LabelWidth);
-    DumpAnnotatedInput(errs(), Req, InputFileText, Annotations, LabelWidth);
+    DumpAnnotatedInput(errs(), Req, DumpInputFilterOnError, DumpInputContext,
+                       InputFileText, Annotations, LabelWidth);
   }
 
   return ExitCode;
 }