[docs] Document pattern of using CHECK-SAME to skip irrelevant lines

[rupprecht]

Summary

This came up during the review for D67656. It's nice but also subtle, so documenting it as an idiom will make tests easier to understand.

[MaskRay]

llvm/docs/CommandGuide/FileCheck.rst

427

Maybe align 1 to the position 2 columns right of Value: above.

[thopre]

llvm/docs/CommandGuide/FileCheck.rst

384

Nit: rather than say that it is common, it might be better to say that CHECK-SAME is useful in dealing with X. Whether it is common or not does not seem very relevant.

[jhenderson]

llvm/docs/CommandGuide/FileCheck.rst

384

Yeah, I agree with this. I wrote it for the first time a few months ago, and as far as I know, it was the first time it was used for this purpose (certainly I came up with it independently!), so "common" seems like a stretch at the moment.

[jhenderson]

llvm/docs/CommandGuide/FileCheck.rst

429–430

Maybe this would be better phrased as "This verifies that the next time "`Value:`" appears in the output, it has a value of 1.

It might also be worth a note saying to use {{$}} immediately after too, to avoid a spurious match on Value: 16 or similar. Not sure if it really belongs here, or somewhere else though.

[MaskRay]

llvm/docs/CommandGuide/FileCheck.rst

429–430

The suggestion looks good. So the example should be

CHECK:      Value:
CHECK-SAME:        1{{$}}

[rupprecht]

• Remove mention of "common" case
• Make the bit about the next line containing Value: more clear
• Fix some indentation

llvm/docs/CommandGuide/FileCheck.rst

384

Fair enough, I just figured everyone besides me was familiar with this trick :)

427

Did you mean 1 column (from the colon)?

429–430

I added {{$}} to the original matcher, it should be there too

[greened]

llvm/docs/CommandGuide/FileCheck.rst

418

This is what CHECK-LABEL is for. This isn't a very motivating example. Is there a better one?

[rupprecht]

llvm/docs/CommandGuide/FileCheck.rst

418

I don't think CHECK-LABEL works in this case. CHECK-LABEL requires unique lines, and the only thing unique here would be "Name: <...>" which is not the thing we have a problem checking.

e.g. something like:

CHECK: Name: foo
CHECK-LABEL: Value: 1

would fail on the example text above because foo and baz both have a value of one. Similarly,

CHECK-LABEL: Name: foo
CHECK: Value: 1

has the same false-passing problem as described in these added docs.

Of course, you could add tests for every name/value pair (and using check-label on the names to make error messages clearer), but then the test overall becomes not a unit test: you really just want to write a unit test that verifies foo has a value of 1; you don't want to assert the universe to do that.

I don't use check-label very often, I'm mostly relying on docs for my understanding. Could you clarify your comment if I'm missing something? Also: if I'm not missing something, let me know how I can make the wording here more clear.

[thopre]

llvm/docs/CommandGuide/FileCheck.rst

418

CHECK-LABEL will constraints the CHECK directive between them to match between the two posititions where the CHECK-LABEL match. In other word, if the name in your example are uniques then

CHECK-LABEL: Name: foo
CHECK: Value: 1
CHECK-LABEL: Name: bar

will guarantee that the Value: 1 will match something in the block of text between "Name: foo" and "Name: bar" or fail to match. Now if you only care about foo, your solution avoids adding a CHECK-LABEL for bar which might not be easy if you do not know the relative ordering of the blocks for foo and bar.

So I do think your use case is useful, but perhaps you should remove the parts about bar and baz (my preference) or mention that the order of blocks cannot be relied upon (which might be solved in some future time by having something like CHECK-LABEL-DAG/CHECK-REGION-DAG which would force to update this text to my first suggestion).

[jhenderson]

llvm/docs/CommandGuide/FileCheck.rst

418

Perhaps replacing the reference to baz with something like "a later symbol in the output" would be sufficient? It might also be worth noting why CHECK-LABEL isn't appropriate in this case (e.g. unknown ordering).

[thopre]

Ping?

[probinson]

llvm/docs/CommandGuide/FileCheck.rst

427

This will also match Value: 21 so it is not equivalent to
CHECK: Value: 1{{$}}

[thopre]

llvm/docs/CommandGuide/FileCheck.rst

427

True, could this dealt by using CHECK: Value: {{ 1$}}?

[probinson]

llvm/docs/CommandGuide/FileCheck.rst

427

Yes, that would work. Nice to have smart people around. :-)

[MaskRay]

Ping:)

[probinson]

@rupprecht if you are still motivated to land this after so long, the only thing I see missing is to update the example from CHECK-SAME: 1{{$}} to CHECK-SAME: {{ 1$}} and then it LGTM. If not, there seems to be enough interest that someone else can land it?

llvm/docs/CommandGuide/FileCheck.rst

418

CHECK-LABEL technically does not require unique lines, any more than it requires the pattern to be a label. So the sequence

CHECK: Name: foo
CHECK-LABEL: Value: 1{{$}}

would not fail in the case where there were multiple Value: 1 lines; the LABEL would match the first one.
However, the test would still allow false passes, for example if foo had value 2 and bar had value 1, the CHECK-LABEL would match on the Value for bar and Name: foo would still precede it.

An alternative would be

CHECK-LABEL: Name: foo
CHECK: Value: 1{{$}}
CHECK-LABEL: Name:

which would constrain the CHECK to be between foo and whatever came next.

But, I don't think that's inherently better than

CHECK: Name: foo
CHECK: Value:
CHECK-SAME: {{ 1$}}

Both sequences will work, and whether to prefer LABEL or SAME depends partly on context of the output (is there a good LABEL candidate?) and the test itself (is this a single case or are we looking at a series?).

All this is to say, I think the motivation is okay as is.

[thopre]

In D68061#2178756, @probinson wrote:

@rupprecht if you are still motivated to land this after so long, the only thing I see missing is to update the example from CHECK-SAME: 1{{$}} to CHECK-SAME: {{ 1$}} and then it LGTM. If not, there seems to be enough interest that someone else can land it?

Done. Please shout if I made a mistake.

llvm/docs/CommandGuide/FileCheck.rst

.. code-block:: llvm
 
    !0 = !DILocation(line: 5, scope: !1, inlinedAt: !2)
 
    ; CHECK:       !DILocation(line: 5,
    ; CHECK-NOT:               column:
    ; CHECK-SAME:              scope: ![[SCOPE:[0-9]+]]
 
 "``CHECK-SAME:``" directives reject the input if there are any newlines between
-it and the previous directive.  A "``CHECK-SAME:``" cannot be the first
-directive in a file.
+it and the previous directive.
+
+"``CHECK-SAME:``" is also useful to avoid writing matchers for irrelevant
+fields. For example, suppose you're writing a test which parses a tool that
+generates output like this:
+
+.. code-block:: text
+
+   Name: foo
+   Field1: ...
+   Field2: ...
+   Field3: ...
+   Value: 1
+
+   Name: bar
+   Field1: ...
+   Field2: ...
+   Field3: ...
+   Value: 2
+
+   Name: baz
+   Field1: ...
+   Field2: ...
+   Field3: ...
+   Value: 1
+
+To write a test that verifies ``foo`` has the value ``1``, you might first
+write this:
+
+.. code-block:: text
+
+   CHECK: Name: foo
+   CHECK: Value: 1{{$}}
+
+However, this would be a bad test: if the value for ``foo`` changes, the test
+would still pass because the "``CHECK: Value: 1``" line would match the value
+from ``baz``. To fix this, you could add ``CHECK-NEXT`` matchers for every
+``FieldN:`` line, but that would be verbose, and need to be updated when
+``Field4`` is added. A more succint way to write the test using the
+"``CHECK-SAME:``" matcher would be as follows:
+
+.. code-block:: text
+
+   CHECK:      Name: foo
+   CHECK:      Value:
+   CHECK-SAME:        {{ 1$}}
+
+This verifies that the *next* time "``Value:``" appears in the ouput, it has
+the value ``1``.
+
+Note: a "``CHECK-SAME:``" cannot be the first directive in a file.
 
 The "CHECK-EMPTY:" directive
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 If you need to check that the next line has nothing on it, not even whitespace,
 you can use the "``CHECK-EMPTY:``" directive.
 
 .. code-block:: llvm