[FileCheck] Make FILECHECK_OPTS useful for its test suite

[jdenny]

Summary

Without this patch, FILECHECK_OPTS isn't propagated to FileCheck's
test suite so that FILECHECK_OPTS doesn't inadvertently affect test
results by affecting the output of FileCheck calls under test. As a
result, FILECHECK_OPTS is useless for debugging FileCheck's test
suite.

In llvm/test/FileCheck/lit.local.cfg, this patch provides a new
subsitution, %ProtectFileCheckOutput, to address this problem for
both FILECHECK_OPTS and the deprecated
FILECHECK_DUMP_INPUT_ON_FAILURE. The rest of the patch uses
%ProtectFileCheckOutput throughout the test suite

Fixes PR40284.

[probinson]

The lit part seems like a good thing. It's unfortunate that lit/test/lit.cfg needs to know about FileCheck environment variables, but them's the breaks. That bit should be its own patch, regardless of how we decide to handle FileCheck itself.

[jdenny]

In D65121#1597486, @probinson wrote:

The lit part seems like a good thing. It's unfortunate that lit/test/lit.cfg needs to know about FileCheck environment variables, but them's the breaks.

If this issue were present in more test suites, I'd be tempted to extend FileCheck with a command-line option to encapsulate this use case in some manner. As things are, that doesn't seem worth the trouble to me, but maybe others feel differently.

That bit should be its own patch, regardless of how we decide to handle FileCheck itself.

Sure, I'll separate out the lit change.

[probinson]

Regarding the FileCheck part, it should be split off to its own patch and the mass modification of FileCheck's test suite incorporated, because it's hard to grasp the effects without seeing those.

I have to say my initial reaction was "omg not another layer of indirection in the FileCheck test suite" but rereading PR40284, my own comments there have persuaded me to rethink that position. :-)

Reading the comments in FileCheck/lit.local.cfg, yes at some low level the point is whether varying the output affects the test, but I don't think that's a properly principled way to look at it. Also thinking that way is likely to cause brain damage in anyone coming along later and trying to decipher when to use which spelling of "run this tool."

Better to look at it this way (this is a multi-paragraph argument, sorry): This is a tool, these are its tests, and we want to run the tool and then FileCheck its output. This is how we test llvm-objdump and llvm-dwarfdump and all the rest of those tools. By a bizarre quirk of fate, the command to run the tool itself is spelled the same as FileCheck. So, let's change the spelling of "run this tool" because we'd like to keep the RUN lines following a pattern of toolname | FileCheck. We can call it something like %FileCheckUnderTest or %TestFileCheck or something else that implies "run the FileCheck I'm trying to test." (This would happen to be implemented as "run FileCheck with no environment.") Then your basic test of FileCheck would have a RUN line something like RUN: %TestFileCheck blah | FileCheck %s and it should be clear to the easily-trained eye which invocation of FileCheck is fulfilling which role. Importantly the *choice* of which spelling to use is based solely on the role within the test RUN line, not some more complicated "does it affect the test result" decision.

Sadly there are a few cases where a test actually does want to set an environment variable, and in those cases the above substitution will not do the right thing. Apparently we want (or also want) a substitution that will clean out the environment but not actually run the tool, so that we can set new environment variables before running it. Maybe call it %CleanEnv. Then your basic FileCheck-with-environment test's RUN line would look something like %CleanEnv env FILECHECK_OPTS=-v FileCheck blah | FileCheck %s

At that point, %TestFileCheck can be considered shorthand for %CleanEnv FileCheck (although likely the substitution wouldn't actually be implemented that way).

WDYT?

[probinson]

In D65121#1597516, @jdenny wrote:

In D65121#1597486, @probinson wrote:

The lit part seems like a good thing. It's unfortunate that lit/test/lit.cfg needs to know about FileCheck environment variables, but them's the breaks.

If this issue were present in more test suites, I'd be tempted to extend FileCheck with a command-line option to encapsulate this use case in some manner. As things are, that doesn't seem worth the trouble to me, but maybe others feel differently.

Then every test that had this issue would have to be modified to add the option. I think it would have to be an environment variable that suppresses the other environment variables: FILECHECK_IGNORE_ENV=1 or something. Then test suites could add that in their config files, and tests would be written as normal.

Luckily, so far we don't seem to need that level of confusion.

[jdenny]

In D65121#1597540, @probinson wrote:

Regarding the FileCheck part, it should be split off to its own patch and the mass modification of FileCheck's test suite incorporated, because it's hard to grasp the effects without seeing those.

I have to say my initial reaction was "omg not another layer of indirection in the FileCheck test suite"

I've been sitting on this patch for a few weeks in part because I had the same concern. :-) But I ultimately convinced myself that this is worth it for debugging FileCheck's test suite, which is challenging.

We can call it something like %FileCheckUnderTest or %TestFileCheck or something else that implies "run the FileCheck I'm trying to test."

This is actually what I originally talked about too in some phab reviews somewhere. One of them led to PR40284. I even started to implement that approach and then became convinced it's actually more complicated than the approach in this patch. See below.

Importantly the *choice* of which spelling to use is based solely on the role within the test RUN line, not some more complicated "does it affect the test result" decision.

This is an interesting point. At first, I thought the same. Then I realized that I cannot easily look at a FileCheck call in a RUN line and determine which role it serves. I can easily determine whether FileCheck's stdout or stderr is redirected/piped for further examination, and then I should surely use %TestFileCheckOut.

Now you might argue that the person originally writing a test surely knows which FileCheck call serves which role. Moreover, if he uses your %FileCheckUnderTest correctly, then everyone in the future will know, and that might be helpful for understanding tests besides just protecting from environment variables.

I'm not optimistic that approach will always be followed correctly and avoid rot over time. Actually, I think my simple check is much easier to follow when looking back at old tests: if the stdout/stderr is examined, then use %TestFileCheckOut. (Hmm, maybe the documentation I wrote could be simpler.) And then I can even use FILECHECK_OPTS to debug some FileCheck calls that are under test.

Sadly there are a few cases where a test actually does want to set an environment variable, and in those cases the above substitution will not do the right thing. Apparently we want (or also want) a substitution that will clean out the environment but not actually run the tool, so that we can set new environment variables before running it. Maybe call it %CleanEnv. Then your basic FileCheck-with-environment test's RUN line would look something like %CleanEnv env FILECHECK_OPTS=-v FileCheck blah | FileCheck %s

I went down this path at one point too... even with my does-the-output-matter approach. That is, %TestFileCheckOut originally included the FileCheck call, so I had a second substitution too.

Then I tried to document it, and I thought about people trying to maintain these in tests. Ugh. Yet another quirky substitution to confuse people. That's when I decided one simple substitution with a simple rule that handles all cases is easier to understand. Anyone who knows how env works can figure out how to set FILECHECK_OPTS just by glancing at the %TestFileCheckOut implementation, and there are examples in the documentation for everyone else.

WDYT?

In summary, I understand every point you made as I thought all the same things as I was working through this. It was my attempt to implement and document that led me to believe my current approach is actually simpler and more useful. Hopefully I didn't become too close to the implementation along the way to see it clearly. :-/

In a moment, I'll update the patch so you can see the mass changes throughout FileCheck's test suite, and I'll simplify the documentation at little. Maybe that'll help.

[jdenny]

• Remove changes for lit's test suite (will post soon in another review).
• Simplify %TestFileCheckOut documentation some.
• Propagate %TestFileCheckOut throughout FileCheck's test suite.

[jdenny]

In D65121#1597553, @probinson wrote:

In D65121#1597516, @jdenny wrote:

In D65121#1597486, @probinson wrote:

The lit part seems like a good thing. It's unfortunate that lit/test/lit.cfg needs to know about FileCheck environment variables, but them's the breaks.

If this issue were present in more test suites, I'd be tempted to extend FileCheck with a command-line option to encapsulate this use case in some manner. As things are, that doesn't seem worth the trouble to me, but maybe others feel differently.

Then every test that had this issue would have to be modified to add the option.

I was thinking of a command-line option that prints something like -u FILECHECK_OPTS -u FILECHECK_DUMP_INPUT_ON_FAILURE. The idea is that lit/test/lit.cfg wouldn't need to maintain a list of variables.

I think it would have to be an environment variable that suppresses the other environment variables: FILECHECK_IGNORE_ENV=1 or something. Then test suites could add that in their config files, and tests would be written as normal.

Makes sense too.

Luckily, so far we don't seem to need that level of confusion.

Agreed. It's just one external test suite, and there's no evidence the list of variables is going to evolve much further. FILECHECK_OPTS is intended to be a catch-all. FILECHECK_DUMP_INPUT_ON_FAILURE is deprecated.

[thopre]

Hi,

Could you add a section FileCheck-specific substitutions section in llvm/docs/TestingGuide.rst to describe %TestFileCheckOut?

[jdenny]

In D65121#1597772, @thopre wrote:

Hi,

Hi Thomas.

Could you add a section FileCheck-specific substitutions section in llvm/docs/TestingGuide.rst to describe %TestFileCheckOut?

That seems like a good idea. We'll need to decide what goes there and what stays in llvm/test/FileCheck/lit.local.cfg.

I'll make a note to work on this after we reach consensus about the actual behavior.

[probinson]

Getting back to this after a long month on first-responder-to-bugs duty. I know you're trying to get the lit/env issue sorted out, but I had a half hour available to look at this.

It makes me very sad to see
RUN: not FileCheck blahblah | FileCheck yadayada
become

RUN: not %TestFileCheckOut \
RUN: FileCheck blahblah | FileCheck yadayada

Wouldn't it work just as well to say

RUN: %TestFileCheckOut \
RUN: not FileCheck blahblah | FileCheck yadayada

which keeps the not adjacent to the FileCheck (and incidentally sidesteps the "not runs env as an external command" problem).

llvm/test/FileCheck/lit.local.cfg

3

Okay, that seems pretty straightforward and easy to understand.

51

This will clearly have to be changed to solve the problems found after D65156. But it can wait until that is sorted out.

[jdenny]

In D65121#1610682, @probinson wrote:

Wouldn't it work just as well to say

RUN: %TestFileCheckOut \
RUN: not FileCheck blahblah | FileCheck yadayada

which keeps the not adjacent to the FileCheck (and incidentally sidesteps the "not runs env as an external command" problem).

Yes, I like that better too. Thanks!

llvm/test/FileCheck/check-count.txt

7–9

Instead of that, just add a preceding line with

; RUN: %TestFileCheckOut \

Likewise everywhere else.

See https://reviews.llvm.org/D65121#1610682.

[jdenny]

Rebase. Specify %TestFileCheckOut before not.

Still need to write the suggested documentation and deal with the "env -u" issue.

[jdenny]

Forgot to git add a change for that last update.

[probinson]

More bikeshedding: The new substitution could be called %TestFileCheckOutput (thus not abbreviating Output to Out) which helps it to be more self-documenting.
Or, it could be called %TestFileCheckEnv because its function is to set the environment, and it can optionally take an environment-variable definition as an argument. (Sorry if I already suggested this and we decided against.)
Everything else about this looks fine, LGTM once the substitution's name is settled.

[jdenny]

In D65121#1621579, @probinson wrote:

More bikeshedding: The new substitution could be called %TestFileCheckOutput (thus not abbreviating Output to Out) which helps it to be more self-documenting.

That works for me, and I'll go with that if there's no further discussion.

Or, it could be called %TestFileCheckEnv because its function is to set the environment, and it can optionally take an environment-variable definition as an argument. (Sorry if I already suggested this and we decided against.)

I like seeing Env there. I don't like dropping Output because I want the reminder that FileCheck output is the reason to use this substitution.

I want to say %SetEnvForTestingFileCheckOutput, but that's too much. Can we cram all that into something shorter?

Another possibility: %ProtectFileCheckOutput

Everything else about this looks fine, LGTM once the substitution's name is settled.

We have time. I want to see how things go with my related lit test suite work first, and there was a suggestion about documentation here that I haven't worked on.

Thanks again.

[probinson]

In D65121#1621616, @jdenny wrote:

Another possibility: %ProtectFileCheckOutput

Oh, I like that better.

[jdenny]

• Rebased.
• Renamed %TestFileCheckOut to %ProtectFileCheckOutput.
• Added documentation to llvm/docs/TestingGuide.rst.

This patch isn't ready to be pushed yet. FileCheck's test suite doesn't use lit's internal shell, and this patch introduces calls to env -u, which D65335 explains is not portable across external env implementations. One solution is to remove the use of env -u here. Another solution is to change FileCheck's test suite to use lit's internal shell. The latter solution is probably generally better for portability, but it won't help this patch without the patches I'm working on for lit's internal shell in order to push D65156 again. I'm inclined to wait on that anyway to be sure we've actually fully understood the trouble there.

[jdenny]

Now that D65156 and its many dependencies seem to have survived the bots, I'm finally returning to this patch.

This update rebases onto a recent master and adds %ProtectFileCheckOutput to a few new RUN lines that have since appeared in FileCheck's test suite.

As discussed at D65121#1637145, we need to decide whether to eliminate env -u uses in this patch or to switch FileCheck's test suite to lit's internal shell. Either looks easy to do, but I vote for the latter as it seems better for portability in the long term. It appears to just require this in llvm/test/FileCheck/lit.local.cfg (llvm/utils/lit/tests/lit.cfg already has this):

import lit
config.test_format = lit.formats.ShTest(execute_external=False)

What do people think? I'll probably make that a separate review if there are no immediate objections to the idea.

[jdenny]

It appears the last update before today did not add documentation to llvm/docs/TestingGuide.rst as I intended, but today's update did, so that's another new change to look at.

llvm/test/FileCheck/check-count.txt

 ;
 ; Basic error checking.
 ;
 
 this is something else
 
-; RUN: not FileCheck %s --input-file %s --check-prefix=CHECK-ERR1 2>&1 | FileCheck %s --check-prefix=ERRCOUNT1
+; RUN: %TestFileCheckOut \
+; RUN: not FileCheck %s --input-file %s --check-prefix=CHECK-ERR1 2>&1 \
+; RUN: | FileCheck %s --check-prefix=ERRCOUNT1
 CHECK-ERR1-COUNT-xx: this
 ERRCOUNT1: [[@LINE-1]]:18: error: invalid count in -COUNT specification on prefix 'CHECK-ERR1'
 
-; RUN: not FileCheck %s --input-file %s --check-prefix=CHECK-ERR2 2>&1 | FileCheck %s --check-prefix=ERRCOUNT2
+; RUN: %TestFileCheckOut \
+; RUN: not FileCheck %s --input-file %s --check-prefix=CHECK-ERR2 2>&1 \
+; RUN: | FileCheck %s --check-prefix=ERRCOUNT2
 CHECK-ERR2-COUNT-0x1: something
 ERRCOUNT2: [[@LINE-1]]:19: error: invalid count in -COUNT specification on prefix 'CHECK-ERR2'
 
-; RUN: not FileCheck %s --input-file %s --check-prefix=CHECK-ERR3 2>&1 | FileCheck %s --check-prefix=ERRCOUNT3
+; RUN: %TestFileCheckOut \
+; RUN: not FileCheck %s --input-file %s --check-prefix=CHECK-ERR3 2>&1 \
+; RUN: | FileCheck %s --check-prefix=ERRCOUNT3
 CHECK-ERR3-COUNT-100x: else
 ERRCOUNT3: [[@LINE-1]]:21: error: invalid count in -COUNT specification on prefix 'CHECK-ERR3'
 
-; RUN: not FileCheck %s --input-file %s --check-prefix=CHECK-ERR4 2>&1 | FileCheck %s --check-prefix=ERRCOUNT4
+; RUN: %TestFileCheckOut \
+; RUN: not FileCheck %s --input-file %s --check-prefix=CHECK-ERR4 2>&1 \
+; RUN: | FileCheck %s --check-prefix=ERRCOUNT4
 CHECK-ERR4-COUNT-0: else
 ERRCOUNT4: [[@LINE-1]]:19: error: invalid count in -COUNT specification on prefix 'CHECK-ERR4'
 
 ;
 ; Main functionality
 ;
 
 this is duplicate

 CHECK-CNTMANY-EMPTY:
 CHECK-CNTMANY-COUNT-16: {{^}}-many-
 CHECK-CNTMANY-EMPTY:
 
 ;
 ; Tests on mismatches:
 ;
 
-; RUN: not FileCheck %s --input-file %s --check-prefix=CHECK-MIS1 2>&1 | FileCheck %s --check-prefix=MISCOUNT1
+; RUN: %TestFileCheckOut \
+; RUN: not FileCheck %s --input-file %s --check-prefix=CHECK-MIS1 2>&1 \
+; RUN: | FileCheck %s --check-prefix=MISCOUNT1
 CHECK-MIS1-COUNT-3: this is duplicate
 CHECK-MIS1: {{^}}this is not duplicate
 MISCOUNT1: [[@LINE-1]]:13: error: CHECK-MIS1: expected string not found in input
 
-; RUN: not FileCheck %s --input-file %s --check-prefix=CHECK-MIS2 2>&1 | FileCheck %s --check-prefix=MISCOUNT2
+; RUN: %TestFileCheckOut \
+; RUN: not FileCheck %s --input-file %s --check-prefix=CHECK-MIS2 2>&1 \
+; RUN: | FileCheck %s --check-prefix=MISCOUNT2
 CHECK-MIS2-COUNT-10: {{^this is duplicate}}
 CHECK-MIS2: {{^}}this is not duplicate
 MISCOUNT2: [[@LINE-2]]:22: error: CHECK-MIS2-COUNT: expected string not found in input
 
-; RUN: not FileCheck %s --input-file %s --check-prefix=CHECK-MIS3 2>&1 | FileCheck %s --check-prefix=MISCOUNT3
+; RUN: %TestFileCheckOut \
+; RUN: not FileCheck %s --input-file %s --check-prefix=CHECK-MIS3 2>&1 \
+; RUN: | FileCheck %s --check-prefix=MISCOUNT3
 CHECK-MIS3-COUNT-5: this is duplicate
 CHECK-MIS3-EMPTY:
 CHECK-MIS3-LABEL: Many-label:
 CHECK-MIS3-EMPTY:
 CHECK-MIS3-COUNT-160: {{^}}-many-
 CHECK-MIS3-EMPTY:
 MISCOUNT3: [[@LINE-2]]:23: error: CHECK-MIS3-COUNT: expected string not found in input (17 out of 160)

llvm/test/FileCheck/check-empty-tag.txt

 ; basic functionality
 ; RUN: FileCheck %s --input-file %s --check-prefix=CHECK1
 foo
 
 bar
 CHECK1: foo
 CHECK1-EMPTY:
 CHECK1-NEXT: bar
 
 ; next line must be blank
-; RUN: not FileCheck %s --input-file %s --check-prefix=CHECK2A 2>&1 | FileCheck %s --check-prefix=CHECK2B
+; RUN: %TestFileCheckOut \
+; RUN: not FileCheck %s --input-file %s --check-prefix=CHECK2A 2>&1 \
+; RUN: | FileCheck %s --check-prefix=CHECK2B
 badger
 CHECK2A: badger
 CHECK2A-EMPTY:
 CHECK2B: CHECK2A-EMPTY: is not on the line after the previous match
 
 ; CHECK-EMPTY must have empty pattern
-; RUN: not FileCheck %s --input-file %s --check-prefix=CHECK3A 2>&1 | FileCheck %s --check-prefix=CHECK3B
+; RUN: %TestFileCheckOut \
+; RUN: not FileCheck %s --input-file %s --check-prefix=CHECK3A 2>&1 \
+; RUN: | FileCheck %s --check-prefix=CHECK3B
 CHECK3A: foo
 CHECK3A-EMPTY: this is not empty
 CHECK3B: found non-empty check string for empty check with prefix 'CHECK3A:'
 
 ; CHECK-EMPTY cannot be the first check
-; RUN: not FileCheck %s --input-file %s --check-prefix=CHECK4A 2>&1 | FileCheck %s --check-prefix=CHECK4B
+; RUN: %TestFileCheckOut \
+; RUN: not FileCheck %s --input-file %s --check-prefix=CHECK4A 2>&1 \
+; RUN: | FileCheck %s --check-prefix=CHECK4B
 CHECK4A-EMPTY:
 CHECK4B: found 'CHECK4A-EMPTY' without previous 'CHECK4A: line
 
 ; CHECK-EMPTY-NOT and CHECK-NOT-EMPTY rejected
-; RUN: not FileCheck %s --input-file %s --check-prefixes=CHECK5A 2>&1 | FileCheck %s --check-prefix=CHECK5C
-; RUN: not FileCheck %s --input-file %s --check-prefixes=CHECK5B 2>&1 | FileCheck %s --check-prefix=CHECK5C
+; RUN: %TestFileCheckOut \
+; RUN: not FileCheck %s --input-file %s --check-prefixes=CHECK5A 2>&1 \
+; RUN: | FileCheck %s --check-prefix=CHECK5C
+; RUN: %TestFileCheckOut \
+; RUN: not FileCheck %s --input-file %s --check-prefixes=CHECK5B 2>&1 \
+; RUN: | FileCheck %s --check-prefix=CHECK5C
 CHECK5A-EMPTY-NOT:
 CHECK5B-NOT-EMPTY:
 CHECK5C: unsupported -NOT combo on prefix 'CHECK5{{A|B}}'
 
 ; whitespace does not count as empty
-; RUN: not FileCheck %s --input-file %s --check-prefix=CHECK6A --match-full-lines 2>&1 | FileCheck %s --check-prefix=CHECK6B
+; RUN: %TestFileCheckOut \
+; RUN: not FileCheck %s --input-file %s --check-prefix=CHECK6A --match-full-lines \
+; RUN: 2>&1 | FileCheck %s --check-prefix=CHECK6B
 CHECK6A: the next line has spaces
 CHECK6A-EMPTY:
 CHECK6B: expected string not found in input
 
 ; ***don't add any further blank lines after this point***
 ; CHECK-EMPTY, like CHECK-NEXT, will report an error if the first matching
 ; line is not the line immediately following the previous check.
 the next line has spaces
   
 No newline at end of file

llvm/test/FileCheck/check-empty.txt

-// RUN: not FileCheck -check-prefix=FOO %s </dev/null 2>&1 | FileCheck -check-prefix=EMPTY-ERR %s
-// RUN: not FileCheck -check-prefix=NOFOO %s </dev/null 2>&1 | FileCheck -check-prefix=EMPTY-ERR %s
-// RUN: not FileCheck -check-prefix=FOO -allow-empty %s </dev/null 2>&1 | FileCheck -check-prefix=NO-EMPTY-ERR -check-prefix=NOT-FOUND %s
-// RUN: FileCheck -check-prefix=NOFOO -allow-empty %s </dev/null 2>&1 | FileCheck -allow-empty -check-prefix=NO-EMPTY-ERR %s
+// RUN: %TestFileCheckOut \
+// RUN: not FileCheck -check-prefix=FOO %s </dev/null 2>&1 \
+// RUN: | FileCheck -check-prefix=EMPTY-ERR %s
+//
+// RUN: %TestFileCheckOut \
+// RUN: not FileCheck -check-prefix=NOFOO %s </dev/null 2>&1 \
+// RUN: | FileCheck -check-prefix=EMPTY-ERR %s
+//
+// RUN: %TestFileCheckOut \
+// RUN: not FileCheck -check-prefix=FOO -allow-empty %s </dev/null 2>&1 \
+// RUN: | FileCheck -check-prefix=NO-EMPTY-ERR -check-prefix=NOT-FOUND %s
+//
+// RUN: %TestFileCheckOut \
+// RUN: FileCheck -check-prefix=NOFOO -allow-empty %s </dev/null 2>&1 \
+// RUN: | FileCheck -allow-empty -check-prefix=NO-EMPTY-ERR %s
 
 ; FOO: foo
 ; NOFOO-NOT: foo
 
 ; EMPTY-ERR: FileCheck error: '-' is empty.
 ; EMPTY-ERR-NEXT: FileCheck command line: {{.*}}{{F|f}}ile{{C|c}}heck{{.*}}-check-prefix={{.*}}FOO {{.*}}check-empty.txt
 ; NO-EMPTY-ERR-NOT: FileCheck error: '-' is empty.
 ; NOT-FOUND: error: FOO: expected string not found in input

llvm/test/FileCheck/check-label-dag.txt

-; RUN: not FileCheck -input-file %s %s 2>&1 | FileCheck --check-prefix=ERROR %s
+; RUN: %TestFileCheckOut not FileCheck -input-file %s %s 2>&1 \
+; RUN: | FileCheck --check-prefix=ERROR %s
 
 bar
 zed
 
 CHECK-LABEL: {{^}}bar
 CHECK-DAG: {{^}}foo
 CHECK-LABEL: {{^}}zed
 
 ERROR: error: CHECK-DAG: expected string not found in input
 ERROR-NEXT: CHECK-DAG: {{.....}}foo

llvm/test/FileCheck/check-label.txt

 ; RUN: FileCheck -input-file %s %s -check-prefix=CHECKOK
-; RUN: not FileCheck -input-file %s %s -check-prefix=CHECKFAIL 2>&1 | FileCheck %s -check-prefix=CHECKERROR
+;
+; RUN: %TestFileCheckOut \
+; RUN: not FileCheck -input-file %s %s -check-prefix=CHECKFAIL 2>&1 \
+; RUN: | FileCheck %s -check-prefix=CHECKERROR
 
 label0:
 a
 b
 
 label1:
 b
 c

llvm/test/FileCheck/check-multiple-prefixes-nomatch-2.txt

-; RUN: not FileCheck -input-file %s %s -check-prefix=FOO -check-prefix=BAR 2>&1 | FileCheck %s
-; RUN: not FileCheck -input-file %s %s -check-prefixes=FOO,BAR 2>&1 | FileCheck %s
-; RUN: not FileCheck -input-file %s %s -check-prefixes=BAR,FOO 2>&1 | FileCheck %s
+; RUN: %TestFileCheckOut \
+; RUN: not FileCheck -input-file %s %s -check-prefix=FOO -check-prefix=BAR \
+; RUN: 2>&1 | FileCheck %s
+;
+; RUN: %TestFileCheckOut \
+; RUN: not FileCheck -input-file %s %s -check-prefixes=FOO,BAR 2>&1 \
+; RUN: | FileCheck %s
+;
+; RUN: %TestFileCheckOut \
+; RUN: not FileCheck -input-file %s %s -check-prefixes=BAR,FOO 2>&1 \
+; RUN: | FileCheck %s
 
 fog
 bar
 ; _FOO not a valid check-line
 ; FOO: fo{{o}}
 ; BAR: ba{{r}}
 
 ; CHECK: {{error: FOO: expected string not found in input}}
 ; CHECK-NEXT: {{F}}OO: fo{{[{][{]o[}][}]}}

llvm/test/FileCheck/check-multiple-prefixes-nomatch.txt

-; RUN: not FileCheck -input-file %s %s -check-prefix=FOO -check-prefix=BAR 2>&1 | FileCheck %s
-; RUN: not FileCheck -input-file %s %s -check-prefixes=FOO,BAR 2>&1 | FileCheck %s
-; RUN: not FileCheck -input-file %s %s -check-prefixes=BAR,FOO 2>&1 | FileCheck %s
+; RUN: %TestFileCheckOut \
+; RUN: not FileCheck -input-file %s %s -check-prefix=FOO -check-prefix=BAR \
+; RUN: 2>&1 | FileCheck %s
+;
+; RUN: %TestFileCheckOut \
+; RUN: not FileCheck -input-file %s %s -check-prefixes=FOO,BAR 2>&1 \
+; RUN: | FileCheck %s
+;
+; RUN: %TestFileCheckOut \
+; RUN: not FileCheck -input-file %s %s -check-prefixes=BAR,FOO 2>&1 \
+; RUN: | FileCheck %s
 
 BAR
 bar
 foo
 ; BAR: ba{{z}}
 ; FOO: fo{{o}}
 
 ; CHECK: {{error: BAR: expected string not found in input}}
 ; CHECK-NEXT: {{B}}AR: ba{{[{][{]z[}][}]}}

llvm/test/FileCheck/check-not-diaginfo.txt

-; RUN: not FileCheck --input-file %s %s 2>&1 \
+; RUN: %TestFileCheckOut not FileCheck --input-file %s %s 2>&1 \
 ; RUN:   | FileCheck --strict-whitespace --check-prefix DIAG %s
 
 CHECK-NOT: test
 
 DIAG:           error: CHECK-NOT: excluded string found in input
 DIAG-NEXT:      CHECK-NOT: test
 DIAG-NEXT: {{^}}           ^{{$}}
 DIAG-NEXT:      note: found here
 DIAG-NEXT:      CHECK-NOT: test
 DIAG-NEXT: {{^}}           ^~~~{{$}}

llvm/test/FileCheck/check-prefixes.txt

 // RUN: FileCheck -check-prefix=ANOTHER-PREFIX -input-file %s %s
-// RUN: not FileCheck -check-prefix=PREFIX -input-file %s %s 2>&1 | FileCheck -check-prefix=CHECK-NONEXISTENT-PREFIX %s
+//
+// RUN: %TestFileCheckOut \
+// RUN: not FileCheck -check-prefix=PREFIX -input-file %s %s 2>&1 \
+// RUN: | FileCheck -check-prefix=CHECK-NONEXISTENT-PREFIX %s
 
 foobar
 ; ANOTHER-PREFIX: foobar
 
 ; We use regex to match the colon so that FileCheck won't think it is a check
 ; prefix.
 ; CHECK-NONEXISTENT-PREFIX: error: no check strings found with prefix 'PREFIX{{:}}'

llvm/test/FileCheck/dump-input-annotations.txt

 ; RUN: echo 'hello world' > %t.in
 ; RUN: echo 'goodbye' >> %t.in
 ; RUN: echo 'world' >> %t.in
 ; RUN: echo 'unicorn' >> %t.in
 
 ; RUN: echo 'CHECK: hello' > %t.chk
 ; RUN: echo 'CHECK: universe' >> %t.chk
 
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk -v 2>&1 \
 ; RUN: | FileCheck -strict-whitespace -match-full-lines -check-prefix=ALIGN \
 ; RUN:             -implicit-check-not='remark:' %s
 
 ; Verbose diagnostics are suppressed but not errors.
 ; ALIGN:{{.*}}error:{{.*}}
 ; ALIGN:{{.*}}possible intended match here{{.*}}
 

 
 ; RUN: echo 'hello'   > %t.in
 ; RUN: echo 'again'   >> %t.in
 ; RUN: echo 'whirled' >> %t.in
 
 ; RUN: echo 'CHECK: hello' > %t.chk
 ; RUN: echo 'CHECK: world' >> %t.chk
 
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefix=CHK \
 ; RUN:             -implicit-check-not='remark:'
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk -v 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefixes=CHK,CHK-V \
 ; RUN:             -implicit-check-not='remark:'
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk -vv 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefixes=CHK,CHK-V \
 ; RUN:             -implicit-check-not='remark:'
 
 ; Verbose diagnostics are suppressed but not errors.
 ; CHK: {{.*}}error:{{.*}}
 ; CHK: {{.*}}possible intended match here{{.*}}
 

 ; Good match and no match.
 
 ; RUN: echo 'pete'   > %t.in
 ; RUN: echo 'repete' >> %t.in
 ; RUN: echo 'repeat' >> %t.in
 
 ; RUN: echo 'CHECK-COUNT-3: pete' > %t.chk
 
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefixes=CNT,CNT-Q \
 ; RUN:             -implicit-check-not='remark:'
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk -v 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefixes=CNT,CNT-V \
 ; RUN:             -implicit-check-not='remark:'
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk -vv 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefixes=CNT,CNT-V \
 ; RUN:             -implicit-check-not='remark:'
 
 ; Verbose diagnostics are suppressed but not errors.
 ; CNT: {{.*}}error:{{.*}}
 
 ; CNT:          <<<<<<

 
 ; RUN: echo 'hello' > %t.in
 ; RUN: echo 'again' >> %t.in
 
 ; RUN: echo 'CHECK: hello' > %t.chk
 ; RUN: echo 'CHECK-NEXT: again' >> %t.chk
 ; RUN: echo 'CHECK-NEXT: world' >> %t.chk
 
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefix=NXT \
 ; RUN:             -implicit-check-not='remark:'
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk -v 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefixes=NXT,NXT-V \
 ; RUN:             -implicit-check-not='remark:'
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk -vv 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefixes=NXT,NXT-V,NXT-VV \
 ; RUN:             -implicit-check-not='remark:'
 
 ; Verbose diagnostics are suppressed but not errors.
 ; NXT: {{.*}}error:{{.*}}
 
 ; NXT:        <<<<<<
 ; NXT-NEXT:            1: hello
 ; NXT-V-NEXT: check:1     ^~~~~
 ; NXT-NEXT:            2: again
 ; NXT-V-NEXT: next:2      ^~~~~
 ; NXT-NEXT:            3:
 ; NXT-NEXT:   next:3      X error: no match found
 ; NXT-NEXT:   >>>>>>
 ; NXT-NOT:    {{.}}
 
 ; Wrong-line match.
 
 ; RUN: echo 'yonder' >> %t.in
 ; RUN: echo 'world' >> %t.in
 
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefix=NXT2
 
 ; NXT2:      <<<<<<
 ; NXT2-NEXT:         1: hello
 ; NXT2-NEXT:         2: again
 ; NXT2-NEXT:         3: yonder
 ; NXT2-NEXT:         4: world

 ; Good match and no match.
 
 ; RUN: echo 'hello world!' > %t.in
 
 ; RUN: echo 'CHECK: hello' > %t.chk
 ; RUN: echo 'CHECK-SAME: world' >> %t.chk
 ; RUN: echo 'CHECK-SAME: again' >> %t.chk
 
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefix=SAM \
 ; RUN:             -implicit-check-not='remark:'
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk -v 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefixes=SAM,SAM-V \
 ; RUN:             -implicit-check-not='remark:'
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk -vv 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefixes=SAM,SAM-V,SAM-VV \
 ; RUN:             -implicit-check-not='remark:'
 
 ; Verbose diagnostics are suppressed but not errors.
 ; SAM: {{.*}}error:{{.*}}
 
 ; SAM:        <<<<<<
 ; SAM-NEXT:            1: hello world!
 ; SAM-V-NEXT: check:1     ^~~~~
 ; SAM-V-NEXT: same:2            ^~~~~
 ; SAM-NEXT:   same:3                 X error: no match found
 ; SAM-NEXT:   >>>>>>
 ; SAM-NOT:    {{.}}
 
 ; Wrong-line match.
 
 ; RUN: echo 'again' >> %t.in
 
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk -v 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefixes=SAM2 \
 ; RUN:             -implicit-check-not='remark:'
 
 ; Verbose diagnostics are suppressed but not errors.
 ; SAM2: {{.*}}error:{{.*}}
 
 ; SAM2:      <<<<<<

 ; RUN: echo 'world' >> %t.in
 ; RUN: echo 'label' >> %t.in
 
 ; RUN: echo 'CHECK: hello' > %t.chk
 ; RUN: echo 'CHECK-EMPTY:' >> %t.chk
 ; RUN: echo 'CHECK-EMPTY:' >> %t.chk
 ; RUN: echo 'CHECK-LABEL: label' >> %t.chk
 
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefix=EMP \
 ; RUN:             -implicit-check-not='remark:'
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk -v 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefixes=EMP,EMP-V \
 ; RUN:             -implicit-check-not='remark:'
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk -vv 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefixes=EMP,EMP-V,EMP-VV \
 ; RUN:             -implicit-check-not='remark:'
 
 ; Verbose diagnostics are suppressed but not errors.
 ; EMP: {{.*}}error:{{.*}}
 
 ; EMP:        <<<<<<

 ; Wrong-line match.
 
 ; RUN: echo 'hello' > %t.in
 ; RUN: echo 'world' >> %t.in
 
 ; RUN: echo 'CHECK: hello' > %t.chk
 ; RUN: echo 'CHECK-EMPTY:' >> %t.chk
 
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefix=EMP2 \
 ; RUN:             -implicit-check-not='remark:'
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk -v 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefixes=EMP2,EMP2-V \
 ; RUN:             -implicit-check-not='remark:'
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk -vv 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefixes=EMP2,EMP2-V,EMP2-VV \
 ; RUN:             -implicit-check-not='remark:'
 
 ; Verbose diagnostics are suppressed but not errors.
 ; EMP2: {{.*}}error:{{.*}}
 
 ; EMP2:        <<<<<<

 
 ; RUN: echo 'hello' > %t.in
 ; RUN: echo 'world' >> %t.in
 ; RUN: echo 'again' >> %t.in
 
 ; RUN: echo 'CHECK-NOT: goodbye' > %t.chk
 ; RUN: echo 'CHECK-NOT: world' >> %t.chk
 
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefix=NOT \
 ; RUN:             -implicit-check-not='remark:'
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk -v 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefixes=NOT,NOT-V \
 ; RUN:             -implicit-check-not='remark:'
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk -vv 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefixes=NOT,NOT-V,NOT-VV \
 ; RUN:             -implicit-check-not='remark:'
 
 ; Verbose diagnostics are suppressed but not errors.
 ; NOT: {{.*}}error:{{.*}}
 
 ; NOT:         <<<<<<
 ; NOT-NEXT:           1: hello
 ; NOT-VV-NEXT: not:1     X~~~~
 ; NOT-NEXT:           2: world
 ; NOT-VV-NEXT: not:1     ~~~~~
 ; NOT-NEXT:    not:2     !~~~~ error: no match expected
 ; NOT-NEXT:           3: again
 ; NOT-VV-NEXT: not:1     ~~~~~
 ; NOT-VV-NEXT:        4:
 ; NOT-VV-NEXT: eof:2     ^
 ; NOT-NEXT:    >>>>>>
 ; NOT-NOT:     {{.}}
 
 ; Again, but with a CHECK instead of EOF as search range end.
 
 ; RUN: echo 'CHECK: ain' >> %t.chk
 
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefix=NOT2 \
 ; RUN:             -implicit-check-not='remark:'
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk -v 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefixes=NOT2,NOT2-V \
 ; RUN:             -implicit-check-not='remark:'
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk -vv 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefixes=NOT2,NOT2-V,NOT2-VV \
 ; RUN:             -implicit-check-not='remark:'
 
 ; Verbose diagnostics are suppressed but not errors.
 ; NOT2: {{.*}}error:{{.*}}
 
 ; NOT2:         <<<<<<

 
 ; Prefixes used here:
 ; DAG    = quiet, -v, or -vv
 ; DAG-Q  = quiet
 ; DAG-V  = -v or -vv (-vv implies -v)
 ; DAG-VQ = -v and not -vv
 ; DAG-VV = -vv
 
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefixes=DAG,DAG-Q \
 ; RUN:             -implicit-check-not='remark:'
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk -v 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefixes=DAG,DAG-V,DAG-VQ \
 ; RUN:             -implicit-check-not='remark:'
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk -vv 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefixes=DAG,DAG-V,DAG-VV \
 ; RUN:             -implicit-check-not='remark:'
 
 ; Verbose diagnostics are suppressed but not errors.
 ; DAG: {{.*}}error:{{.*}}
 
 ; DAG:         <<<<<<

 ; RUN: echo 'foo' >> %t.in
 ; RUN: echo 'lab1' >> %t.in
 ; RUN: echo 'bar' >> %t.in
 
 ; RUN: echo 'CHECK-LABEL: lab0' > %t.chk
 ; RUN: echo 'CHECK: foo' >> %t.chk
 ; RUN: echo 'CHECK-LABEL: lab2' >> %t.chk
 
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefixes=LAB \
 ; RUN:             -implicit-check-not='remark:'
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk -v 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefixes=LAB,LAB-V \
 ; RUN:             -implicit-check-not='remark:'
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -dump-input=always -input-file %t.in %t.chk -vv 2>&1 \
 ; RUN: | FileCheck -match-full-lines %s -check-prefixes=LAB,LAB-V,LAB-VV \
 ; RUN:             -implicit-check-not='remark:'
 
 ; Verbose diagnostics are suppressed but not errors.
 ; LAB: {{.*}}error:{{.*}}
 ; LAB: {{.*}}possible intended match{{.*}}
 

llvm/test/FileCheck/dump-input-enable.txt

 
 ; RUN: echo 'CHECK: hello' > %t.check
 ; RUN: echo 'CHECK-NEXT: world' >> %t.check
 
 ;--------------------------------------------------
 ; Check -dump-input=<bad value>.
 ;--------------------------------------------------
 
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -input-file %t.good %t.check -check-prefix=CHECK \
 ; RUN:               -match-full-lines -dump-input=foobar 2>&1 \
 ; RUN: | FileCheck %s -match-full-lines -check-prefix=BADVAL
 
 ; No positional arg.
-; RUN: not FileCheck -dump-input=foobar 2>&1 \
+; RUN: %TestFileCheckOut not FileCheck -dump-input=foobar 2>&1 \
 ; RUN: | FileCheck %s -match-full-lines -check-prefix=BADVAL
 
 BADVAL: {{F|f}}ile{{C|c}}heck{{.*}}: for the --dump-input option: Cannot find option named 'foobar'!
 
 ;--------------------------------------------------
 ; Check -dump-input=help.
 ;--------------------------------------------------
 
 ; Appended to normal command line.
+; RUN: %TestFileCheckOut \
 ; RUN: FileCheck -input-file %t.err -color %t.check -dump-input=help \
 ; RUN: | FileCheck %s -check-prefix=HELP
 
 ; No positional arg.
-; RUN: FileCheck -dump-input=help | FileCheck %s -check-prefix=HELP
+; RUN: %TestFileCheckOut FileCheck -dump-input=help \
+; RUN: | FileCheck %s -check-prefix=HELP
 
 HELP-NOT: {{.}}
 HELP: The following description was requested by -dump-input=help
 HELP: try{{.*}}-color
 HELP-NOT: {{.}}
 
 ;--------------------------------------------------
 ; Check -dump-input=never.
 ;
 ; Include the case without -v, which isn't covered elsewhere.
 ;--------------------------------------------------
 
+; RUN: %TestFileCheckOut \
 ; RUN: FileCheck -input-file %t.good %t.check -check-prefix=CHECK \
 ; RUN:           -match-full-lines -dump-input=never 2>&1 \
 ; RUN: | FileCheck %s -match-full-lines -allow-empty \
 ; RUN:             -check-prefixes=NOTRACE,NODUMP
 
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -input-file %t.err %t.check -check-prefix=CHECK \
 ; RUN:               -match-full-lines -dump-input=never 2>&1 \
 ; RUN: | FileCheck %s -match-full-lines -check-prefixes=NOTRACE,ERR,NODUMP
 
+; RUN: %TestFileCheckOut \
 ; RUN: FileCheck -input-file %t.good %t.check -check-prefix=CHECK \
 ; RUN:           -match-full-lines -dump-input=never -v 2>&1 \
 ; RUN: | FileCheck %s -match-full-lines -check-prefixes=TRACE,NODUMP
 
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -input-file %t.err %t.check -check-prefix=CHECK \
 ; RUN:               -match-full-lines -dump-input=never -v 2>&1 \
 ; RUN: | FileCheck %s -match-full-lines -check-prefixes=TRACE,ERR,NODUMP
 
 ;--------------------------------------------------
 ; Check no -dump-input, which defaults to never.
 ;--------------------------------------------------
 
+; RUN: %TestFileCheckOut \
 ; RUN: FileCheck -input-file %t.good %t.check -check-prefix=CHECK \
 ; RUN:           -match-full-lines -v 2>&1 \
 ; RUN: | FileCheck %s -match-full-lines -check-prefixes=TRACE,NODUMP
 
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -input-file %t.err %t.check -check-prefix=CHECK \
 ; RUN:               -match-full-lines -v 2>&1 \
 ; RUN: | FileCheck %s -match-full-lines -check-prefixes=TRACE,ERR,NODUMP
 
 ;--------------------------------------------------
 ; Check -dump-input=fail.
 ;
 ; Include the case without -v, which isn't covered elsewhere.
 ;--------------------------------------------------
 
+; RUN: %TestFileCheckOut \
 ; RUN: FileCheck -input-file %t.good %t.check -check-prefix=CHECK \
 ; RUN:           -match-full-lines -dump-input=fail 2>&1 \
 ; RUN: | FileCheck %s -match-full-lines -allow-empty \
 ; RUN:             -check-prefixes=NOTRACE,NODUMP
 
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -input-file %t.err %t.check -check-prefix=CHECK \
 ; RUN:               -match-full-lines -dump-input=fail 2>&1 \
 ; RUN: | FileCheck %s -match-full-lines -check-prefixes=NOTRACE,ERR,DUMP-ERR
 
+; RUN: %TestFileCheckOut \
 ; RUN: FileCheck -input-file %t.good %t.check -check-prefix=CHECK \
 ; RUN:           -match-full-lines -dump-input=fail -v 2>&1 \
 ; RUN: | FileCheck %s -match-full-lines -allow-empty \
 ; RUN:             -check-prefixes=NOTRACE,NODUMP
 
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -input-file %t.err %t.check -check-prefix=CHECK \
 ; RUN:               -match-full-lines -dump-input=fail -v 2>&1 \
 ; RUN: | FileCheck %s -match-full-lines \
 ; RUN:                -check-prefixes=NOTRACE,ERR,DUMP-ERR,DUMP-ERR-V
 
 ;--------------------------------------------------
 ; Check -dump-input-on-failure.
 ;--------------------------------------------------
 
+; RUN: %TestFileCheckOut \
 ; RUN: FileCheck -input-file %t.good %t.check -check-prefix=CHECK \
 ; RUN:           -match-full-lines -dump-input-on-failure -v 2>&1 \
 ; RUN: | FileCheck %s -match-full-lines -allow-empty \
 ; RUN:             -check-prefixes=NOTRACE,NODUMP
 
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -input-file %t.err %t.check -check-prefix=CHECK \
 ; RUN:               -match-full-lines -dump-input-on-failure -v 2>&1 \
 ; RUN: | FileCheck %s -match-full-lines \
 ; RUN:                -check-prefixes=NOTRACE,ERR,DUMP-ERR,DUMP-ERR-V
 
-; RUN: env FILECHECK_DUMP_INPUT_ON_FAILURE=1 \
+; RUN: %TestFileCheckOut FILECHECK_DUMP_INPUT_ON_FAILURE=1 \
 ; RUN: FileCheck -input-file %t.good %t.check -check-prefix=CHECK \
 ; RUN:           -match-full-lines -v 2>&1 \
 ; RUN: | FileCheck %s -match-full-lines -allow-empty \
-; RUN:             -check-prefixes=NOTRACE,NODUMP
+; RUN:                -check-prefixes=NOTRACE,NODUMP
 
-; RUN: env FILECHECK_DUMP_INPUT_ON_FAILURE=1 \
+; RUN: %TestFileCheckOut FILECHECK_DUMP_INPUT_ON_FAILURE=1 \
 ; RUN: not FileCheck -input-file %t.err %t.check -check-prefix=CHECK \
 ; RUN:               -match-full-lines -v 2>&1 \
 ; RUN: | FileCheck %s -match-full-lines \
 ; RUN:                -check-prefixes=NOTRACE,ERR,DUMP-ERR,DUMP-ERR-V
 
 ;--------------------------------------------------
 ; Check -dump-input=always.
 ;--------------------------------------------------
 
+; RUN: %TestFileCheckOut \
 ; RUN: FileCheck -input-file %t.good %t.check -check-prefix=CHECK \
 ; RUN:           -match-full-lines -dump-input=always -v 2>&1 \
 ; RUN: | FileCheck %s -match-full-lines -check-prefixes=NOTRACE,DUMP-OK
 
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -input-file %t.err %t.check -check-prefix=CHECK \
 ; RUN:               -match-full-lines -dump-input=always -v 2>&1 \
 ; RUN: | FileCheck %s -match-full-lines \
 ; RUN:                -check-prefixes=NOTRACE,ERR,DUMP-ERR,DUMP-ERR-V
 
 ; END.
 
 ;--------------------------------------------------

llvm/test/FileCheck/envvar-opts.txt

 ; Create a case that produces a simple diagnostic.
 ; RUN: echo foo > %t.in
 ; CHECK: foo
 ; CHECK: bar
 
-; RUN: env FILECHECK_OPTS= \
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck %s -input-file %t.in 2>&1 \
 ; RUN: | FileCheck -check-prefix QUIET %s
 
-; RUN: env FILECHECK_OPTS=-v \
+; RUN: %TestFileCheckOut FILECHECK_OPTS= \
+; RUN: not FileCheck %s -input-file %t.in 2>&1 \
+; RUN: | FileCheck -check-prefix QUIET %s
+
+; RUN: %TestFileCheckOut FILECHECK_OPTS=-v \
 ; RUN: not FileCheck %s -input-file %t.in 2>&1 \
 ; RUN: | FileCheck -check-prefix VERB %s
 
 ; QUIET-NOT: remark: {{CHECK}}: expected string found in input
 ; VERB:      remark: {{CHECK}}: expected string found in input

llvm/test/FileCheck/implicit-check-not.txt

 ; RUN: sed 's#^;.*##' %s | FileCheck -check-prefix=CHECK-PASS -implicit-check-not=warning: %s
-; RUN: sed 's#^;.*##' %s | not FileCheck -check-prefix=CHECK-FAIL1 -implicit-check-not=warning: %s 2>&1 | FileCheck %s -check-prefix CHECK-ERROR1
-; RUN: sed 's#^;.*##' %s | not FileCheck -check-prefix=CHECK-FAIL2 -implicit-check-not=warning: %s 2>&1 | FileCheck %s -check-prefix CHECK-ERROR2
-; RUN: sed 's#^;.*##' %s | not FileCheck -check-prefix=CHECK-FAIL3 -implicit-check-not=warning: %s 2>&1 | FileCheck %s -check-prefix CHECK-ERROR3
-; RUN: sed 's#^;.*##' %s | not FileCheck -check-prefix=CHECK-FAIL1 -implicit-check-not='{{aaa|bbb|ccc}}' %s 2>&1 | FileCheck %s -check-prefix CHECK-ERROR4
-; RUN: sed 's#^;.*##' %s | not FileCheck -check-prefix=CHECK-FAIL1 -implicit-check-not=aaa -implicit-check-not=bbb -implicit-check-not=ccc %s 2>&1 | FileCheck %s -check-prefix CHECK-ERROR5
-; RUN: sed 's#^;.*##' %s | not FileCheck -check-prefix=CHECK-FAIL2 -implicit-check-not=aaa -implicit-check-not=bbb -implicit-check-not=ccc %s 2>&1 | FileCheck %s -check-prefix CHECK-ERROR6
-; RUN: sed 's#^;.*##' %s | not FileCheck -check-prefix=CHECK-FAIL3 -implicit-check-not=aaa -implicit-check-not=bbb -implicit-check-not=ccc %s 2>&1 | FileCheck %s -check-prefix CHECK-ERROR7
+; RUN: sed 's#^;.*##' %s | %TestFileCheckOut not FileCheck -check-prefix=CHECK-FAIL1 -implicit-check-not=warning: %s 2>&1 | FileCheck %s -check-prefix CHECK-ERROR1
+; RUN: sed 's#^;.*##' %s | %TestFileCheckOut not FileCheck -check-prefix=CHECK-FAIL2 -implicit-check-not=warning: %s 2>&1 | FileCheck %s -check-prefix CHECK-ERROR2
+; RUN: sed 's#^;.*##' %s | %TestFileCheckOut not FileCheck -check-prefix=CHECK-FAIL3 -implicit-check-not=warning: %s 2>&1 | FileCheck %s -check-prefix CHECK-ERROR3
+; RUN: sed 's#^;.*##' %s | %TestFileCheckOut not FileCheck -check-prefix=CHECK-FAIL1 -implicit-check-not='{{aaa|bbb|ccc}}' %s 2>&1 | FileCheck %s -check-prefix CHECK-ERROR4
+; RUN: sed 's#^;.*##' %s | %TestFileCheckOut not FileCheck -check-prefix=CHECK-FAIL1 -implicit-check-not=aaa -implicit-check-not=bbb -implicit-check-not=ccc %s 2>&1 | FileCheck %s -check-prefix CHECK-ERROR5
+; RUN: sed 's#^;.*##' %s | %TestFileCheckOut not FileCheck -check-prefix=CHECK-FAIL2 -implicit-check-not=aaa -implicit-check-not=bbb -implicit-check-not=ccc %s 2>&1 | FileCheck %s -check-prefix CHECK-ERROR6
+; RUN: sed 's#^;.*##' %s | %TestFileCheckOut not FileCheck -check-prefix=CHECK-FAIL3 -implicit-check-not=aaa -implicit-check-not=bbb -implicit-check-not=ccc %s 2>&1 | FileCheck %s -check-prefix CHECK-ERROR7
 
 warning: aaa
 ; CHECK-PASS: warning: aaa
 ; CHECK-ERROR1: command line:1:22: error: CHECK-FAIL1-NOT: excluded string found in input
 ; CHECK-ERROR1-NEXT: -implicit-check-not='warning:'
 ; CHECK-ERROR1: note: found here
 ; CHECK-FAIL2: warning: aaa
 ; CHECK-FAIL3: warning: aaa

llvm/test/FileCheck/line-count.txt

 ; RUN: FileCheck -input-file %s %s
-; RUN: not FileCheck -check-prefix BAD1 -input-file %s %s 2>&1 | FileCheck -check-prefix ERR1 %s
-; RUN: not FileCheck -check-prefix BAD2 -input-file %s %s 2>&1 | FileCheck -check-prefix ERR2 %s
-; RUN: not FileCheck -check-prefix BAD3 -input-file %s %s 2>&1 | FileCheck -check-prefix ERR3 %s
-; RUN: not FileCheck -check-prefix BAD4 -input-file %s %s 2>&1 | FileCheck -check-prefix ERR4 %s
-; RUN: not FileCheck -check-prefix BAD5 -input-file %s %s 2>&1 | FileCheck -check-prefix ERR5 %s
-; RUN: not FileCheck -check-prefix BAD6 -input-file %s %s 2>&1 | FileCheck -check-prefix ERR6 %s
-; RUN: not FileCheck -check-prefix BAD7 -input-file %s %s 2>&1 | FileCheck -check-prefix ERR7 %s
-; RUN: not FileCheck -check-prefix BAD8 -input-file %s %s 2>&1 | FileCheck -check-prefix ERR8 %s
-; RUN: not FileCheck -check-prefix BAD9 -input-file %s %s 2>&1 | FileCheck -check-prefix ERR9 %s
-; RUN: not FileCheck -check-prefix BAD10 -input-file %s %s 2>&1 | FileCheck -check-prefix ERR10 %s
-; RUN: not FileCheck -check-prefix BAD11 -input-file %s %s 2>&1 | FileCheck -check-prefix ERR11 %s
+; RUN: %TestFileCheckOut not FileCheck -check-prefix BAD1 -input-file %s %s 2>&1 | FileCheck -check-prefix ERR1 %s
+; RUN: %TestFileCheckOut not FileCheck -check-prefix BAD2 -input-file %s %s 2>&1 | FileCheck -check-prefix ERR2 %s
+; RUN: %TestFileCheckOut not FileCheck -check-prefix BAD3 -input-file %s %s 2>&1 | FileCheck -check-prefix ERR3 %s
+; RUN: %TestFileCheckOut not FileCheck -check-prefix BAD4 -input-file %s %s 2>&1 | FileCheck -check-prefix ERR4 %s
+; RUN: %TestFileCheckOut not FileCheck -check-prefix BAD5 -input-file %s %s 2>&1 | FileCheck -check-prefix ERR5 %s
+; RUN: %TestFileCheckOut not FileCheck -check-prefix BAD6 -input-file %s %s 2>&1 | FileCheck -check-prefix ERR6 %s
+; RUN: %TestFileCheckOut not FileCheck -check-prefix BAD7 -input-file %s %s 2>&1 | FileCheck -check-prefix ERR7 %s
+; RUN: %TestFileCheckOut not FileCheck -check-prefix BAD8 -input-file %s %s 2>&1 | FileCheck -check-prefix ERR8 %s
+; RUN: %TestFileCheckOut not FileCheck -check-prefix BAD9 -input-file %s %s 2>&1 | FileCheck -check-prefix ERR9 %s
+; RUN: %TestFileCheckOut not FileCheck -check-prefix BAD10 -input-file %s %s 2>&1 | FileCheck -check-prefix ERR10 %s
+; RUN: %TestFileCheckOut not FileCheck -check-prefix BAD11 -input-file %s %s 2>&1 | FileCheck -check-prefix ERR11 %s
 13
 14 aaa
 15 bbb
 16 ccc
 17 CHECK: [[@LINE-3]] {{a}}aa
 18 CHECK: [[@LINE-3]] {{b}}bb
 19 CHECK: [[@LINE-3]] {{c}}cc
 20 foobar

 50 ERR9: line-count.txt:[[#@LINE-1]]:17: error: unsupported operation '*'
 51
 52 BAD10: [[@LINE-x]]
 53 ERR10: line-count.txt:[[#@LINE-1]]:19: error: invalid operand format 'x'
 54
 55 BAD11: [[@LINE-1x]]
 56 ERR11: line-count.txt:[[#@LINE-1]]:20: error: unexpected characters at end of expression 'x'
 57
+; RUN: %TestFileCheckOut \
 ; RUN: not FileCheck -check-prefix BAD12 -input-file %s %s 2>&1 \
 ; RUN:   | FileCheck -check-prefix ERR12 %s
-60
-61 BAD12: [[#@LINE-1]] NOT HERE
-62 ERR12: note: with "@LINE-1" equal to "60"
-63
-64 CHECK: [[#@LINE]] CHECK
-65 CHECK: [[# @LINE]] CHECK
-66 CHECK: [[# @LINE ]] CHECK
-67
-68 CHECK: [[#@LINE-1]]
-69 CHECK: [[# @LINE-1]] CHECK
-70 CHECK: [[# @LINE -1]] CHECK
-71 CHECK: [[# @LINE - 1]] CHECK
-72 CHECK: [[# @LINE - 1 ]] CHECK
+61
+62 BAD12: [[#@LINE-1]] NOT HERE
+63 ERR12: note: with "@LINE-1" equal to "61"
+64
+65 CHECK: [[#@LINE]] CHECK
+66 CHECK: [[# @LINE]] CHECK
+67 CHECK: [[# @LINE ]] CHECK
+68
+69 CHECK: [[#@LINE-1]]
+70 CHECK: [[# @LINE-1]] CHECK
+71 CHECK: [[# @LINE -1]] CHECK
+72 CHECK: [[# @LINE - 1]] CHECK
+73 CHECK: [[# @LINE - 1 ]] CHECK

llvm/test/FileCheck/lit.local.cfg

-# Unset environment variables that the FileCheck tests
-# expect not to be set.
-file_check_expected_unset_vars = [
-  'FILECHECK_DUMP_INPUT_ON_FAILURE',
-  'FILECHECK_OPTS',
-]
-
-for env_var in file_check_expected_unset_vars:
-  if env_var in config.environment:
-    lit_config.note('Removing {} from environment for FileCheck tests'.format(
-      env_var)
-    )
-    config.environment.pop(env_var)
+# %TestFileCheckOut should precede a FileCheck call if and only if the call's
+# textual output affects test results.  It's usually easy to tell: just look
+# for redirection or piping of the FileCheck call's stdout or stderr.
+#
+# Examples:
+#
+#   ; Test another program, using FileCheck to verify its textual output.
+#   ; Only FileCheck's exit status affects test results, so a bare FileCheck
+#   ; call is sufficient and more convenient for debugging.
+#   ;
+#   ; RUN: %t | FileCheck %s
+#   ; CHECK: {{[0-9]+\.0}}
+#
+#   ; Test FileCheck itself, but only examine its exit status, so a bare
+#   ; FileCheck call is still sufficient and more convenient for debugging.
+#   ;
+#   ; RUN: FileCheck -input-file %s %s
+#   ; CHECK: {{[0-9]+\.0}}
+#   ; 5.0
+#
+#   ; Check that the FileCheck trace is off by default.  The first FileCheck
+#   ; call's textual output affects test results, so it requires
+#   ; %TestFileCheckOut to be safe.
+#   ;
+#   ; RUN: %TestFileCheckOut FileCheck -input-file %s %s 2>&1 \
+#   ; RUN: | FileCheck -allow-empty -check-prefix QUIET %s
+#   ;
+#   ; CHECK: {{[0-9]+\.0}}
+#   ; 5.0
+#   ; QUIET-NOT: expected string found in input
+#
+#   ; Check that the FileCheck trace is on when FILECHECK_OPTS=-v.
+#   ; FILECHECK_OPTS must be set after %TestFileCheckOut, which clears
+#   ; FILECHECK_OPTS beforehand.
+#   ;
+#   ; RUN: %TestFileCheckOut FILECHECK_OPTS=-v \
+#   ; RUN: FileCheck -input-file %s %s 2>&1 \
+#   ; RUN: | FileCheck -check-prefix TRACE %s
+#   ;
+#   ; CHECK: {{[0-9]+\.0}}
+#   ; 5.0
+#   ; TRACE: expected string found in input
+#
+# %TestFileCheckOut's purpose is to ensure correct test results when developers
+# set FileCheck environment variables (e.g., FILECHECK_OPTS=-dump-input=fail)
+# to tweak the output of FileCheck while debugging tests.  If a developer sets
+# values that affect FileCheck's exit status (e.g.,
+# FILECHECK_OPTS=-strict-whitespace), he shouldn't be surprised that test
+# results throughout all test suites are affected.
+config.substitutions.append(('%TestFileCheckOut',
+    'env -u FILECHECK_OPTS -u FILECHECK_DUMP_INPUT_ON_FAILURE'))

llvm/test/FileCheck/match-full-lines.txt

+// RUN: %TestFileCheckOut \
 // RUN: not FileCheck -match-full-lines -input-file %s %s  2>&1 \
 // RUN:   | FileCheck --check-prefix=ERROR --implicit-check-not=error: %s
-// RUN: not FileCheck -match-full-lines -strict-whitespace -input-file %s %s  2>&1 \
-// RUN:   | FileCheck --check-prefix=ERROR-STRICT --check-prefix=ERROR --implicit-check-not=error: %s
+// RUN: %TestFileCheckOut \
+// RUN: not FileCheck -match-full-lines -strict-whitespace -input-file %s %s \
+// RUN: 2>&1 | FileCheck --check-prefix=ERROR-STRICT --check-prefix=ERROR \
+// RUN:                  --implicit-check-not=error: %s
 
 Label 1
 a line
 trailing whitespace   
 trailing more whitespace   
 Label 2
 a line
    leading whitespace

llvm/test/FileCheck/multiple-missing-prefixes.txt

 // RUN: FileCheck -check-prefix=ANOTHER-PREFIX -input-file %s %s
-// RUN: not FileCheck -check-prefix=PREFIX1 -check-prefix=PREFIX2 -input-file %s %s 2>&1 | FileCheck -strict-whitespace -check-prefix=CHECK-NONEXISTENT-PREFIX -check-prefix=ALSO-NONEXISTENT %s
-// RUN: not FileCheck -check-prefixes=PREFIX1,PREFIX2 -input-file %s %s 2>&1 | FileCheck -strict-whitespace -check-prefixes=CHECK-NONEXISTENT-PREFIX,ALSO-NONEXISTENT %s
+// RUN: %TestFileCheckOut not FileCheck -check-prefix=PREFIX1 -check-prefix=PREFIX2 -input-file %s %s 2>&1 | FileCheck -strict-whitespace -check-prefix=CHECK-NONEXISTENT-PREFIX -check-prefix=ALSO-NONEXISTENT %s
+// RUN: %TestFileCheckOut not FileCheck -check-prefixes=PREFIX1,PREFIX2 -input-file %s %s 2>&1 | FileCheck -strict-whitespace -check-prefixes=CHECK-NONEXISTENT-PREFIX,ALSO-NONEXISTENT %s
 
 foobar
 ; ANOTHER-PREFIX: foobar
 
 ; We use regex to match the colon so that FileCheck won't think it is a check
 ; prefix.
 ; CHECK-NONEXISTENT-PREFIX: error: no check strings found with prefixes 'PREFIX1{{:}}', 'PREFIX2{{:}}'

llvm/test/FileCheck/no-check-file.txt

-; RUN: not FileCheck 2>&1 | FileCheck %s
+; RUN: %TestFileCheckOut not FileCheck 2>&1 | FileCheck %s
 
 CHECK: <check-file> not specified

llvm/test/FileCheck/no-multi-suffixes.txt

-; RUN: not FileCheck -check-prefix=DAGNOT -input-file %s %s 2>&1 | FileCheck -check-prefix=ERROR -check-prefix=DAGNOT-ERROR %s
-; RUN: not FileCheck -check-prefix=NOTDAG -input-file %s %s 2>&1 | FileCheck -check-prefix=ERROR -check-prefix=NOTDAG-ERROR %s
-; RUN: not FileCheck -check-prefix=NEXTNOT -input-file %s %s 2>&1 | FileCheck -check-prefix=ERROR -check-prefix=NEXTNOT-ERROR %s
-; RUN: not FileCheck -check-prefix=NOTNEXT -input-file %s %s 2>&1 | FileCheck -check-prefix=ERROR -check-prefix=NOTNEXT-ERROR %s
-; RUN: not FileCheck -check-prefix=SAMENOT -input-file %s %s 2>&1 | FileCheck -check-prefix=ERROR -check-prefix=SAMENOT-ERROR %s
-; RUN: not FileCheck -check-prefix=NOTSAME -input-file %s %s 2>&1 | FileCheck -check-prefix=ERROR -check-prefix=NOTSAME-ERROR %s
+; RUN: %TestFileCheckOut not FileCheck -check-prefix=DAGNOT -input-file %s %s 2>&1 | FileCheck -check-prefix=ERROR -check-prefix=DAGNOT-ERROR %s
+; RUN: %TestFileCheckOut not FileCheck -check-prefix=NOTDAG -input-file %s %s 2>&1 | FileCheck -check-prefix=ERROR -check-prefix=NOTDAG-ERROR %s
+; RUN: %TestFileCheckOut not FileCheck -check-prefix=NEXTNOT -input-file %s %s 2>&1 | FileCheck -check-prefix=ERROR -check-prefix=NEXTNOT-ERROR %s
+; RUN: %TestFileCheckOut not FileCheck -check-prefix=NOTNEXT -input-file %s %s 2>&1 | FileCheck -check-prefix=ERROR -check-prefix=NOTNEXT-ERROR %s
+; RUN: %TestFileCheckOut not FileCheck -check-prefix=SAMENOT -input-file %s %s 2>&1 | FileCheck -check-prefix=ERROR -check-prefix=SAMENOT-ERROR %s
+; RUN: %TestFileCheckOut not FileCheck -check-prefix=NOTSAME -input-file %s %s 2>&1 | FileCheck -check-prefix=ERROR -check-prefix=NOTSAME-ERROR %s
 
 foo
 ; DAGNOT: foo
 ; DAGNOT-DAG-NOT: bar
 ; NOTDAG: foo
 ; NOTDAG-NOT-DAG: bar
 ; NEXTNOT: foo
 ; NEXTNOT-NEXT-NOT: bar

llvm/test/FileCheck/numeric-defines-diagnostics.txt

 ; Test incorrect syntax for -D# option is correctly diagnosed.
 
 ; Invalid variable name: starts with a digit.
-RUN: not FileCheck -D#10VALUE=10 --input-file %s %s 2>&1 \
+RUN: %TestFileCheckOut not FileCheck -D#10VALUE=10 --input-file %s %s 2>&1 \
 RUN:   | FileCheck %s --strict-whitespace --match-full-lines --check-prefix NUMERRCLIFMT
 
 NUMERRCLIFMT:Global defines:1:46: error: invalid variable name
 NUMERRCLIFMT-NEXT:Global define #1: #10VALUE=10 (parsed as: {{\[\[#10VALUE:10\]\]}})
     NUMERRCLIFMT-NEXT:                                             ^
 
 ; Invalid definition of pseudo variable.
-RUN: not FileCheck -D#@VALUE=10 --input-file %s %s 2>&1 \
+RUN: %TestFileCheckOut not FileCheck -D#@VALUE=10 --input-file %s %s 2>&1 \
 RUN:   | FileCheck %s --strict-whitespace --match-full-lines --check-prefix NUMERRCLIPSEUDO
 
 NUMERRCLIPSEUDO:Global defines:1:45: error: definition of pseudo numeric variable unsupported
 NUMERRCLIPSEUDO-NEXT:Global define #1: #@VALUE=10 (parsed as: {{\[\[#@VALUE:10\]\]}})
     NUMERRCLIPSEUDO-NEXT:                                            ^
 
 ; Invalid definition of an expression.
-RUN: not FileCheck -D#VALUE+2=10 --input-file %s %s 2>&1 \
+RUN: %TestFileCheckOut not FileCheck -D#VALUE+2=10 --input-file %s %s 2>&1 \
 RUN:   | FileCheck %s --strict-whitespace --match-full-lines --check-prefix NUMERRCLITRAIL
 
 NUMERRCLITRAIL:Global defines:1:51: error: unexpected characters after numeric variable name
 NUMERRCLITRAIL-NEXT:Global define #1: #VALUE+2=10 (parsed as: {{\[\[#VALUE\+2:10\]\]}})
      NUMERRCLITRAIL-NEXT:                                                  ^

llvm/test/FileCheck/numeric-defines.txt

 ; Test functionality of -D# option: numeric variables are defined to the right
 ; value and CHECK directives using them match as expected given the value set.
 
 RUN: FileCheck -D#NUMVAL1=8 -D#NUMVAL2='NUMVAL1 + 4' -check-prefixes CHECKNUM1,CHECKNUM2 -input-file %s %s
+
+RUN: %TestFileCheckOut \
 RUN: not FileCheck -D#NUMVAL1=7 -D#NUMVAL2=12 -check-prefix CHECKNUM1 -input-file %s %s 2>&1 \
 RUN:   | FileCheck %s --strict-whitespace -check-prefix NUMERRMSG1
+
+RUN: %TestFileCheckOut \
 RUN: not FileCheck -D#NUMVAL1=8 -D#NUMVAL2=8 -check-prefix CHECKNUM2 -input-file %s %s 2>&1 \
 RUN:   | FileCheck %s --strict-whitespace -check-prefix NUMERRMSG2
+
+RUN: %TestFileCheckOut \
 RUN: not FileCheck -D#NUMVAL1=8 -D#NUMVAL2=8 -check-prefix NUMNOT -input-file %s %s 2>&1 \
 RUN:   | FileCheck %s --strict-whitespace -check-prefixes NOT-NUMERRMSG1
+
+RUN: %TestFileCheckOut \
 RUN: not FileCheck -D#NUMVAL1=7 -D#NUMVAL2=12 -check-prefix NUMNOT -input-file %s %s 2>&1 \
 RUN:   | FileCheck %s --strict-whitespace -check-prefixes NOT-NUMERRMSG2
 RUN: FileCheck -D#NUMVAL1=7 -D#NUMVAL2=8 -check-prefixes NUMNOT -input-file %s %s
 
 Numeric value #1 = 8
 Numeric value #2 = 12
 CHECKNUM1: Numeric value #1 = [[#NUMVAL1]]
 CHECKNUM2: Numeric value #2 = [[#NUMVAL2]]

llvm/test/FileCheck/numeric-expression.txt

 
 ; Empty numeric expression.
 EMPTY NUM EXPR
 foo 104 bar
 ; CHECK-LABEL: EMPTY NUM EXPR
 ; CHECK-NEXT: foo [[#]] bar
 
 ; Numeric expression using undefined variables.
+RUN: %TestFileCheckOut \
 RUN: not FileCheck --check-prefix UNDEF-USE --input-file %s %s 2>&1 \
 RUN:   | FileCheck --strict-whitespace --check-prefix UNDEF-USE-MSG %s
 
 UNDEF VAR USE
 UNDEFVAR: 11
 UNDEF-USE-LABEL: UNDEF VAR USE
 UNDEF-USE-NEXT: UNDEFVAR: [[#UNDEFVAR1+UNDEFVAR2]]
 UNDEF-USE-MSG: numeric-expression.txt:[[#@LINE-1]]:17: error: {{U}}NDEF-USE-NEXT: expected string not found in input
 UNDEF-USE-MSG-NEXT:      {{U}}NDEF-USE-NEXT: UNDEFVAR: {{\[\[#UNDEFVAR1\+UNDEFVAR2\]\]}}
 UNDEF-USE-MSG-NEXT:     {{^}}                ^{{$}}
 UNDEF-USE-MSG-NEXT:      numeric-expression.txt:[[#@LINE-6]]:1: note: scanning from here
 UNDEF-USE-MSG-NEXT:      UNDEFVAR: 11
 UNDEF-USE-MSG-NEXT: {{^}}^{{$}}
 UNDEF-USE-MSG-NEXT:      numeric-expression.txt:[[#@LINE-9]]:1: note: uses undefined variable(s): "UNDEFVAR1" "UNDEFVAR2"
 UNDEF-USE-MSG-NEXT:      UNDEFVAR: 11
 UNDEF-USE-MSG-NEXT: {{^}}^{{$}}
 
 ; Numeric expression with unsupported operator.
-RUN: not FileCheck -D#NUMVAR=10 --check-prefix INVAL-OP --input-file %s %s 2>&1 \
+RUN: %TestFileCheckOut not FileCheck -D#NUMVAR=10 --check-prefix INVAL-OP \
+RUN:                                 --input-file %s %s 2>&1 \
 RUN:   | FileCheck --strict-whitespace --check-prefix INVAL-OP-MSG %s
 
 INVALID OPERATOR
 NUMVAR*2: 22
 INVAL-OP-LABEL: INVALID OPERATOR
 INVAL-OP-NEXT: NUMVAR*2: [[#NUMVAR*2]]
 INVAL-OP-MSG: numeric-expression.txt:[[#@LINE-1]]:35: error: unsupported operation '*'
 INVAL-OP-MSG-NEXT: {{I}}NVAL-OP-NEXT: NUMVAR*2: {{\[\[#NUMVAR\*2\]\]}}
 INVAL-OP-MSG-NEXT:     {{^}}                                  ^{{$}}
 
 ; Name conflict between Numeric variable definition and string variable
 ; definition whether from the command-line or input text.
-RUN: not FileCheck --check-prefixes CONFLICT,CONFLICT1,CONFLICT2 --input-file %s %s 2>&1 \
+RUN: %TestFileCheckOut \
+RUN: not FileCheck --check-prefixes CONFLICT,CONFLICT1,CONFLICT2 \
+RUN:               --input-file %s %s 2>&1 \
 RUN:   | FileCheck --strict-whitespace --check-prefix INPUT-STR-CONFLICT %s
-RUN: not FileCheck -D#NUMVAR=42 --check-prefixes CONFLICT,CONFLICT2 --input-file %s %s 2>&1 \
+RUN: %TestFileCheckOut \
+RUN: not FileCheck -D#NUMVAR=42 --check-prefixes CONFLICT,CONFLICT2 \
+RUN:               --input-file %s %s 2>&1 \
 RUN:   | FileCheck --strict-whitespace --check-prefix INPUT-STR-CONFLICT %s
-RUN: not FileCheck -D#NUMVAR=42 -DNUMVAR=foobar --check-prefix CONFLICT --input-file %s %s 2>&1 \
+RUN: %TestFileCheckOut \
+RUN: not FileCheck -D#NUMVAR=42 -DNUMVAR=foobar --check-prefix CONFLICT \
+RUN:               --input-file %s %s 2>&1 \
 RUN:   | FileCheck --strict-whitespace --check-prefix CLI-STR-CONFLICT %s
-RUN: not FileCheck --check-prefixes CONFLICT,CONFLICT3,CONFLICT4 --input-file %s %s 2>&1 \
+RUN: %TestFileCheckOut \
+RUN: not FileCheck --check-prefixes CONFLICT,CONFLICT3,CONFLICT4 \
+RUN:               --input-file %s %s 2>&1 \
 RUN:   | FileCheck --strict-whitespace --check-prefix INPUT-NUM-CONFLICT %s
-RUN: not FileCheck -DSTRVAR=foobar --check-prefixes CONFLICT,CONFLICT4 --input-file %s %s 2>&1 \
+RUN: %TestFileCheckOut \
+RUN: not FileCheck -DSTRVAR=foobar --check-prefixes CONFLICT,CONFLICT4 \
+RUN:               --input-file %s %s 2>&1 \
 RUN:   | FileCheck --strict-whitespace --check-prefix INPUT-NUM-CONFLICT %s
-RUN: not FileCheck -DSTRVAR=foobar -D#STRVAR=42 --check-prefix CONFLICT --input-file %s %s 2>&1 \
+RUN: %TestFileCheckOut \
+RUN: not FileCheck -DSTRVAR=foobar -D#STRVAR=42 --check-prefix CONFLICT \
+RUN:               --input-file %s %s 2>&1 \
 RUN:   | FileCheck --strict-whitespace --check-prefix CLI-NUM-CONFLICT %s
 
 STRVAR NUMVAR CONFLICT
 redef1 42
 foobar
 redef2 42
 CONFLICT-LABEL: STRVAR NUMVAR CONFLICT
 CONFLICT1-NEXT: redef1 [[#NUMVAR:]]

 INPUT-NUM-CONFLICT: numeric-expression.txt:[[#@LINE-7]]:22: error: string variable with name 'STRVAR' already exists
  INPUT-NUM-CONFLICT-NEXT: CONFLICT4: redef2 {{\[\[#STRVAR:\]\]}}
 INPUT-NUM-CONFLICT-NEXT: {{^}}                     ^{{$}}
 CLI-NUM-CONFLICT: Global defines:2:45: error: string variable with name 'STRVAR' already exists
  CLI-NUM-CONFLICT-NEXT: Global define #2: #STRVAR=42 (parsed as: {{\[\[#STRVAR:42\]\]}})
 CLI-NUM-CONFLICT-NEXT: {{^}}                                            ^{{$}}
 
 ; Numeric variable definition with too big value.
+RUN: %TestFileCheckOut \
 RUN: not FileCheck --check-prefix BIGVAL --input-file %s %s 2>&1 \
 RUN:   | FileCheck --strict-whitespace --check-prefix BIGVAL-MSG %s
 
 BIG VALUE
 NUMVAR: 10000000000000000000000
 BIGVAL-LABEL: BIG VALUE
 BIGVAL-NEXT: NUMVAR: [[#NUMVAR:]]
 BIGVAL-MSG: numeric-expression.txt:[[#@LINE-3]]:9: error: Unable to represent numeric value
  BIGVAL-MSG-NEXT: {{N}}UMVAR: 10000000000000000000000
 BIGVAL-MSG-NEXT: {{^}}        ^{{$}}
 
 ; Verify that when a variable is set to an expression the expression is still
 ; checked.
+RUN: %TestFileCheckOut \
 RUN: not FileCheck --check-prefix DEF-EXPR-FAIL --input-file %s %s 2>&1 \
 RUN:   | FileCheck --strict-whitespace --check-prefix DEF-EXPR-FAIL-MSG %s
 
 DEF EXPR WRONG MATCH
 20
 43
 DEF-EXPR-FAIL-LABEL: DEF EXPR WRONG MATCH
 DEF-EXPR-FAIL-NEXT: [[# VAR20:]]
 DEF-EXPR-FAIL-NEXT: [[# VAR42: VAR20+22]]
 DEF-EXPR-FAIL-MSG: numeric-expression.txt:[[#@LINE-1]]:21: error: {{D}}EF-EXPR-FAIL-NEXT: is not on the line after the previous match
 DEF-EXPR-FAIL-MSG-NEXT: {{D}}EF-EXPR-FAIL-NEXT: {{\[\[# VAR42: VAR20\+22\]\]}}
 DEF-EXPR-FAIL-MSG-NEXT:   {{^}}                    ^{{$}}

llvm/test/FileCheck/opt-color.txt

 ; Create a case that produces a simple diagnostic.
 ; RUN: echo foo > %t.in
 ; CHECK: bar
 
 ; Run without and with -color.  In the former case, FileCheck should suppress
 ; color in its diagnostics because stderr is a file.
-; RUN: not FileCheck %s < %t.in 2> %t.no-color
-; RUN: not FileCheck -color %s < %t.in 2> %t.color
+; RUN: %TestFileCheckOut not FileCheck %s < %t.in 2> %t.no-color
+; RUN: %TestFileCheckOut not FileCheck -color %s < %t.in 2> %t.color
 
 ; Check whether color was produced.
 ; RUN: FileCheck -check-prefix NO-COLOR %s < %t.no-color
 ; RUN: FileCheck -check-prefix COLOR %s < %t.color
 
 ; Make sure our NO-COLOR and COLOR patterns are sane: they don't match the
 ; opposite cases.
 ; RUN: not FileCheck -check-prefix COLOR %s < %t.no-color
 ; RUN: not FileCheck -check-prefix NO-COLOR %s < %t.color
 
 ; I don't know of a good way to check for ANSI color codes, so just make sure
 ; some new characters show up where those codes should appear.
 ; NO-COLOR: : error: CHECK: expected string not found in input
 ; COLOR: : {{.+}}error: {{.+}}CHECK: expected string not found in input

llvm/test/FileCheck/string-defines-diagnostics.txt

 ; Test incorrect syntax for -D option is correctly diagnosed.
 
 ; Missing equal sign.
-RUN: not FileCheck -DVALUE10 --input-file %s %s 2>&1 \
+RUN: %TestFileCheckOut not FileCheck -DVALUE10 --input-file %s %s 2>&1 \
 RUN:   | FileCheck %s --check-prefix ERRCLIEQ1
 
 ERRCLIEQ1: Missing equal sign in command-line definition '-DVALUE10'
 
 ; Missing definition expression (variable name, equal sign and value).
-RUN: not FileCheck -D --input-file %s %s 2>&1 \
+RUN: %TestFileCheckOut not FileCheck -D --input-file %s %s 2>&1 \
 RUN:   | FileCheck %s --check-prefix ERRCLIEQ2
 
 ERRCLIEQ2: {{F|f}}ile{{C|c}}heck{{[^:]*}}: for the -D option: requires a value!
 
 ; Missing variable name.
-RUN: not FileCheck -D=10 --input-file %s %s 2>&1 \
+RUN: %TestFileCheckOut not FileCheck -D=10 --input-file %s %s 2>&1 \
 RUN:   | FileCheck %s --check-prefix ERRCLIVAR1
 
 ERRCLIVAR1: Missing variable name in command-line definition '-D=10'
 
 ; Missing variable name.
-RUN: not FileCheck -D= --input-file %s %s 2>&1 \
+RUN: %TestFileCheckOut not FileCheck -D= --input-file %s %s 2>&1 \
 RUN:   | FileCheck %s --check-prefix ERRCLIVAR2
 
 ERRCLIVAR2: Missing variable name in command-line definition '-D='
 
 ; Invalid variable name: starts with a digit.
-RUN: not FileCheck -D10VALUE=10 --input-file %s %s 2>&1 \
+RUN: %TestFileCheckOut not FileCheck -D10VALUE=10 --input-file %s %s 2>&1 \
 RUN:   | FileCheck %s --strict-whitespace --match-full-lines --check-prefix ERRCLIFMT
 
 ERRCLIFMT:Global defines:1:19: error: invalid variable name
 ERRCLIFMT-NEXT:Global define #1: 10VALUE=10
 ERRCLIFMT-NEXT:                  ^
 
 ; Invalid definition of pseudo variable.
-RUN: not FileCheck -D@VALUE=10 --input-file %s %s 2>&1 \
+RUN: %TestFileCheckOut not FileCheck -D@VALUE=10 --input-file %s %s 2>&1 \
 RUN:   | FileCheck %s --strict-whitespace --match-full-lines --check-prefix ERRCLIPSEUDO
 
 ERRCLIPSEUDO:Global defines:1:19: error: invalid name in string variable definition '@VALUE'
 ERRCLIPSEUDO-NEXT:Global define #1: @VALUE=10
 ERRCLIPSEUDO-NEXT:                  ^
 
 ; Invalid definition of an expression.
-RUN: not FileCheck -D'VALUE + 2=10' --input-file %s %s 2>&1 \
+RUN: %TestFileCheckOut not FileCheck -D'VALUE + 2=10' --input-file %s %s 2>&1 \
 RUN:   | FileCheck %s --strict-whitespace --check-prefix ERRCLITRAIL
 
 ERRCLITRAIL:Global defines:1:19: error: invalid name in string variable definition 'VALUE + 2'
 ERRCLITRAIL-NEXT:Global define #1: VALUE + 2=10
 ERRCLITRAIL-NEXT:                  ^

llvm/test/FileCheck/string-defines.txt

 ; Test functionality of -D option: string variables are defined to the right
 ; value and CHECK directives using them match as expected given the value set.
 
 ; String variable correctly defined to a non-empty string.
 RUN: FileCheck -DVALUE=10 --input-file %s %s
-RUN: not FileCheck -DVALUE=20 --input-file %s %s 2>&1 \
+RUN: %TestFileCheckOut not FileCheck -DVALUE=20 --input-file %s %s 2>&1 \
 RUN:   | FileCheck %s --check-prefix ERRMSG
-RUN: not FileCheck -DVALUE=10 --check-prefix NOT --input-file %s %s 2>&1 \
+RUN: %TestFileCheckOut not FileCheck -DVALUE=10 --check-prefix NOT \
+RUN:                                 --input-file %s %s 2>&1 \
 RUN:   | FileCheck %s --check-prefix NOT-ERRMSG
 RUN: FileCheck -DVALUE=20 --check-prefix NOT --input-file %s %s
 
 Value = 10
 CHECK: Value = [[VALUE]]
 NOT-NOT: Value = [[VALUE]]
 
 ERRMSG: defines.txt:[[@LINE-3]]:8: error: CHECK: expected string not found in input

llvm/test/FileCheck/validate-check-prefix.txt

-// RUN: not FileCheck -check-prefix=A! -input-file %s %s 2>&1 | FileCheck -check-prefix=BAD_PREFIX %s
+// RUN: %TestFileCheckOut not FileCheck -check-prefix=A! -input-file %s %s 2>&1 | FileCheck -check-prefix=BAD_PREFIX %s
 // RUN: FileCheck -check-prefix=A1a-B_c -input-file %s %s
-// RUN: not FileCheck -check-prefix=REPEAT -check-prefix=REPEAT -input-file %s %s 2>&1 | FileCheck -check-prefix=BAD_PREFIX %s
-// RUN: not FileCheck -check-prefix=VALID -check-prefix=A! -input-file %s %s 2>&1 | FileCheck -check-prefix=BAD_PREFIX %s
-// RUN: not FileCheck -check-prefix= -input-file %s %s 2>&1 | FileCheck -check-prefix=BAD_PREFIX %s
+// RUN: %TestFileCheckOut not FileCheck -check-prefix=REPEAT -check-prefix=REPEAT -input-file %s %s 2>&1 | FileCheck -check-prefix=BAD_PREFIX %s
+// RUN: %TestFileCheckOut not FileCheck -check-prefix=VALID -check-prefix=A! -input-file %s %s 2>&1 | FileCheck -check-prefix=BAD_PREFIX %s
+// RUN: %TestFileCheckOut not FileCheck -check-prefix= -input-file %s %s 2>&1 | FileCheck -check-prefix=BAD_PREFIX %s
 foobar
 ; A1a-B_c: foobar
 
 ; BAD_PREFIX: Supplied check-prefix is invalid! Prefixes must be
   unique and start with a letter and contain only alphanumeric characters, hyphens and underscores

llvm/test/FileCheck/var-scope.txt

 ; Test that variables not starting with dollar sign get undefined after a
 ; CHECK-LABEL directive iff --enable-var-scope is used.
 
 ; Reference run: variables remain defined at all time when not using
 ; --enable-var-scope option.
 RUN: FileCheck --check-prefixes CHECK,LOCAL3,GLOBAL --input-file %s %s
 
 RUN: FileCheck --check-prefixes CHECK,GLOBAL --enable-var-scope --input-file %s %s
-RUN: not FileCheck --check-prefixes CHECK,LOCAL1 --enable-var-scope --input-file %s %s 2>&1 \
+RUN: %TestFileCheckOut not FileCheck --check-prefixes CHECK,LOCAL1 --enable-var-scope --input-file %s %s 2>&1 \
 RUN:   | FileCheck --check-prefixes ERRUNDEF,ERRUNDEFLOCAL %s
-RUN: not FileCheck --check-prefixes CHECK,LOCAL2 --enable-var-scope --input-file %s %s 2>&1 \
+RUN: %TestFileCheckOut not FileCheck --check-prefixes CHECK,LOCAL2 --enable-var-scope --input-file %s %s 2>&1 \
 RUN:   | FileCheck --check-prefixes ERRUNDEF,ERRUNDEFLOCNUM %s
-RUN: not FileCheck --check-prefixes CHECK,LOCAL3 --enable-var-scope --input-file %s %s 2>&1 \
+RUN: %TestFileCheckOut not FileCheck --check-prefixes CHECK,LOCAL3 --enable-var-scope --input-file %s %s 2>&1 \
 RUN:   | FileCheck --check-prefixes ERRUNDEF,ERRUNDEFLOCAL,ERRUNDEFLOCNUM %s
 
 local1
 global1
 CHECK: [[LOCAL:loc[^[:digit:]]*]][[#LOCNUM:]]
 CHECK: [[$GLOBAL:glo[^[:digit:]]*]][[#$GLOBNUM:]]
 
 local2

llvm/test/FileCheck/verbose.txt

-; RUN: FileCheck -input-file %s %s 2>&1 | FileCheck -check-prefix QUIET --allow-empty %s
-; RUN: FileCheck -v -input-file %s %s 2>&1 | FileCheck --strict-whitespace -check-prefix V %s
-; RUN: FileCheck -vv -input-file %s %s 2>&1 | FileCheck --strict-whitespace -check-prefixes V,VV %s
+; RUN: %TestFileCheckOut FileCheck -input-file %s %s 2>&1 \
+; RUN: | FileCheck -check-prefix QUIET --allow-empty %s
+; RUN: %TestFileCheckOut FileCheck -v -input-file %s %s 2>&1 \
+; RUN: | FileCheck --strict-whitespace -check-prefix V %s
+; RUN: %TestFileCheckOut FileCheck -vv -input-file %s %s 2>&1 \
+; RUN: | FileCheck --strict-whitespace -check-prefixes V,VV %s
 
 foo
 bar
 CHECK: foo
 CHECK-NOT: raboof
 CHECK-NEXT: bar
 
 V:      verbose.txt:[[@LINE-4]]:8: remark: {{C}}HECK: expected string found in input