[lit] Drop "Script:", make -v and -a imply -vv

[jdenny]

Summary

This patch and D156954 were discussed in https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839.

Motivation: -a shows output from all tests, and -v shows output from just failed tests. Without this patch, that output from each test includes a section called "Script:", which includes all shell commands that lit has computed from RUN directives and will attempt to run for that test. The effect of -vv (which also implies -v if neither -a or -v is specified) is to extend that output with shell commands as they are executing so you can easily see which one failed.

For example, when using lit's internal shell and -vv:

Script:                                                                                                                 
--                                                                                                                      
: 'RUN: at line 1'; echo hello world                                                                                    
: 'RUN: at line 2'; 3c40 hello world                                                                                    
: 'RUN: at line 3'; echo hello world                                                                                    
--                                                                                                                      
Exit Code: 127                                                                                                          
                                                                                                                        
Command Output (stdout):                                                                                                
--                                                                                                                      
$ ":" "RUN: at line 1"                                                                                                  
$ "echo" "hello" "world"                                                                                                
# command output:                                                                                                       
hello world                                                                                                             
                                                                                                                        
$ ":" "RUN: at line 2"                                                                                                  
$ "3c40" "hello" "world"                                                                                                
# command stderr:                                                                                                       
'3c40': command not found                                                                                               
error: command failed with exit status: 127                                                                             
                                                                                                                        
--

Notice that all shell commands that actually execute appear in the output twice, once for "Script:" and once for -vv. Especially for tests with many RUN directives, the result is noisy. When searching through the output for a particular shell command, it is easy to get lost and mistake shell commands under "Script:" for shell commands that actually executed.

Change: With this patch, a test's output changes in two ways. First, the "Script:" section is never shown. Second, omitting -vv no longer disables printing of shell commands as they execute. That is, -a and -v imply -vv, and so -vv is deprecated as it is just an alias for -v.

Secondary motivation: We are also working to introduce a PYTHON directive, which can appear between RUN directives. How should PYTHON directives be represented in the "Script:" section, which has previously been just a shell script? We could probably think of something, but adding info about PYTHON directive execution in the -vv trace seems more straight-forward and more useful.

(This patch also removes a confusing point in the -vv documentation: at least when using bash as an external shell, -vv echoes commands to the shell's stderr not stdout.)

[awarzynski]

This is a very nice clean-up, thanks!

I've only skimmed through so far and just wanted to post a quick question about this:

Change: After this patch, there is no "Script:" section

IMO, it would be nice to be able to simply print the RUN lines that LIT has collected. Especially for complex tests for which RUN lines depend on various CMake flags (to help developing such tests). So, I would keep the "Script" section, but:

• only print it when explicitly requested (with a dedicated flag, e.g. -print-script),
• make it default to "collect the RUN lines, print them and exit" (so the tests wouldn't run).

Would this be difficult and would it make sense to you? Or perhaps that's already supported? :)

[awarzynski]

llvm/utils/lit/tests/Inputs/shtest-if-else/test.txt

9–12

IIUC, full RUN line would be printed here, right?

[jdenny]

In D154984#4491339, @awarzynski wrote:

This is a very nice clean-up, thanks!

I've only skimmed through so far and just wanted to post a quick question about this:

Thanks for taking a look.

Change: After this patch, there is no "Script:" section

IMO, it would be nice to be able to simply print the RUN lines that LIT has collected. Especially for complex tests for which RUN lines depend on various CMake flags (to help developing such tests).

In case it wasn't clear, with this patch, you do see the fully expanded RUN lines that actually execute, as in the trace of any shell language (or any that I've used).

If a RUN line fails, you don't see the remaining RUN lines that didn't execute. Are you saying you would like to see those too? In my usage, I have never wanted to because I always wanted to address the current failure and then retry before looking any farther in case there's an interaction. Can you help me to understand your use case a bit more?

So, I would keep the "Script" section, but:

• only print it when explicitly requested (with a dedicated flag, e.g. -print-script),
• make it default to "collect the RUN lines, print them and exit" (so the tests wouldn't run).

Would this be difficult and would it make sense to you? Or perhaps that's already supported? :)

This patch does not currently support that. It would not be difficult now but will require more thought if something like the PYTHON directive is accepted, especially if it grows the ability to affect substitutions and thus subsequent RUN lines.

[jdenny]

llvm/utils/lit/tests/Inputs/shtest-if-else/test.txt

9–12

Yes. Previously, this test was looking in the "Script:" section to verify correct expansion of %if. Now that "Script:" is gone, I changed it to just look for correct stdout from the executed command. Perhaps I should have changed it to look at the printed RUN line (from -vv)? What do you think? @asavonic might also want to comment, as he wrote the original test.

[Endill]

LGTM, but I'd wait for more approvals

[jdenny]

In D154984#4493485, @Endill wrote:

LGTM, but I'd wait for more approvals

Thanks for reviewing. I agree about waiting in case there are use cases for "Script:" I hadn't imagined.

[awarzynski]

In D154984#4491399, @jdenny wrote:

In case it wasn't clear, with this patch, you do see the fully expanded RUN lines that actually execute, as in the trace of any shell language (or any that I've used).

Yes, but together with all the other test output, which can be very noisy :(

If a RUN line fails, you don't see the remaining RUN lines that didn't execute. Are you saying you would like to see those too?

I meant something slightly different. Some RUN get very complicated - especially when adding {RE}DEFINE to the mix and e.g. emulators to run tests on. I would like to be able to simply check that LIT correctly interprets my RUN lines without running the tests (which can take some time to run and generate a lot of output). I think that that would be very useful. And looks like it's going to get harder to achieve :/

Btw, this patch changes the semantics of LIT. I think that it would be good post a short RFC on Discourse and to advertise this. Some people might rely on the current behavior. Or perhaps this has already been discussed somewhere?

llvm/utils/lit/tests/Inputs/shtest-if-else/test.txt

9–12

Hm, so this patch gets rid of "Script", but everything else remains intact, right. The changes in tests suggest that it;s a bit more intrusive, hence my original question.

[jdenny]

In D154984#4493743, @awarzynski wrote:

If a RUN line fails, you don't see the remaining RUN lines that didn't execute. Are you saying you would like to see those too?

I meant something slightly different. Some RUN get very complicated - especially when adding {RE}DEFINE to the mix and e.g. emulators to run tests on. I would like to be able to simply check that LIT correctly interprets my RUN lines without running the tests (which can take some time to run and generate a lot of output). I think that that would be very useful.

So I'm hearing there are two issues with using -vv-like traces for your use case:

1. They are noisy. I wonder if the real problem is that we just need a more readable format for the trace. For example, I wonder if indenting all stdout/stderr lines with "> " or something would help distinguish them from command lines. There's also the issue that we can easily control lit's internal shell, but external shells format the trace differently.
2. They take a long time to run. I don't know how to address that in the -vv trace. Indeed it seems you would need something like the "Script:" section.

And looks like it's going to get harder to achieve :/

Right, PYTHON directives make shell commands more dynamic, so lit cannot always fully expand the shell commands until it's time to execute them.

However, one simple possibility is for your proposed -print-script to print all RUN lines before the first PYTHON directive. If you don't use PYTHON directives (e.g., if the community rejects PYTHON directives), you'll get exactly what you want. If you do use PYTHON directives, you'll get only any initial RUN commands. Of course, we could also discuss printing "Script:" sections between PYTHON blocks as the test executes, but that runs into the execution time problem above.

Btw, this patch changes the semantics of LIT. I think that it would be good post a short RFC on Discourse and to advertise this. Some people might rely on the current behavior.

After the points you raise, I agree we should have an RFC before landing this patch. However, do you mind if we have more discussion here first? I think that would produce a better RFC.

Or perhaps this has already been discussed somewhere?

I don't recall anything.

[jdenny]

llvm/utils/lit/tests/Inputs/shtest-if-else/test.txt

9–12

I tried to update tests while maintaining their original focus (as well as I understood) and not clutter them with checking of the trace. I think the changes to shtest-run-at-line.py show the effect of this patch the best. However, look only at the internal shell checks. The external shell checks intentionally avoid showing most of the trace given that we have no control over the way they format traces.

Thanks for this feedback. I'm going to work on adding an example to the patch summary as well.

[jdenny]

llvm/utils/lit/tests/Inputs/shtest-if-else/test.txt

9–12

Hm, so this patch gets rid of "Script", but everything else remains intact, right.

Yes, that's right. It also includes the -vv trace any time -v or -a shows output for a test.

[ldionne]

In D154984#4493630, @jdenny wrote:

In D154984#4493485, @Endill wrote:

LGTM, but I'd wait for more approvals

Thanks for reviewing. I agree about waiting in case there are use cases for "Script:" I hadn't imagined.

This is non-blocking, but I do use Script: a lot to reproduce libc++ test failures. Here's an example:

FAIL: llvm-libc++-shared.cfg.in :: std/input.output/filesystems/fs.op.funcs/fs.op.proximate/proximate.pass.cpp (1 of 1)
******************** TEST 'llvm-libc++-shared.cfg.in :: std/input.output/filesystems/fs.op.funcs/fs.op.proximate/proximate.pass.cpp' FAILED ********************
Script:
--
: 'COMPILED WITH';  <TOOLCHAIN>/usr/bin/clang++ <LLVM>/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.proximate/proximate.pass.cpp -pthread -isysroot <SDK> --target=arm64-apple-darwin21.4.0 -nostdinc++ -I <LLVM>/build/default/include/c++/v1 -I <LLVM>/build/default/include/c++/v1 -I <LLVM>/libcxx/test/support -std=c++2b LOTS-OF-FLAGS -nostdlib++ -L <LLVM>/build/default/lib -Wl,-rpath,<LLVM>/build/default/lib -lc++ -o <LLVM>/build/default/test/std/Output/proximate.pass.cpp.dir/t.tmp.exe
: 'EXECUTED AS';  "python3.11" <LLVM>/libcxx/test/../utils/run.py --execdir <LLVM>/build/default/test/std/Output/proximate.pass.cpp.dir --  <LLVM>/build/default/test/std/Output/proximate.pass.cpp.dir/t.tmp.exe
--
Exit Code: 250

Command Output (stdout):
--
$ ":" "COMPILED WITH"
$ "<TOOLCHAIN>/usr/bin/clang++" "<LLVM>/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.proximate/proximate.pass.cpp" "-pthread" "-isysroot" "<SDK>" "--target=arm64-apple-darwin21.4.0" "-nostdinc++" "-I" "<LLVM>/build/default/include/c++/v1" "-I" "<LLVM>/build/default/include/c++/v1" "-I" "<LLVM>/libcxx/test/support" "-std=c++2b" LOTS-OF-FLAGS "-nostdlib++" "-L" "<LLVM>/build/default/lib" "-Wl,-rpath,<LLVM>/build/default/lib" "-lc++" "-o" "<LLVM>/build/default/test/std/Output/proximate.pass.cpp.dir/t.tmp.exe"
# command stderr:
ld: warning: object file (<LLVM>/build/default/lib/libc++experimental.a(memory_resource.cpp.o)) was built for newer macOS version (13.3) than being linked (13.0)
ld: warning: dylib (<LLVM>/build/default/lib/libc++.dylib) was built for newer macOS version (13.3) than being linked (13.0)

$ ":" "EXECUTED AS"
$ "python3.11" "<LLVM>/libcxx/test/../utils/run.py" "--execdir" "<LLVM>/build/default/test/std/Output/proximate.pass.cpp.dir" "--" "<LLVM>/build/default/test/std/Output/proximate.pass.cpp.dir/t.tmp.exe"
# command stderr:
Assertion failed: (!PathEq(output, expect)), function basic_test, file proximate.pass.cpp, line 151.

error: command failed with exit status: 250

There are two properties of Script that I like here:

1. It is not quoted. It is easier to copy-paste, modify and run into a terminal to reproduce the issue and iterate outside of Lit.
2. It shows all commands at once instead of showing only the ones that were executed, with output in between. If I want to know how to execute the program that was compiled above, I can easily use Script to figure that out. If I only have the Command Output, I have to search for EXECUTED AS through potentially a lot of output from the first command. And if the first command fails, then I will never know how this would get executed if this were to compile successfully.

I wouldn't say either of these issues should block this patch, however as-is this will cause a small quality-of-life regression for libc++ development. I don't have anything concrete to suggest here, so I'm fine with this, but if you can think of a way to retain either of these properties, it might be worth exploring.

[jdenny]

In D154984#4493934, @jdenny wrote:

2. They take a long time to run. I don't know how to address that in the -vv trace. Indeed it seems you would need something like the "Script:" section.

It just occurred to me that lit doesn't show the "Script:" section until the execution has terminated (lit prints it at the same time as the script's exit code). So this execution time issue is more of a feature request. It is not a regression that would be caused by this patch.

In D154984#4494259, @ldionne wrote:

This is non-blocking, but I do use Script: a lot to reproduce libc++ test failures. Here's an example:

Thanks for describing your use case.

There are two properties of Script that I like here:

1. It is not quoted. It is easier to copy-paste, modify and run into a terminal to reproduce the issue and iterate outside of Lit.

Are you saying you have to do some sort of fixup in order to copy-paste and run it in a terminal? Or are you saying the quoting is just annoying if you want to modify the command line? Either way, for lit's internal shell (which your example appears to be using), I'm sure we can find a way to reduce unnecessary quoting.

2. It shows all commands at once instead of showing only the ones that were executed, with output in between. If I want to know how to execute the program that was compiled above, I can easily use Script to figure that out. If I only have the Command Output, I have to search for EXECUTED AS through potentially a lot of output from the first command.

Would it be difficult in your workflow to send it through grep? Is there any way to format the -vv trace to make that easier?

And if the first command fails, then I will never know how this would get executed if this were to compile successfully.

I don't think we can address this issue generally with PYTHON directives, but it's still possible to print all leading RUN directives.

I wouldn't say either of these issues should block this patch, however as-is this will cause a small quality-of-life regression for libc++ development.

Well, my main motivation was to improve quality of life. For me, "Script:" has been noise for years, but obviously that's not true for everyone.

Would you be ok with @awarzynski's request that "Script:" is printed only with -print-script, which doesn't execute the test?

[MaskRay]

Thanks for the clean up. I think the new option (-a -v) semantics are more orthogonal. I always use -a or -vv and avoid -v.

My understanding of the behavior of $build/bin/llvm-lit -a test: for a succeeded test, all commands will be printed. For a failed test, all commands til the failed one will be printed.

I wonder whether we can change the Command Output (stderr): (pipe a|b is printed as + a\n+ b; for copy pasting, we need to remove the plus sign) to be like Script: (a|b is printed as is).
Perhaps it will fulfill @ldionne's request "This is non-blocking, but I do use Script: a lot to reproduce libc++ test failures. Here's an example:"

---

If changing Command Output (stderr): may lose functionality that someone may use, we can add a new option for the current + a\n+ b behavior (IMHO not useful). For dumped commands from -a and -v, I wish that they are copyable without needing to strip +.

[awarzynski]

In D154984#4493934, @jdenny wrote:

After the points you raise, I agree we should have an RFC before landing this patch. However, do you mind if we have more discussion here first? I think that would produce a better RFC.

Sure! Just to clarify my view:

• strong +1 to how the relation between -v/-a/-vv are cleaned-up in this patch,
• preserving "Script" would be helpful.

I rely on "Script" when debugging test set-ups. Specifically, MLIR integration tests (e.g. sparse_conversion.mlir) that can:

• run natively or via an emulator,
• use host or x-compiled lli (or mlir-cpu-runner),
• run 3-4 tools per RUN line (i.e. tool-1 | tool-2 | tool-3 | tool-4).

The resulting RUN lines tend to be quite complex. Having them summarised in one place helps.

For reference, this is what I get with "Script" (ToT LLVM):

: 'RUN: at line 10';  <WORKDIR>build/release/bin/mlir-opt <WORKDIR>/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_index.mlir --sparse-compiler=enable-runtime-library=true | <WORKDIR>/build/release/bin/mlir-cpu-runner -e entry -entry-point-result=void -shared-libs=<WORKDIR>/build/release/lib/libmlir_c_runner_utils.so | <WORKDIR>/build/release/bin/FileCheck <WORKDIR>/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_index.mlir

This can be c&p into a shell and will "just work". That's very useful.

This is what I get with -a with your patch:

+ <WORKDIR>/build/release/bin/mlir-opt <WORKDIR>/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_index.mlir --sparse-compiler=enable-runtime-library=true
+ <WORKDIR>/build/release/bin/mlir-cpu-runner -e entry -entry-point-result=void -shared-libs=<WORKDIR>/build/release/lib/libmlir_c_runner_utils.so
+ <WORKDIR>/build/release/bin/FileCheck <WORKDIR>/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_index.mlir

Sadly, this won't work in a shell without a bit of formatting. Could this be updated to match "Script"? If yes then that's not a big issue. I would still like for "Script" to be preserved, but that's not a blocker for me (I would still suggest checking on Discourse). Btw, IMHO it would be helpful to include the ideas proposed in https://reviews.llvm.org/D154987 in any potential RFC.

Last, but not least, thank you for caring about LIT and improving it for us - that's greatly appreciated!

[ldionne]

In D154984#4494549, @jdenny wrote:

In D154984#4493934, @jdenny wrote:

There are two properties of Script that I like here:

1. It is not quoted. It is easier to copy-paste, modify and run into a terminal to reproduce the issue and iterate outside of Lit.

Are you saying you have to do some sort of fixup in order to copy-paste and run it in a terminal? Or are you saying the quoting is just annoying if you want to modify the command line? Either way, for lit's internal shell (which your example appears to be using), I'm sure we can find a way to reduce unnecessary quoting.

It's just that the quoting is annoying. I'd be more than happy if we manage to reduce unnecessary quoting, or perhaps I just want to learn to live with it. After all, quoted commands are more copy-paste safe than unquoted ones.

2. It shows all commands at once instead of showing only the ones that were executed, with output in between. If I want to know how to execute the program that was compiled above, I can easily use Script to figure that out. If I only have the Command Output, I have to search for EXECUTED AS through potentially a lot of output from the first command.

Would it be difficult in your workflow to send it through grep? Is there any way to format the -vv trace to make that easier?

I'm usually just eyeballing at the Lit results, so yeah I don't really want to use grep for that. At the same time, our SCRIPTs are usually not too large, so I don't think this is a huge issue.

Would you be ok with @awarzynski's request that "Script:" is printed only with -print-script, which doesn't execute the test?

TBH, I'd love to avoid introducing another flag to control the output of Lit. This just seems like adding complexity to me. I think your proposed changes are better and more general, and they unblock useful feature work. I would go for the pure approach and then we can all learn to live with the new output results and improve them (without adding new flags) as we gain experience with that new output.

Again, just my .02 but I'd be fine if you didn't try to satisfy everyone's current usage. We'll learn to live with it and we'll improve the baseline output of the tool as we go, but let's not add complexity to the tool just for the sake of getting consensus here.

[jdenny]

@MaskRay and @awarzynski: Thanks for your comments. You both have pointed out that the -vv-style command trace breaks apart commands joined by | (and other command separators) so it's hard to copy-and-paste command lines to a terminal.

Which of the following would you prefer?

1. Keep "Script:". To address the redundancy problem that motivated this patch, disable "Script:" by default when just given -a or -v, and add some new command-line option to enable it (or maybe recycle -vv). To address the PYTHON line problem that motivated this patch, "Script:" would not include RUN lines beyond the first PYTHON line.

2. Extend lit's internal shell to print full shell command lines before executing them. Reconfigure tests suites that need this functionality to use lit's internal shell. (The examples you showed appear to be using external shells.) Perhaps improve the formatting of stdout/stderr to better distinguish it so shell commands are easier to find.

1 would be the easiest to implement now, but its solutions to the original motivations for this patch feel like a bad design with an inconsistent user interface.

2 requires more work, including adjusting some test suites that want to use it. However, if people can live with it, I feel like it's the cleanest path forward.

I started to propose a third alternative to handle external shells. However, I feel like we should encourage use of lit's internal shell for portability anyway, and we should avoid hacking around external shells to try to make them do what we want.

In D154984#4504368, @awarzynski wrote:

Last, but not least, thank you for caring about LIT and improving it for us - that's greatly appreciated!

You're welcome!

In D154984#4508213, @ldionne wrote:

TBH, I'd love to avoid introducing another flag to control the output of Lit. This just seems like adding complexity to me. I think your proposed changes are better and more general, and they unblock useful feature work. I would go for the pure approach and then we can all learn to live with the new output results and improve them (without adding new flags) as we gain experience with that new output.

Again, just my .02 but I'd be fine if you didn't try to satisfy everyone's current usage. We'll learn to live with it and we'll improve the baseline output of the tool as we go, but let's not add complexity to the tool just for the sake of getting consensus here.

Thanks for this opinion and your other answers. It sounds to me like your thinking is compatible with option 2 above... but with the added note that we don't have to perfect the trace format immediately.

[Endill]

In D154984#4508361, @jdenny wrote:

1. Keep "Script:". To address the redundancy problem that motivated this patch, disable "Script:" by default when just given -a or -v, and add some new command-line option to enable it (or maybe recycle -vv).

I can easily imagine people being confused that what was -vv is now -v, but -vv lives on with new semantics.

To address the PYTHON line problem that motivated this patch, "Script:" would not include RUN lines beyond the first PYTHON line.

This doesn't feel intuitive for users as well. More like an implementation detail leaking to user interface, which users will have to learn about.

Because of that, I'd discourage going with option 1 even as stopgap until landing another option.

[MaskRay]

In D154984#4508661, @Endill wrote:

In D154984#4508361, @jdenny wrote:

1. Keep "Script:". To address the redundancy problem that motivated this patch, disable "Script:" by default when just given -a or -v, and add some new command-line option to enable it (or maybe recycle -vv).

I can easily imagine people being confused that what was -vv is now -v, but -vv lives on with new semantics.

To address the redundancy problem, disabling "Script:" in -a and -v modes looks good to me. Keeping "Script:" functionality with -vv looks good if users need this feature.

I have had the habit to always use -vv instead of -v to know the failed command number. Some users may be like me.
Removing Script: from -v and reporting the failed commands will make the -v mode usable.

To address the PYTHON line problem that motivated this patch, "Script:" would not include RUN lines beyond the first PYTHON line.

This doesn't feel intuitive for users as well. More like an implementation detail leaking to user interface, which users will have to learn about.

Because of that, I'd discourage going with option 1 even as stopgap until landing another option.

2. Extend lit's internal shell to print full shell command lines before executing them. Reconfigure tests suites that need this functionality to use lit's internal shell.

Does this mean to print a pipe as one single command instead of two? Current output is:

Script:
--
: 'RUN: at line 1';   echo xxx | cat
: 'RUN: at line 2';   echo yyy | cat >&2
--
Exit Code: 0

Command Output (stdout):
--
$ ":" "RUN: at line 1"
$ "echo" "xxx"
# command output:
xxx

$ "cat"
$ ":" "RUN: at line 2"
$ "echo" "yyy"
# command output:
yyy

$ "cat"

I can see that preserving the pipe and redirection ($ "echo" "xxx" | "cat" 2>&1) will make the command copyable and possibly more useful.

I don't know how people feel about printing # command output: immediately after printing a command with the internal shell (LIT_USE_INTERNAL_SHELL=1).
It seems to get in the way if we want to replace Script:.

Regarding the external shell, it collects all stdout and stderr. I think just removing the + sign at the beginning of a command will make the output a replacement for Script:.

[jdenny]

In D154984#4508883, @MaskRay wrote:

2. Extend lit's internal shell to print full shell command lines before executing them. Reconfigure tests suites that need this functionality to use lit's internal shell.

Does this mean to print a pipe as one single command instead of two?

Yes, it would print command lines the way "Script:" did except it would print them right before executing them.

I don't know how people feel about printing # command output: immediately after printing a command with the internal shell (LIT_USE_INTERNAL_SHELL=1).
It seems to get in the way if we want to replace Script:.

I'm not sure I understand your meaning. How does it get in the way?

Regarding the external shell, it collects all stdout and stderr. I think just removing the + sign at the beginning of a command will make the output a replacement for Script:.

Do you know how to tell the external shells to change the way they format their trace output? Is there a reason test suites still need to use external shells instead of lit's more portable internal shell?

[MaskRay]

In D154984#4529966, @jdenny wrote:

In D154984#4508883, @MaskRay wrote:

2. Extend lit's internal shell to print full shell command lines before executing them. Reconfigure tests suites that need this functionality to use lit's internal shell.

Does this mean to print a pipe as one single command instead of two?

Yes, it would print command lines the way "Script:" did except it would print them right before executing them.

I don't know how people feel about printing # command output: immediately after printing a command with the internal shell (LIT_USE_INTERNAL_SHELL=1).
It seems to get in the way if we want to replace Script:.

I'm not sure I understand your meaning. How does it get in the way?

It gets in the way because if people want to copy all commands, they'd copy the command output as well.
And to differentiate command output from RUN lines , we have to print prompts like $ .

If the internal shell can collect Command Output (stdout): and stderr and print them after all RUN commands are dumped, we can remove the prompts.

Regarding the external shell, it collects all stdout and stderr. I think just removing the + sign at the beginning of a command will make the output a replacement for Script:.

Do you know how to tell the external shells to change the way they format their trace output? Is there a reason test suites still need to use external shells instead of lit's more portable internal shell?

I think many more developers work on non-Windows platforms and get external shell by default, and very few people know LIT_USE_INTERNAL_SHELL=1.

Command Output (stdout)
$ ...

If we can make the internal shell work well enough and make it the default for non-Windows platforms, that's fine with me.

For now, I mostly care about the behavior of the external behavior:)

[jdenny]

In D154984#4536987, @MaskRay wrote:

If the internal shell can collect Command Output (stdout): and stderr and print them after all RUN commands are dumped, we can remove the prompts.

I specifically want to see which command printed which part of the output. Separating them as you suggest would lose that information.

What if we put the output in shell comments so it has no effect when pasted into a terminal?

I think many more developers work on non-Windows platforms and get external shell by default, and very few people know LIT_USE_INTERNAL_SHELL=1.

I don't normally use that environment variable. Instead, I configure the test suite permanently with the following in its lit.cfg:

config.test_format = lit.formats.ShTest(execute_external=False)

If we can make the internal shell work well enough and make it the default for non-Windows platforms, that's fine with me.

I'm not sure we can change the lit default in general. Some test suites might still require external shell features that lit's internal shell doesn't support. Instead, we can change it per test suite, as shown above.

[awarzynski]

In D154984#4508361, @jdenny wrote:

@MaskRay and @awarzynski: Thanks for your comments. You both have pointed out that the -vv-style command trace breaks apart commands joined by | (and other command separators) so it's hard to copy-and-paste command lines to a terminal.

Which of the following would you prefer?

1. Keep "Script:". To address the redundancy problem that motivated this patch, disable "Script:" by default when just given -a or -v, and add some new command-line option to enable it (or maybe recycle -vv). To address the PYTHON line problem that motivated this patch, "Script:" would not include RUN lines beyond the first PYTHON line.

2. Extend lit's internal shell to print full shell command lines before executing them. Reconfigure tests suites that need this functionality to use lit's internal shell. (The examples you showed appear to be using external shells.) Perhaps improve the formatting of stdout/stderr to better distinguish it so shell commands are easier to find.

We should probably aim for Option 2 as the cleaner and more future-proof approach. Personally, I mostly care about being able to copy & paste the command. "Script" (in the output) is nice as it makes finding RUN lines easy, but it's definitely not a show-stopper.

I don't quite follow the "internal vs external shell" argument - could you explain? Why can't we make this work with both internal _and_ external shell?

[jdenny]

In D154984#4537086, @jdenny wrote:

What if we put the output in shell comments so it has no effect when pasted into a terminal?

Here's an example of what I mean, but we can certainly tweak more:

echo hello world
# command stdout
# hello world
echo goodbye world
# command stdout
# goodbye world

I found I can copy and paste that entire text into a terminal, and just the actual command lines execute, as desired.

In D154984#4539495, @awarzynski wrote:

Personally, I mostly care about being able to copy & paste the command. "Script" (in the output) is nice as it makes finding RUN lines easy, but it's definitely not a show-stopper.

Would the above work for you?

In D154984#4539495, @awarzynski wrote:

I don't quite follow the "internal vs external shell" argument - could you explain? Why can't we make this work with both internal _and_ external shell?

To execute RUN lines, lit can use its own internal shell, which is written in python. As lit developers, we have complete control over how it formats execution traces.

Alternatively, lit can use external shells, such as cmd for windows or bash/sh for unix. In those cases, lit uses the external shell's debug features, such as @echo on or set -x. I just checked the sh standard, and I now see that the PS4 variable controls the prefix (defaults to +) printed before a command. If the format we settle on requires no changes beyond that, then we're probably fine. However, I don't yet know how to prevent it from breaking apart pipelines into separate commands. I don't yet know how to prevent it from mixing the commands and their stderr (where it prints the commands). I don't yet know how to control windows.

[awarzynski]

In D154984#4539712, @jdenny wrote:

Here's an example of what I mean, but we can certainly tweak more:

Thanks for the example.

Would the above work for you?

Yes, that should work.

Alternatively, lit can use external shells, such as cmd for windows or bash/sh for unix.

Thanks for the explanation - now I understand what you were referring to.

I don't yet know how to prevent it from mixing the commands and their stderr (where it prints the commands).

I think that I have an example for that. For this "broken" version of RUN line from concatenate_dim_1.mlir:

<WORKDIR>build/release/bin/mlir-opt <WORKDIR>/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_index.mlir --sparse-compiler=enableruntime-library=true | <WORKDIR>/build/release/bin/mlir-cpu-runner -e entry -entry-point-result=void -shared-libs=<WORKDIR>/build/release/lib/libmlir_c_runner_utils.so | <WORKDIR>/build/release/bin/FileCheck <WORKDIR>/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_index.mlir

I get:

+ <WORKDIR>/build/release/bin/mlir-opt <WORKDIR>/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_index.mlir --sparse-compiler=enableruntime-library=true
+ <WORKDIR>/build/release/bin/mlir-cpu-runner -e entry -entry-point-result=void -shared-libs=<WORKDIR>/build/release/lib/libmlir_c_runner_utils.so
+ <WORKDIR>/build/release/bin/FileCheck <WORKDIR>/mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_index.mlir
<Pass-Options-Parser>: no such option enableruntime-library
Error: entry point not found
FileCheck error: '<stdin>' is empty.

This output is quite noisy - it's the 1st command that fails, but the error doesn't get printed until after the 3rd command from the RUN line. I am not aware of any way to improve this. One could use BASH_XTRACEFD to redirect the debug output, but I'm not convinced that that would be an improvement.

[jdenny]

In D154984#4540226, @awarzynski wrote:

Yes, that should work.

Thanks for confirming. If others here like that proposal and are ok with not addressing external shells, then I'll prototype it and start thinking about an RFC.

One could use BASH_XTRACEFD to redirect the debug output, but I'm not convinced that that would be an improvement.

It also seems limited to bash. For example, there is a path in lit where it uses /bin/sh, which on my linux box is dash, which is "POSIX-compliant", "small as possible", and not bash. And then there's also windows.

[jdenny]

Rebased.

[jdenny]

The requested RFC for D154984 and D156954: https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839

[jdenny]

The RFC has been up for a week with no significant activity for several days. The response was very positive. Thanks for all the feedback so far in this review, which led to D156954 and thus helped to make the RFC a success.

I saw no feedback in the RFC that would impact this patch. I think this patch is ready for a formal review/accept.

[MaskRay]

I've gone ahead and accepted the patch. I agree with https://reviews.llvm.org/D156954#4617746 , especially this sentence

However, people seemed fine to wait to do that in a separate patch if an actual user who cares about that use case speaks up.

It seems much easier to have this patch in place so that people can check the changes and possibly suggest future improvement.
Making -vv usable is a significant improvement to both external and internal lit shell workflows.

[MaskRay]

Consider linking to https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839 for this patch and D156954.

[jdenny]

In D154984#4617772, @MaskRay wrote:

Consider linking to https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839 for this patch and D156954.

Good idea. I'll make sure that's in their commit logs.

Thanks for reviewing. I'll plan to land these early next week when I can keep a closer eye on the bots.

[awarzynski]

https://reviews.llvm.org/D156954 addresses all my concerns raised here and from the Discourse RFC we know that the community is in favor of this change as well. Thanks for all the great work, LGTM!

[jdenny]

@awarzynski Thanks again for reviewing.

[uabelho]

Hi,

I'm not sure if this has been discussed already but I just noticed a change with this patch that at least makes it a bit annoying.
So before running

build-all//bin/llvm-lit -a test/Transforms/ArgumentPromotion/opaque-ptr.ll

yielded

: 'RUN: at line 2';   /repo/uabelho/dev-main/llvm/build-all/bin/opt -S -passes=argpromotion < /repo/uabelho/dev-main/llvm/test/Transforms/ArgumentPromotion/opaque-ptr.ll | /repo/uabelho/dev-main/llvm/build-all/bin/FileCheck /repo/uabelho/dev-main/llvm/test/Transforms/ArgumentPromotion/opaque-ptr.ll

which I could directly copy and paste into my terminal to reproduce the test.
Now I instead get

+ : 'RUN: at line 2'
+ /repo/uabelho/dev-main/llvm/build-all/bin/opt -S -passes=argpromotion
+ /repo/uabelho/dev-main/llvm/build-all/bin/FileCheck /repo/uabelho/dev-main/llvm/test/Transforms/ArgumentPromotion/opaque-ptr.ll

where the "opt" line doesn't even include the test file fed to opt via "< %s". Sure you see the file in the FileCheck command but before reproducing could be done with one single copy/paste, now it takes three.

This will be annoying every time a lit test fails and you want to reproduce it...

[c-rhodes]

this has also broke my workflow, I use lit -a and copy/paste the command to shell to see the output or add different flags, it seems like a fundamental feature and I can't understand why this has been removed?

[awarzynski]

I'm not sure if this has been discussed already

Yes, it has. That was my original feedback for this change and in reply Joel has done 2 very helpful things:

• implemented https://reviews.llvm.org/D156954 as a "workaround" that I personally find very satisfactory,
• posted an RFC to discourse to raise awareness of these changes: https://discourse.llvm.org/t/rfc-improving-lits-debug-output/.

Please be mindful that this change (as well as other LIT patches that are being worked on in tandem) bring various improvements to the overall LIT infrastructure. These things often require compromises. In the case of -a flag (and the "Script" block being removed), it means having to use LIT's internal shell. That was mentioned:

• as a caveat on Discourse in the original RFC,
• in this very helpful and informative reply on Discourse.

Tl;Dr Please try and see whether that is sufficient for you:

LIT_USE_INTERNAL_SHELL=1 path/to/llvm-lit ...

If not then I suggest taking this discussion back to Discourse. For better visibility.

[jdenny]

In D154984#4628289, @awarzynski wrote:

I'm not sure if this has been discussed already

Yes, it has.

Thanks for explaining the situation.

@uabelho, @c-rhodes The goal is to make lit more usable. I apologize if it has broken your workflow despite all the efforts to avoid doing so. There are multiple ways we might think to improve your situation, so please do continue the discourse discussion as @awarzynski suggested (unless his LIT_USE_INTERNAL_SHELL=1 suggestion is sufficient). Thanks.

[uabelho]

In D154984#4628289, @awarzynski wrote:

I'm not sure if this has been discussed already

Yes, it has. That was my original feedback for this change and in reply Joel has done 2 very helpful things:

• implemented https://reviews.llvm.org/D156954 as a "workaround" that I personally find very satisfactory,
• posted an RFC to discourse to raise awareness of these changes: https://discourse.llvm.org/t/rfc-improving-lits-debug-output/.

Please be mindful that this change (as well as other LIT patches that are being worked on in tandem) bring various improvements to the overall LIT infrastructure. These things often require compromises. In the case of -a flag (and the "Script" block being removed), it means having to use LIT's internal shell. That was mentioned:

• as a caveat on Discourse in the original RFC,
• in this very helpful and informative reply on Discourse.

Tl;Dr Please try and see whether that is sufficient for you:

LIT_USE_INTERNAL_SHELL=1 path/to/llvm-lit ...

If not then I suggest taking this discussion back to Discourse. For better visibility.

Thanks! I had no idea about this lit internal shell thing.
So, using

LIT_USE_INTERNAL_SHELL=1 path/to/llvm-lit ...

makes it look nice. Thanks for pointing this out!

Next question then would be why isn't LIT_USE_INTERNAL_SHELL=1 the default. But yeah I saw from the RFC thread not all tests work with that.

[jdenny]

Rebased onto today's main. This brings in PR #65242, which included some code to accommodate the Script: section. The current patch eliminates the Script: section, so this update cleans up that code. Otherwise, the current patch hasn't changed much.

I don't think this patch requires another review before I land it, but of course reviewers are welcome to comment. I will wait to re-land it along with PR #65267 (edit: that's now PR #66408), which still requires more work and review.

llvm/docs/CommandGuide/lit.rst

  Show less output, for example don't show information on tests that pass.
  Also show a progress bar, unless ``--no-progress-bar`` is specified.
 
 .. option:: -v, --verbose
 
  Show more information on test failures, for example the entire test output
  instead of just the test result.
 
+ Each command is printed before it is executed. This can be valuable for
+ debugging test failures, as the last printed command is the one that failed.
+ Moreover, :program:`lit` inserts a no-op command (``:`` in the case of bash)
+ with argument ``'RUN: at line N'`` before each command pipeline, and those
+ no-op commands are also printed to help you locate the source line of the
+ failed command.
+
 .. option:: -vv, --echo-all-commands
 
- On test failure, echo all commands to stdout as they are being executed.
- This can be valuable for debugging test failures, as the last echoed command
- will be the one which has failed.
- :program:`lit` normally inserts a no-op command (``:`` in the case of bash)
- with argument ``'RUN: at line N'`` before each command pipeline, and this
- option also causes those no-op commands to be echoed to stdout to help you
- locate the source line of the failed command.
- This option implies ``--verbose``.
+ Deprecated alias for -v.
 
 .. option:: -a, --show-all
 
- Show more information about all tests, for example the entire test
- commandline and output.
+ Enable -v, but for all tests not just failed tests.
 
 .. option:: --no-progress-bar
 
  Do not use curses based progress bar.
 
 .. option:: --show-unsupported
 
  Show the names of unsupported tests.

llvm/utils/lit/lit/LitConfig.py

def __init__(
         noExecute,
         debug,
         isWindows,
         order,
         params,
         config_prefix=None,
         maxIndividualTestTime=0,
         parallelism_groups={},
-        echo_all_commands=False,
         per_test_coverage=False,
     ):
         # The name of the test runner.
         self.progname = progname
         # The items to add to the PATH environment variable.
         self.path = [str(p) for p in path]
         self.quiet = bool(quiet)
         self.useValgrind = bool(useValgrind)
):
                 self.valgrindArgs.append("--leak-check=full")
             else:
                 # The default is 'summary'.
                 self.valgrindArgs.append("--leak-check=no")
             self.valgrindArgs.extend(self.valgrindUserArgs)
 
         self.maxIndividualTestTime = maxIndividualTestTime
         self.parallelism_groups = parallelism_groups
-        self.echo_all_commands = echo_all_commands
         self.per_test_coverage = per_test_coverage
 
     @property
     def maxIndividualTestTime(self):
         """
         Interface for getting maximum time to spend executing
         a single test
         """

llvm/utils/lit/lit/TestRunner.py

def executeScript(test, litConfig, tmpBase, commands, cwd):
     if isWin32CMDEXE:
         for i, ln in enumerate(commands):
             match = re.fullmatch(kPdbgRegex, ln)
             if match:
                 command = match.group(2)
                 commands[i] = match.expand(
                     "echo '\\1' > nul && " if command else "echo '\\1' > nul"
                 )
-        if litConfig.echo_all_commands:
-            f.write("@echo on\n")
-        else:
-            f.write("@echo off\n")
         f.write("\n@if %ERRORLEVEL% NEQ 0 EXIT\n".join(commands))
     else:
         for i, ln in enumerate(commands):
             match = re.fullmatch(kPdbgRegex, ln)
             if match:
                 command = match.group(2)
                 commands[i] = match.expand(": '\\1'; \\2" if command else ": '\\1'")
         if test.config.pipefail:
             f.write(b"set -o pipefail;" if mode == "wb" else "set -o pipefail;")
-        if litConfig.echo_all_commands:
-            f.write(b"set -x;" if mode == "wb" else "set -x;")
         if sys.version_info > (3, 0) and mode == "wb":
             f.write(bytes("{ " + "; } &&\n{ ".join(commands) + "; }", "utf-8"))
         else:
             f.write("{ " + "; } &&\n{ ".join(commands) + "; }")
     f.write(b"\n" if mode == "wb" else "\n")
     f.close()
 
     if isWin32CMDEXE:
if not test.isWithinFeatureLimits():
             "specified in limit_to_features: %s" % msg,
         )
 
     return script
 
 
 def _runShTest(test, litConfig, useExternalSh, script, tmpBase):
     def runOnce(execdir):
+        # script is modified below (for litConfig.per_test_coverage, and for
+        # %dbg expansions).  runOnce can be called multiple times, but applying
+        # the modifications multiple times can corrupt script, so always modify
+        # a copy.
+        scriptCopy = script[:]
         # Set unique LLVM_PROFILE_FILE for each run command
         if litConfig.per_test_coverage:
             # Extract the test case name from the test object, and remove the
             # file extension.
             test_case_name = test.path_in_suite[-1]
             test_case_name = test_case_name.rsplit(".", 1)[0]
             coverage_index = 0  # Counter for coverage file index
-            for i, ln in enumerate(script):
+            for i, ln in enumerate(scriptCopy):
                 match = re.fullmatch(kPdbgRegex, ln)
                 if match:
                     dbg = match.group(1)
                     command = match.group(2)
                 else:
                     command = ln
                 profile = f"{test_case_name}{coverage_index}.profraw"
                 coverage_index += 1
                 command = f"export LLVM_PROFILE_FILE={profile}; {command}"
                 if match:
                     command = buildPdbgCommand(dbg, command)
-                script[i] = command
+                scriptCopy[i] = command
 
         if useExternalSh:
-            res = executeScript(test, litConfig, tmpBase, script, execdir)
+            res = executeScript(test, litConfig, tmpBase, scriptCopy, execdir)
         else:
-            res = executeScriptInternal(test, litConfig, tmpBase, script, execdir)
+            res = executeScriptInternal(test, litConfig, tmpBase, scriptCopy, execdir)
         if isinstance(res, lit.Test.Result):
             return res
 
         out, err, exitCode, timeoutInfo = res
         if exitCode == 0:
             status = Test.PASS
         else:
             if timeoutInfo is None:
                 status = Test.FAIL
             else:
                 status = Test.TIMEOUT
         return out, err, exitCode, timeoutInfo, status
 
     # Create the output directory if it does not already exist.
     lit.util.mkdir_p(os.path.dirname(tmpBase))
 
     # Re-run failed tests up to test.allowed_retries times.
     execdir = os.path.dirname(test.getExecPath())
     attempts = test.allowed_retries + 1
-    scriptInit = script
     for i in range(attempts):
-        # runOnce modifies script, but applying the modifications again to the
-        # result can corrupt script, so we restore the original upon a retry.
-        # A cleaner solution would be for runOnce to encapsulate operating on a
-        # copy of script, but we actually want it to modify the original script
-        # so we can print the modified version under "Script:" below.
-        script = scriptInit[:]
         res = runOnce(execdir)
         if isinstance(res, lit.Test.Result):
             return res
 
         out, err, exitCode, timeoutInfo, status = res
         if status != Test.FAIL:
             break
 
     # If we had to run the test more than once, count it as a flaky pass. These
     # will be printed separately in the test summary.
     if i > 0 and status == Test.PASS:
         status = Test.FLAKYPASS
 
     # Form the output log.
-    output = """Script:\n--\n%s\n--\nExit Code: %d\n""" % ("\n".join(script), exitCode)
+    output = f"Exit Code: {exitCode}\n"
 
     if timeoutInfo is not None:
         output += """Timeout: %s\n""" % (timeoutInfo,)
     output += "\n"
 
     # Append the outputs, if present.
     if out:
         output += """Command Output (stdout):\n--\n%s\n--\n""" % (out,)

llvm/utils/lit/lit/cl_arguments.py

format_group.add_argument(
         " Additionally, show a progress bar,"
         " unless --no-progress-bar is specified.",
         action="store_true",
     )
     format_group.add_argument(
         "-v",
         "--verbose",
         dest="showOutput",
-        help="Show test output for failures",
+        help="For failed tests, show all output. For example, each command is"
+        " printed before it is executed, so the last printed command is the one"
+        " that failed.",
         action="store_true",
     )
     format_group.add_argument(
         "-vv",
         "--echo-all-commands",
-        dest="echoAllCommands",
+        dest="showOutput",
+        help="Deprecated alias for -v.",
         action="store_true",
-        help="Echo all commands as they are executed to stdout. In case of "
-        "failure, last command shown will be the failing one.",
     )
     format_group.add_argument(
         "-a",
         "--show-all",
         dest="showAllOutput",
-        help="Display all commandlines and output",
+        help="Enable -v, but for all tests not just failed tests.",
         action="store_true",
     )
     format_group.add_argument(
         "-o",
         "--output",
         type=lit.reports.JsonReport,
         help="Write test results to the provided path",
         metavar="PATH",
def parse_args():
     )
 
     # LIT is special: environment variables override command line arguments.
     env_args = shlex.split(os.environ.get("LIT_OPTS", ""))
     args = sys.argv[1:] + env_args
     opts = parser.parse_args(args)
 
     # Validate command line options
-    if opts.echoAllCommands:
-        opts.showOutput = True
-
     if opts.incremental:
         print(
             "WARNING: --incremental is deprecated. Failing tests now always run first."
         )
 
     if opts.numShards or opts.runShard:
         if not opts.numShards or not opts.runShard:
             parser.error("--num-shards and --run-shard must be used together")

llvm/utils/lit/lit/main.py

lit_config = lit.LitConfig.LitConfig(
         valgrindLeakCheck=opts.valgrindLeakCheck,
         valgrindArgs=opts.valgrindArgs,
         noExecute=opts.noExecute,
         debug=opts.debug,
         isWindows=is_windows,
         order=opts.order,
         params=params,
         config_prefix=opts.configPrefix,
-        echo_all_commands=opts.echoAllCommands,
         per_test_coverage=opts.per_test_coverage,
     )
 
     discovered_tests = lit.discovery.find_tests_for_inputs(
         lit_config, opts.test_paths
     )
     if not discovered_tests:
         sys.stderr.write("error: did not discover any tests for provided path(s)\n")

llvm/utils/lit/tests/Inputs/shtest-if-else/test.txt

-# CHECK: -- Testing:{{.*}}
-# CHECK-NEXT: PASS: shtest-if-else :: test.txt (1 of 1)
-# CHECK-NEXT: Script:
-# CHECK-NEXT: --
+#       CHECK: -- Testing:{{.*}}
+#  CHECK-NEXT: PASS: shtest-if-else :: test.txt (1 of 1)
+#  CHECK-NEXT: Exit Code: 0
+# CHECK-EMPTY:
+#  CHECK-NEXT: Command Output (stdout):
+#  CHECK-NEXT: --
 
 # RUN: %if feature %{ echo "test-1" %}
-# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-1]]'; echo "test-1"
+#  CHECK-NEXT: {{^.*"RUN}}: at line [[#@LINE-1]]"
+#       CHECK: # command output:
+#  CHECK-NEXT: test-1
+# CHECK-EMPTY:
 
 # If %else is not present it is treated like %else %{%}. Empty commands
 # are ignored.
 #
 # RUN: %if nofeature %{ echo "fail" %}
-# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-1]]'
-# CHECK-NOT: fail
+#  CHECK-NEXT: {{^.*"RUN}}: at line [[#@LINE-1]]"
+#   CHECK-NOT: fail
 
 # RUN: %if nofeature %{ echo "fail" %} %else %{ echo "test-2" %}
-# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-1]]'; echo "test-2"
+#  CHECK-NEXT: {{^.*"RUN}}: at line [[#@LINE-1]]"
+#       CHECK: # command output:
+#  CHECK-NEXT: test-2
+# CHECK-EMPTY:
 
 # Spaces inside curly braces are not ignored
 #
 # RUN: echo test-%if feature %{ 3 %} %else %{ fail %}-test
 # RUN: echo test-%if feature %{ 4 4 %} %else %{ fail %}-test
 # RUN: echo test-%if nofeature %{ fail %} %else %{ 5 5 %}-test
-# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-3]]'; echo test- 3 -test
-# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-3]]'; echo test- 4 4 -test
-# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-3]]'; echo test- 5 5 -test
+#  CHECK-NEXT: {{^.*"RUN}}: at line [[#@LINE-3]]"
+#       CHECK: # command output:
+#  CHECK-NEXT: test- 3 -test
+# CHECK-EMPTY:
+#  CHECK-NEXT: {{^.*"RUN}}: at line [[#@LINE-6]]"
+#       CHECK: # command output:
+#  CHECK-NEXT: test- 4 4 -test
+# CHECK-EMPTY:
+#  CHECK-NEXT: {{^.*"RUN}}: at line [[#@LINE-9]]"
+#       CHECK: # command output:
+#  CHECK-NEXT: test- 5 5 -test
+# CHECK-EMPTY:
 
 # Escape line breaks for multi-line expressions
 #
 # RUN: %if feature  \
 # RUN:   %{ echo     \
 # RUN:     "test-5" \
 # RUN:   %}
-# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-4]]'; echo "test-5"
+#  CHECK-NEXT: {{^.*"RUN}}: at line [[#@LINE-4]]"
+#       CHECK: # command output:
+#  CHECK-NEXT: test-5
+# CHECK-EMPTY:
 
 # RUN: %if nofeature       \
 # RUN:   %{ echo "fail" %}   \
 # RUN: %else               \
 # RUN:   %{ echo "test-6" %}
-# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-4]]'; echo "test-6"
+#  CHECK-NEXT: {{^.*"RUN}}: at line [[#@LINE-4]]"
+#       CHECK: # command output:
+#  CHECK-NEXT: test-6
+# CHECK-EMPTY:
 
 # RUN: echo "test%if feature %{%} %else %{%}-7"
-# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-1]]'; echo "test-7"
+#  CHECK-NEXT: {{^.*"RUN}}: at line [[#@LINE-1]]"
+#       CHECK: # command output:
+#  CHECK-NEXT: test-7
+# CHECK-EMPTY:
 
 # Escape %if. Without %if..%else context '%{' and '%}' are treated
 # literally.
 #
 # RUN: echo %%if feature %{ echo "test-8" %}
-# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-1]]'; echo %if feature %{ echo "test-8" %}
+#  CHECK-NEXT: {{^.*"RUN}}: at line [[#@LINE-1]]"
+#       CHECK: # command output:
+#  CHECK-NEXT: %if feature %{ echo test-8 %}
+# CHECK-EMPTY:
 
 # Nested expressions are supported:
 #
 # RUN: echo %if feature %{ %if feature %{ %if nofeature %{"fail"%} %else %{"test-9"%} %} %}
-# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-1]]'; echo "test-9"
+#  CHECK-NEXT: {{^.*"RUN}}: at line [[#@LINE-1]]"
+#       CHECK: # command output:
+#  CHECK-NEXT: test-9
+# CHECK-EMPTY:
 
 # Binary expression evaluation and regex match can be used as
 # conditions.
 #
 # RUN: echo %if feature && !nofeature %{ "test-10" %}
 # RUN: echo %if feature && nofeature %{ "fail" %} %else %{ "test-11" %}
 # RUN: echo %if {{fea.+}} %{ "test-12" %} %else %{ "fail" %}
-# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-3]]'; echo "test-10"
-# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-3]]'; echo "test-11"
-# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-3]]'; echo "test-12"
+#  CHECK-NEXT: {{^.*"RUN}}: at line [[#@LINE-3]]"
+#       CHECK: # command output:
+#  CHECK-NEXT: test-10
+# CHECK-EMPTY:
+#  CHECK-NEXT: {{^.*"RUN}}: at line [[#@LINE-6]]"
+#       CHECK: # command output:
+#  CHECK-NEXT: test-11
+# CHECK-EMPTY:
+#  CHECK-NEXT: {{^.*"RUN}}: at line [[#@LINE-9]]"
+#       CHECK: # command output:
+#  CHECK-NEXT: test-12
+# CHECK-EMPTY:
 
 # Spaces between %if and %else are ignored. If there is no %else -
 # space after %if %{...%} is not ignored.
 #
 # RUN: echo XX %if feature %{YY%} ZZ
 # RUN: echo AA %if feature %{BB%} %else %{CC%} DD
 # RUN: echo AA %if nofeature %{BB%} %else %{CC%} DD
-# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-3]]'; echo XX YY ZZ
-# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-3]]'; echo AA BB DD
-# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-3]]'; echo AA CC DD
+#  CHECK-NEXT: {{^.*"RUN}}: at line [[#@LINE-3]]"
+#       CHECK: # command output:
+#  CHECK-NEXT: XX YY ZZ
+# CHECK-EMPTY:
+#  CHECK-NEXT: {{^.*"RUN}}: at line [[#@LINE-6]]"
+#       CHECK: # command output:
+#  CHECK-NEXT: AA BB DD
+# CHECK-EMPTY:
+#  CHECK-NEXT: {{^.*"RUN}}: at line [[#@LINE-9]]"
+#       CHECK: # command output:
+#  CHECK-NEXT: AA CC DD
+# CHECK-EMPTY:
 
 # '{' and '}' can be used without escaping
 #
 # RUN: %if feature %{echo {}%}
-# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-1]]'; echo {}
+#  CHECK-NEXT: {{^.*"RUN}}: at line [[#@LINE-1]]"
+#       CHECK: # command output:
+#  CHECK-NEXT: {}
+# CHECK-EMPTY:
 
 # Spaces are not required
 #
 # RUN: echo %if feature%{"ok"%}%else%{"fail"%}
-# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-1]]'; echo "ok"
+#  CHECK-NEXT: {{^.*"RUN}}: at line [[#@LINE-1]]"
+#       CHECK: # command output:
+#  CHECK-NEXT: ok
+# CHECK-EMPTY:
 
 # Substitutions with braces are handled correctly
 #
 # RUN: echo %{sub} %if feature%{test-%{sub}%}%else%{"fail"%}
-# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-1]]'; echo ok test-ok
-
-# CHECK-NEXT: --
-# CHECK-NEXT: Exit Code: 0
+#  CHECK-NEXT: {{^.*"RUN}}: at line [[#@LINE-1]]"
+#       CHECK: # command output:
+#  CHECK-NEXT: ok test-ok
+# CHECK-EMPTY:

llvm/utils/lit/tests/shtest-format.py

 # CHECK-NEXT: *** TEST 'shtest-format :: external_shell/fail_with_control_chars.txt' FAILED ***
 # CHECK: Command Output (stdout):
 # CHECK-NEXT: --
 # CHECK-NEXT: a line with {{.*}}control characters{{.*}}.
 # CHECK: --
 
 # CHECK: PASS: shtest-format :: external_shell/pass.txt
 
-# CHECK: FAIL: shtest-format :: fail.txt
-# CHECK-NEXT: *** TEST 'shtest-format :: fail.txt' FAILED ***
-# CHECK-NEXT: Script:
-# CHECK-NEXT: --
-# CHECK-NEXT: printf "line 1
-# CHECK-NEXT: false
-# CHECK-NEXT: --
-# CHECK-NEXT: Exit Code: 1
-#
-# CHECK: Command Output (stdout):
-# CHECK-NEXT: --
-# CHECK-NEXT: $ ":" "RUN: at line 1"
-# CHECK-NEXT: $ "printf"
-# CHECK-NEXT: # command output:
-# CHECK-NEXT: line 1: failed test output on stdout
-# CHECK-NEXT: line 2: failed test output on stdout
+#       CHECK: FAIL: shtest-format :: fail.txt
+#  CHECK-NEXT: *** TEST 'shtest-format :: fail.txt' FAILED ***
+#  CHECK-NEXT: Exit Code: 1
+# CHECK-EMPTY:
+#  CHECK-NEXT: Command Output (stdout):
+#  CHECK-NEXT: --
+#  CHECK-NEXT: $ ":" "RUN: at line 1"
+#  CHECK-NEXT: $ "printf"
+#  CHECK-NEXT: # command output:
+#  CHECK-NEXT: line 1: failed test output on stdout
+#  CHECK-NEXT: line 2: failed test output on stdout
+#  CHECK-NEXT: $ ":" "RUN: at line 2"
+#  CHECK-NEXT: $ "false"
 
 # CHECK: UNRESOLVED: shtest-format :: no-test-line.txt
 # CHECK: PASS: shtest-format :: pass.txt
 # CHECK: UNSUPPORTED: shtest-format :: requires-missing.txt
 # CHECK: PASS: shtest-format :: requires-present.txt
 # CHECK: UNRESOLVED: shtest-format :: requires-star.txt
 # CHECK: PASS: shtest-format :: requires-triple.txt
 # CHECK: PASS: shtest-format :: unsupported-expr-false.txt
 # CHECK: UNSUPPORTED: shtest-format :: unsupported-expr-true.txt
 # CHECK: UNRESOLVED: shtest-format :: unsupported-star.txt
 # CHECK: UNSUPPORTED: shtest-format :: unsupported_dir/some-test.txt
 # CHECK: PASS: shtest-format :: xfail-expr-false.txt
 # CHECK: XFAIL: shtest-format :: xfail-expr-true.txt
 # CHECK: XFAIL: shtest-format :: xfail-feature.txt
 # CHECK: XFAIL: shtest-format :: xfail-target.txt
 # CHECK: XFAIL: shtest-format :: xfail.txt
-# CHECK: XPASS: shtest-format :: xpass.txt
-# CHECK-NEXT: *** TEST 'shtest-format :: xpass.txt' FAILED ***
-# CHECK-NEXT: Script
-# CHECK-NEXT: --
-# CHECK-NEXT: true
-# CHECK-NEXT: --
+
+#       CHECK: XPASS: shtest-format :: xpass.txt
+#  CHECK-NEXT: *** TEST 'shtest-format :: xpass.txt' FAILED ***
+#  CHECK-NEXT: Exit Code: 0
+# CHECK-EMPTY:
+#  CHECK-NEXT: Command Output (stdout):
+#  CHECK-NEXT: --
+#  CHECK-NEXT: $ ":" "RUN: at line 1"
+#  CHECK-NEXT: $ "true"
 
 # CHECK: Failed Tests (4)
 # CHECK: shtest-format :: external_shell/fail.txt
 # CHECK: shtest-format :: external_shell/fail_with_bad_encoding.txt
 # CHECK: shtest-format :: external_shell/fail_with_control_chars.txt
 # CHECK: shtest-format :: fail.txt
 
 # CHECK: Unexpectedly Passed Tests (1)

 # XUNIT-NEXT: </testcase>
 
 
 # XUNIT: <testcase classname="shtest-format.external_shell" name="fail_with_bad_encoding.txt" time="{{[0-9]+\.[0-9]+}}">
 # XUNIT-NEXT: <failure{{[ ]*}}>
 # XUNIT: </failure>
 # XUNIT-NEXT: </testcase>
 
-# XUNIT: <testcase classname="shtest-format.external_shell" name="fail_with_control_chars.txt" time="{{[0-9]+\.[0-9]+}}">
-# XUNIT-NEXT: <failure><![CDATA[Script:
-# XUNIT: Command Output (stdout):
-# XUNIT-NEXT: --
-# XUNIT-NEXT: a line with [2;30;41mcontrol characters[0m.
-# XUNIT: </failure>
-# XUNIT-NEXT: </testcase>
+#       XUNIT: <testcase classname="shtest-format.external_shell" name="fail_with_control_chars.txt" time="{{[0-9]+\.[0-9]+}}">
+#  XUNIT-NEXT: <failure><![CDATA[Exit Code: 1
+# XUNIT-EMPTY:
+#  XUNIT-NEXT: Command Output (stdout):
+#  XUNIT-NEXT: --
+#  XUNIT-NEXT: a line with [2;30;41mcontrol characters[0m.
+# XUNIT-EMPTY:
+#  XUNIT-NEXT: --
+#       XUNIT: ]]></failure>
+#  XUNIT-NEXT: </testcase>
 
 # XUNIT: <testcase classname="shtest-format.external_shell" name="pass.txt" time="{{[0-9]+\.[0-9]+}}"/>
 
 # XUNIT: <testcase classname="shtest-format.shtest-format" name="fail.txt" time="{{[0-9]+\.[0-9]+}}">
 # XUNIT-NEXT: <failure{{[ ]*}}>
 # XUNIT: </failure>
 # XUNIT-NEXT: </testcase>
 

llvm/utils/lit/tests/shtest-inject.py

 # Check that we can inject commands at the beginning of a ShTest.
 
 # RUN: %{lit} %{inputs}/shtest-inject/test-empty.txt --show-all | FileCheck --check-prefix=CHECK-TEST1 %s
 #
-# CHECK-TEST1: Script:
-# CHECK-TEST1: --
-# CHECK-TEST1: echo "THIS WAS"
-# CHECK-TEST1: echo "INJECTED"
-# CHECK-TEST1: --
-#
 # CHECK-TEST1: THIS WAS
 # CHECK-TEST1: INJECTED
 #
 # CHECK-TEST1: Passed: 1
 
 # RUN: %{lit} %{inputs}/shtest-inject/test-one.txt --show-all | FileCheck --check-prefix=CHECK-TEST2 %s
 #
-# CHECK-TEST2: Script:
-# CHECK-TEST2: --
-# CHECK-TEST2: echo "THIS WAS"
-# CHECK-TEST2: echo "INJECTED"
-# CHECK-TEST2: echo "IN THE FILE"
-# CHECK-TEST2: --
-#
 # CHECK-TEST2: THIS WAS
 # CHECK-TEST2: INJECTED
 # CHECK-TEST2: IN THE FILE
 #
 # CHECK-TEST2: Passed: 1
 
 # RUN: %{lit} %{inputs}/shtest-inject/test-many.txt --show-all | FileCheck --check-prefix=CHECK-TEST3 %s
 #
-# CHECK-TEST3: Script:
-# CHECK-TEST3: --
-# CHECK-TEST3: echo "THIS WAS"
-# CHECK-TEST3: echo "INJECTED"
-# CHECK-TEST3: echo "IN THE FILE"
-# CHECK-TEST3: echo "IF IT WORKS"
-# CHECK-TEST3: echo "AS EXPECTED"
-# CHECK-TEST3: --
-#
 # CHECK-TEST3: THIS WAS
 # CHECK-TEST3: INJECTED
 # CHECK-TEST3: IN THE FILE
 # CHECK-TEST3: IF IT WORKS
 # CHECK-TEST3: AS EXPECTED
 #
 # CHECK-TEST3: Passed: 1

llvm/utils/lit/tests/shtest-not.py

 
 # CHECK: $ "not" "env" "FOO=1" "BAR=1" "env" "FOO=2" "BAR=2" "not" "--crash" "{{[^"]*}}" "pass.py"
 # CHECK: $ "not" "env" "FOO=1" "BAR=1" "not" "--crash" "not" "{{[^"]*}}" "pass.py"
 # CHECK: $ "not" "not" "--crash" "env" "-u" "BAR" "not" "env" "-u" "FOO" "BAR=1" "{{[^"]*}}" "pass.py"
 
 
 # CHECK: FAIL: shtest-not :: not-calls-fail2.txt {{.*}}
 # CHECK-NEXT: {{.*}} TEST 'shtest-not :: not-calls-fail2.txt' FAILED {{.*}}
-# CHECK-NEXT: Script:
-# CHECK-NEXT: --
-# CHECK:      --
 # CHECK-NEXT: Exit Code: 1
 
 # CHECK: FAIL: shtest-not :: not-calls-mkdir.txt {{.*}}
 # CHECK: $ "not" "mkdir" {{.*}}
 # CHECK: $ "not" "--crash" "mkdir" "foobar"
 # CHECK: Error: 'not --crash' cannot call 'mkdir'
 # CHECK: error: command failed with exit status: {{.*}}
 

llvm/utils/lit/tests/shtest-output-printing.py

 # Check the various features of the ShTest format.
 #
 # RUN: not %{lit} -v %{inputs}/shtest-output-printing > %t.out
 # RUN: FileCheck --input-file %t.out %s
 #
 # END.
 
 # CHECK: -- Testing:
 
 # CHECK: FAIL: shtest-output-printing :: basic.txt
 # CHECK-NEXT: *** TEST 'shtest-output-printing :: basic.txt' FAILED ***
-# CHECK-NEXT: Script:
-# CHECK-NEXT: --
-# CHECK:      --
 # CHECK-NEXT: Exit Code: 1
 #
 # CHECK:      Command Output
 # CHECK-NEXT: --
 # CHECK-NEXT: $ ":" "RUN: at line 1"
 # CHECK-NEXT: $ "true"
 # CHECK-NEXT: $ ":" "RUN: at line 2"
 # CHECK-NEXT: $ "echo" "hi"

llvm/utils/lit/tests/shtest-run-at-line.py

-# Check that -vv makes the line number of the failing RUN command clear.
-# (-v is actually sufficient in the case of the internal shell.)
+# Check that -a/-v/-vv makes the line number of the failing RUN command clear.
 
-# RUN: not %{lit} -vv %{inputs}/shtest-run-at-line > %t.out
-# RUN: FileCheck --input-file %t.out %s
-#
+# RUN: not %{lit} -a %{inputs}/shtest-run-at-line | FileCheck %s
+# RUN: not %{lit} -v %{inputs}/shtest-run-at-line | FileCheck %s
+# RUN: not %{lit} -vv %{inputs}/shtest-run-at-line | FileCheck %s
 # END.
 
 
 # CHECK: Testing: 4 tests
 
 
 # In the case of the external shell, we check for only RUN lines in stderr in
 # case some shell implementations format "set -x" output differently.
 
 # CHECK-LABEL: FAIL: shtest-run-at-line :: external-shell/basic.txt
 
-# CHECK:      Script:
-# CHECK:      RUN: at line 4{{.*}}  true
-# CHECK-NEXT: RUN: at line 5{{.*}}  false
-# CHECK-NEXT: RUN: at line 6{{.*}}  true
-
 # CHECK:     RUN: at line 4
 # CHECK:     RUN: at line 5
 # CHECK-NOT: RUN
 
 # CHECK-LABEL: FAIL: shtest-run-at-line :: external-shell/line-continuation.txt
 
-# CHECK:      Script:
-# CHECK:      RUN: at line 4{{.*}}  echo 'foo bar'  | FileCheck
-# CHECK-NEXT: RUN: at line 6{{.*}}  echo 'foo baz'  | FileCheck
-# CHECK-NEXT: RUN: at line 9{{.*}}  echo 'foo bar'  | FileCheck
-
 # CHECK:     RUN: at line 4
 # CHECK:     RUN: at line 6
 # CHECK-NOT: RUN
 
 
 # CHECK-LABEL: FAIL: shtest-run-at-line :: internal-shell/basic.txt
 
-# CHECK:      Script:
-# CHECK:      : 'RUN: at line 1';  true
-# CHECK-NEXT: : 'RUN: at line 2';  false
-# CHECK-NEXT: : 'RUN: at line 3';  true
-
 # CHECK:      Command Output (stdout)
 # CHECK:      $ ":" "RUN: at line 1"
 # CHECK-NEXT: $ "true"
 # CHECK-NEXT: $ ":" "RUN: at line 2"
 # CHECK-NEXT: $ "false"
 # CHECK-NOT:  RUN
 
 # CHECK-LABEL: FAIL: shtest-run-at-line :: internal-shell/line-continuation.txt
 
-# CHECK:      Script:
-# CHECK:      : 'RUN: at line 1';  : first line continued to second line
-# CHECK-NEXT: : 'RUN: at line 3';  echo 'foo bar'  | FileCheck
-# CHECK-NEXT: : 'RUN: at line 5';  echo  'foo baz'  | FileCheck
-# CHECK-NEXT: : 'RUN: at line 8';  echo 'foo bar'  | FileCheck
-
 # CHECK:      Command Output (stdout)
 # CHECK:      $ ":" "RUN: at line 1"
 # CHECK-NEXT: $ ":" "first" "line" "continued" "to" "second" "line"
 # CHECK-NEXT: $ ":" "RUN: at line 3"
 # CHECK-NEXT: $ "echo" "foo bar"
 # CHECK-NEXT: $ "FileCheck" "{{.*}}"
 # CHECK-NEXT: $ ":" "RUN: at line 5"
 # CHECK-NEXT: $ "echo" "foo baz"
 # CHECK-NEXT: $ "FileCheck" "{{.*}}"
 # CHECK-NOT:  RUN