[lit] Implement PYTHON directive and config.prologue

[jdenny]

Summary

NOTE: This patch should be applied after PR #68496, whose branch is jdenny-ornl:lit-python-script-fatal and whose final commit is currently 909a48a4d1dd04b1841d5e6953797001a560c432.

For example, you can put this in a lit test:

// PYTHON: def check(cflags, fcprefix):
// PYTHON:     lit.run(f"""
// PYTHON:         %clang_cc1 -verify -fopenmp -fopenmp-version=51
// PYTHON:                    {cflags} -emit-llvm -o - %s |
// PYTHON:           FileCheck -check-prefix={fcprefix} %s
// PYTHON:     """)
// PYTHON:
// PYTHON: check("-triple x86_64-apple-darwin10.6.0 -fopenmp-simd", "SIMD")
// PYTHON: check("-triple x86_64-apple-darwin10.6.0", "NO-SIMD")
// PYTHON: check("-triple x86_64-unknown-linux-gnu", "NO-SIMD")

where lit.run(cmd) behaves like RUN: cmd.

To define check at the start of every test file in a test directory, you can move it to a lit.prologue.py file and, in a lit configuration file, set config.prologue to that file.

The idea of a PYTHON directive was proposed at https://reviews.llvm.org/D150856#4436879 (but some details have evolved since then). The rationale is that we are continuing to incrementally evolve lit RUN lines toward a full blown scripting language. A simpler alternative is to just use an existing scripting language, and python seems like the obvious choice. That will lower the learning curve for lit users, quickly and significantly increase the flexibility of lit RUN lines, and lower the burden for lit developers and reviewers.

Toward those goals, PYTHON directives can also eventually replace some other existing and proposed lit features. Lit's existing %if x substitutions can become PYTHON: if lit.has('x'):. Python functions can replace parameterized substitutions and my proposed function-like substitutions.

For further discussion and examples, see this patch's additions to llvm/docs/TestingGuide.rst.

[arichardson]

Are you also planning to adjust the update_tests.py scripts to support this syntax? Might be a good motivation to have them use the lit parser to extract commands rather than regexes.

[jdenny]

In D154987#4491993, @arichardson wrote:

Are you also planning to adjust the update_tests.py scripts to support this syntax?

Thanks, I hadn't thought of that. Is that something you would like to use? Regardless of use cases, I'm inclined to wait to see the reaction to PYTHON directives generally.

Might be a good motivation to have them use the lit parser to extract commands rather than regexes.

Sounds reasonable to me, but I haven't spent much time with those scripts in a while.

[jdenny]

@ldionne: It looks like this patch has revealed an existing bug in libcxx/test/libcxx/selftest/sh.cpp/empty.sh.cpp, but I'm not sure how best to handle it. Can you help?

That test intends to check the case when no RUN line is specified, but its comments accidentally have an embedded RUN line:

// XFAIL: *

// Make sure the test DOES NOT pass if it has no 'RUN:' steps

As a result, the actual lit error is a shell parse error not an error about missing RUN lines. From the pre-commit CI fails above:

Script:
--
%dbg(RUN: at line 11) ' steps
--
Exit Code: 1

Command Output (stdout):
--
shell parser error on: ": 'RUN: at line 11';  ' steps"

--

The XFAIL directive turns that into an expected test fail. That's why the test has not caused the test suite to fail before this patch. After this patch, a shell parse error produces an unresolved test status not a failed test status, so the test suite now fails. That seems like a good side effect of this patch as it catches this malformed test.

I tried to fix the test so it exercises what it was meant to: no RUN lines. To do so, I removed the : after RUN. Both before and after this patch, that produces an unresolved test status, and again that seems like the right lit behavior for a malformed test. However, again, that means the test suite now fails. How would you like to handle that?

[Endill]

Thank you for coming up with the idea, and then implementing it!
Unfortunately, I can't review your implementation from lit standpoint, but it looks solid judging by documentation.

llvm/docs/TestingGuide.rst

317–318

I think we can provide an example for this case as well:

from itertools import product

stds = (98, 11, 14, 17, 20, 23)
triples = (
  'x86_64-apple-darwin10.6.0',
  'x86_64-unknown-linux-gnu'
)
for std, triple in product(stds, triples):
  lit.run(f'%clang_cc1 -std=c++{std} -triple {triple} %s')

But it could be better to hold it until we come up with a variant of run that runs in parallel with other runs, in order to avoid the issue you're describing.

377

I suggest to provide complete example, because description is somewhat difficult to follow, especially for people not too well-versed in Python.

llvm/utils/lit/lit/TestRunner.py

2360–2361

I don't see why we shouldn't provide such an API to PYTHON directives, but it's out of scope of this patch.

[jdenny]

Addressed reviewer comment: make import example clearer.

Made punctuation spacing more consistent.

[jdenny]

In D154987#4501813, @Endill wrote:

Thank you for coming up with the idea, and then implementing it!
Unfortunately, I can't review your implementation from lit standpoint, but it looks solid judging by documentation.

Thanks for reviewing and for the helpful conversations that led here.

llvm/docs/TestingGuide.rst

317–318

Agreed. Let's wait until we have a better handle on such cases.

377

Thanks for that feedback. Please let me know if the new version is still confusing.

llvm/utils/lit/lit/TestRunner.py

2360–2361

I agree we should consider that later.

One reason is we need to think about the interface. (For example, I'm currently inclined to have separate define and redefine functions for safety, but others might disagree.)

Another reason is that, if PYTHON directives can change substitutions, it would not generally be possible to expand substitutions in RUN lines appearing after PYTHON directives until those PYTHON directives have been executed. That might impact the "Script:" discussion under D154984, so let's settle that discussion first.

[ldionne]

In D154987#4501125, @jdenny wrote:

@ldionne: It looks like this patch has revealed an existing bug in libcxx/test/libcxx/selftest/sh.cpp/empty.sh.cpp, but I'm not sure how best to handle it. Can you help?

[...]

I tried to fix the test so it exercises what it was meant to: no RUN lines. To do so, I removed the : after RUN. Both before and after this patch, that produces an unresolved test status, and again that seems like the right lit behavior for a malformed test. However, again, that means the test suite now fails. How would you like to handle that?

Thanks for the heads up and the detailed explanation. It looks like there's no reliable way to test what I was trying to test, and that test isn't *that* important anyway. I'll remove it since it was broken in the first place, so just rebase your patch. Sorry for the trouble.

[Endill]

llvm/utils/lit/lit/TestRunner.py

2360–2361

For example, I'm currently inclined to have separate define and redefine functions for safety

I (again) missed that it's DEFINE and REDEFINE we're talking about here. Sorry for that. As I expressed earlier in our private conversation, I hope PYTHON to subsume REDEFINE, so we wouldn't need to think hard about exposing it to PYTHON directive. But DEFINE is a different story. I agree that we should wait for that discussion to settle.

[MaskRay]

Thank you for implementing this feature! It may simplify some of the 133 rg -l -g '!flang' '%python ' tests we have today.
flang/ has 700+ tests. They may be helped by lit.prologue.py

[sammccall]

I have to offer a dissenting voice here...

Lit shell tests are not written in Python or C++, but in a constrained language specifically for testing.
This has advantages and some disadvantages: e.g. it's easy to see concretely what a lit test does, but can be hard to understand its intent. Assertions are quite readable, but brittle. Output is not always easy to read but fits into a linear sequence of RUN lines.
Personally I think we should be less reliant on lit tests.

But mashing python into lit doesn't produce a coherent result.
Almost all lit features have a python equivalent: CHECK vs check(), DEFINEs vs variables, %if vs if etc. Now there are always several ways to do the same thing, it's unclear when each is best, and the inconsistency will make it hard to read & maintain tests across the repo in various dialects, which many people have to do.
It's possible to process lit shell tests with alternate implementations (without python!), I know of at least two...
Python is a serious general-purpose programming language with tools available, but none of these tools work with python embedded in lit tests (not even syntax highlighting!).

The lit test runner supports having multiple types of tests (this is how we run gtest + shell tests).
Could the tests you're targeting be written as actual python files, with an appropriate library of operations? (Some of this library could of course be shared with the shell test implementation).

[jdenny]

In D154987#4529228, @sammccall wrote:

I have to offer a dissenting voice here...

Thanks for explaining your view.

In D154987#4529228, @sammccall wrote:

Almost all lit features have a python equivalent: CHECK vs check(),

How would a check() work?

DEFINEs vs variables, %if vs if etc.

My hope was that people would gradually migrate away from lit constructs like %if and REDEFINE, which some people find ugly to use and complex in lit's implementation. I do think DEFINE and simple lit substitutions will be around for a long time because they're useful with simple RUN lines.

Now there are always several ways to do the same thing, it's unclear when each is best, and the inconsistency will make it hard to read & maintain tests across the repo in various dialects, which many people have to do.

Maybe, but most tools/languages eventually grow that issue. An intended advantage is that people would be reading python instead of new add-ons for lit's own scripting language, which LLVM developers (understandably) keep wanting to propose. That is, a lit with PYTHON should make tests easier to read and should make lit easier to maintain than a lit with its own evolving scripting language.

It's possible to process lit shell tests with alternate implementations (without python!), I know of at least two...

I don't think I've heard about this. Can you explain more?

Python is a serious general-purpose programming language with tools available, but none of these tools work with python embedded in lit tests (not even syntax highlighting!).

That sounds like a nice integration to work on... instead of continuing to grow lit's own scripting language.

The lit test runner supports having multiple types of tests (this is how we run gtest + shell tests).
Could the tests you're targeting be written as actual python files, with an appropriate library of operations? (Some of this library could of course be shared with the shell test implementation).

Are you suggesting that, instead of embedding python inside C/Fortran/IR/whatever, people should embed C/Fortran/IR/whatever inside python? Either way, libraries can be developed to minimize the complexity of python in the individual tests. However, I think the latter would create greater inconsistency among lit tests than the former: just to add a loop to a shell command in an existing lit test, you would have to reorganize the entire test.

[Endill]

In D154987#4529228, @sammccall wrote:

Could the tests you're targeting be written as actual python files, with an appropriate library of operations? (Some of this library could of course be shared with the shell test implementation).

I think it would accurate to say that the idea this patch implements was born out of use case I have for C++ defect report test suite. As I see it, restructuring that test suite as actual Python files will introduce more noise to the tests than other options. We would also lose C++ syntax highlighting, which would make tests even less readable in majority of contexts.

[awarzynski]

In D154987#4529228, @sammccall wrote:

I have to offer a dissenting voice here...

I am also unsure whether this patch is the right direction for LIT. One major advantage of LIT is that it is "constrained" and hence:

• the number of "idioms" is rather limited, so it's relatively easy to switch between sub-projects and still be able to parse/contribute new tests,
• while "creativity" is limited (it's not Python), it allows us to keep the tests relatively coherent.

While new LIT features make it resemble Python more and more, it is still quite far from Python itself. Also, there's a difference between "adding _a_ new feature" vs "allowing arbitrary Python code". But I appreciate that LIT has it's limitations and we need to evolve it.

That is, a lit with PYTHON should make tests easier to read and should make lit easier to maintain than a lit with its own evolving scripting language.

I am not sure. It will give developers more power, but with extra power comes extra responsibility :) Also, this "power" will require people to understand LIT internals better, right? That wouldn't be a bad thing, but I just want to highlight that there are some advantages of having a dedicated scripting language (which doesn't require one to understand how LIT is implemented).

[jhenderson]

Just chiming in here to give my 2p. lit already has the ability to execute python scripts (call %python), but any such scripts are not directly tied into the lit, so running them etc won't produce output that fits with the lit diagnostics, as I understand it. As I understand it, this patch is essentially just an improvement of the "execute arbitrary python script" option. I'm in favour of it, but if it isn't already clear, I think it would be worth outlining in this patch why simply calling an external python script (which could actually be embedded within the file) is insufficient for the motivating case.

[sammccall]

In D154987#4530187, @jdenny wrote:

In D154987#4529228, @sammccall wrote:

Almost all lit features have a python equivalent: CHECK vs check(),

How would a check() work?

Sorry, this was an incomplete thought I should have revised.
I'm expecting we'll end up with some way of writing assertions directly within python, rather than just calling out to shell commands that verify themselves.
(If this isn't added deliberately, then I'd expect it'll turn up anyway via raise or lit.run("false") or something)

DEFINEs vs variables, %if vs if etc.

My hope was that people would gradually migrate away from lit constructs like %if and REDEFINE, which some people find ugly to use and complex in lit's implementation.

I think gradual migration is a recipe for getting stuck in a halfway state. I agree with the ugliness and complexity, and wish these features had also not been added. The mixture is yet more complicated.

Maybe, but most tools/languages eventually grow that issue. An intended advantage is that people would be reading python instead of new add-ons for lit's own scripting language, which LLVM developers (understandably) keep wanting to propose. That is, a lit with PYTHON should make tests easier to read and should make lit easier to maintain than a lit with its own evolving scripting language.

The difficulty in extending lit cleanly is a good reason for an alternative, and I think python is a good choice. (just not embedded in lit)

It's possible to process lit shell tests with alternate implementations (without python!), I know of at least two...

I don't think I've heard about this. Can you explain more?

Google uses a custom lit test runner to run LLVM tests as part of CI.
This has different performance characteristics and fits better into the infrastructure (each test is run as a separate sandboxed action in a distributed build/test environment).
This is possible because lit is a spec as much as a limitation (though recent feature additions have strained this).
The main limitation is that the config definition & discovery scheme is unsuitable for multiple reasons, so config needs to be configured separately.

The current iteration of this system implements a shell test interpreter as a Go binary (gtests are run in a different way).
The previous iteration mangled tests into shell scripts through regexps.

This is downstream and LLVM doesn't have any obligation to support any of it, but the ability to be integrated into different environments is a useful LLVM feature, and having clear interfaces for infra allows it to evolve in ways other than bolting on more features.

Python is a serious general-purpose programming language with tools available, but none of these tools work with python embedded in lit tests (not even syntax highlighting!).

That sounds like a nice integration to work on... instead of continuing to grow lit's own scripting language.

Agreed. Realistically, none of the standard tools for editing python code will ever work if the code is prefixed with PYTHON: lines and lives in *.cpp files, though.

The lit test runner supports having multiple types of tests (this is how we run gtest + shell tests).
Could the tests you're targeting be written as actual python files, with an appropriate library of operations? (Some of this library could of course be shared with the shell test implementation).

Are you suggesting that, instead of embedding python inside C/Fortran/IR/whatever, people should embed C/Fortran/IR/whatever inside python?

I do like the trick where a single lit test file provides multiple kinds of inputs (e.g. to lit + clang + FileCheck), but it is hard to reason about, and would not like to see it used when the test contains control flow.
I'm not sure whether it's best to have C etc inputs as string literals, or as separate files.

String literals are workable, because python has multiline raw strings (vs PYTHON: prefix), and the embedded C programs tend to be self-contained (vs interacting with e.g. lit).
So readability & tools are a bit better as python-in-lit, and the need for tools isn't so strong.
This is the approach we mostly used in clangd (e.g. llvm-project/clang-tools-extra/clangd/unittests/FindTargetTests.cpp)

Separate files for the C++ inputs are in some ways a more principled option, and tools work well, but more hassle to organize and navigate.

Either way, libraries can be developed to minimize the complexity of python in the individual tests. However, I think the latter would create greater inconsistency among lit tests than the former: just to add a loop to a shell command in an existing lit test, you would have to reorganize the entire test.

FWIW, this seems like a feature to me: you have to choose between terse/cryptic/self-referential lit test syntax, and powerful logic. (Hiding the logic in libraries makes it both better and worse!)
Having to rewrite the test is work, but starting with spaghetti and adding structure *without* rewriting may be worse...

[jdenny]

In D154987#4531624, @Endill wrote:

In D154987#4529228, @sammccall wrote:

Could the tests you're targeting be written as actual python files, with an appropriate library of operations? (Some of this library could of course be shared with the shell test implementation).

I think it would accurate to say that the idea this patch implements was born out of use case I have for C++ defect report test suite.

Definitely. Other feedback/experience for DEFINE, REDEFINE, function substitutions, and %if also contributed.

In D154987#4531641, @awarzynski wrote:

I am also unsure whether this patch is the right direction for LIT.

Thanks for chiming in.

One major advantage of LIT is that it is "constrained" and hence:

• the number of "idioms" is rather limited, so it's relatively easy to switch between sub-projects and still be able to parse/contribute new tests,
• while "creativity" is limited (it's not Python), it allows us to keep the tests relatively coherent.

I haven't found that to be true when test suites invent their own custom lit test "formats". I admit I haven't investigated their usage generally throughout LLVM. However, when I've come across one, I have found it to be a significant paradigm shift, and I have had to study the new format's implementation to grasp what the custom rules are. In my estimation, the difficulty of understanding a lit shell test with PYTHON when you're used to understanding a lit shell test without PYTHON would be much smaller in comparison. (To be clear, I'm not arguing for avoiding custom lit test formats or replacing them with PYTHON. I'm just comparing the difficulty of understanding PYTHON with variations that already appear in lit test suites.)

That is, a lit with PYTHON should make tests easier to read and should make lit easier to maintain than a lit with its own evolving scripting language.

I am not sure. It will give developers more power, but with extra power comes extra responsibility :) Also, this "power" will require people to understand LIT internals better, right?

What lit internals do you mean? Do you mean it enables tests to directly access functions and symbols within lit's implementation? This patch specifically avoids that. Access to lit-provided features is via a carefully defined lit object in python. Or do you mean it requires test authors to learn various complex subtleties of new user-visible lit features? (I think people sometimes refer to those subtleties as lit internals.) That's exactly a problem this patch is trying to avoid by having people use a widely used and well documented language like python instead of new lit scripting language features.

That wouldn't be a bad thing, but I just want to highlight that there are some advantages of having a dedicated scripting language (which doesn't require one to understand how LIT is implemented).

I previously thought that too. In practice, I don't think it's working out that way.

We tend to start off with a simple and familiar idea we're used to in other languages (recent examples: variables/macros, functions, if-then-else, for-each). Then we try to force that into lit's existing shell language, and we find there are many lit-specific subtleties to handle, debate, and ultimately teach. Then we start using the feature and find it doesn't adapt well to other use cases. Then people say they regret it. If we had a dedicated dream team for developing lit features, maybe we could design the perfect set of lit features that give LLVM developers just what they need without adding too much power. In the real world, I think it's more likely we'll keep evolving a messy collection of features that many people dislike. Instead, I say let's use python and get back to developing compilers.

[jdenny]

In D154987#4531915, @sammccall wrote:

I'm expecting we'll end up with some way of writing assertions directly within python, rather than just calling out to shell commands that verify themselves.
(If this isn't added deliberately, then I'd expect it'll turn up anyway via raise or lit.run("false") or something)

In this patch's current implementation, python compile failures (similar to RUN line shell parse errors) produce "unresolved" tests, and python execution failures (similar to RUN line non-zero exit status) produce "failed" tests. Yes, test authors could use raise or some lit.run fallacy to induce test failures. I don't see why that's bad if the check they want to perform is easier to write in python.

I think gradual migration is a recipe for getting stuck in a halfway state.

We could migrate lit and all llvm upstream test suites at the same time. However, I think it would be better to give people time to adjust and make sure there are no surprises. Moreover, I don't think the halfway state in this case is so dire. My hope is that PYTHON will stop the development of new lit scripting language features so we don't find ourselves with more features to maintain and possibly migrate away from. Then we would gradually remove the few that have already landed unless we find a good reason to keep them.

The difficulty in extending lit cleanly is a good reason for an alternative, and I think python is a good choice. (just not embedded in lit)

So, as @jhenderson suggested, I think this is the issue we should focus discussion on right now. That is, why embed python (using PYTHON) rather than just call a separate python script? Why embed multiple shell commands (using RUN) rather than just call a separate shell script? In my opinion, test authors should do what makes their tests easiest to understand while avoiding repeated code. Sometimes that means making test code visible directly in the individual test file, and sometimes that means encapsulating it in another file.

It's possible to process lit shell tests with alternate implementations (without python!), I know of at least two...

I don't think I've heard about this. Can you explain more?

Google uses a custom lit test runner to run LLVM tests as part of CI.

Thanks for explaining that.

This is downstream and LLVM doesn't have any obligation to support any of it, but the ability to be integrated into different environments is a useful LLVM feature, and having clear interfaces for infra allows it to evolve in ways other than bolting on more features.

I think it's too much to ask upstream lit developers to avoid adding new features indefinitely because it causes trouble for downstream users.

Realistically, none of the standard tools for editing python code will ever work if the code is prefixed with PYTHON: lines and lives in *.cpp files, though.

I agree that tools that support standard python would require extensions or shims. I don't have experience with python tools to know how hard that is. Then again, people might find it's really not worthwhile. Complex python will likely end up in a python module that is simply called from a PYTHON line. That would look a lot like what people already do with %python in RUN. But PYTHON gives people a convenient, well integrated way to keep their python local to the test when that makes the test easier to read and maintain.

I do like the trick where a single lit test file provides multiple kinds of inputs (e.g. to lit + clang + FileCheck), but it is hard to reason about, and would not like to see it used when the test contains control flow.

Why is control flow the cutoff?

So readability & tools are a bit better as python-in-lit, and the need for tools isn't so strong.

I'm not sure I'm following. Which approach does "python-in-lit" refer to here?

[Endill]

In D154987#4532960, @jdenny wrote:

Complex python will likely end up in a python module that is simply called from a PYTHON line. That would look a lot like what people already do with %python in RUN. But PYTHON gives people a convenient, well integrated way to keep their python local to the test when that makes the test easier to read and maintain.

I'd like to double this. I've already prepared a bit of python code to generate -verify prefixes that we need for DR testing, which I need in every test. config.prologue gives me an opportunity to move it to a separate file. We can even strive to have 0 explicit RUN lines in DR tests, generating everything in Python code defined in a single file. This way we wouldn't have to duplicate list of C++ language modes 27 times per each test file.

This helps establishing an environment where Clang contributors can focus on writing DR tests simply by following an example, without bothering with RUN lines. It also saves time, because there will be less to teach and less to iterate over during code review.

As a bottom line, there is a problem to solve for at least DR tests. I attempted to solve it in a less ambitious way in https://reviews.llvm.org/D150856. There you can also find detailed discussion of the problem I'm trying to solve. If this patch is going to be rejected, I'd like opponents to give me a different path forward.

[aaron.ballman]

I definitely understand the desire to keep lit as simple and focused as we can, and from that perspective, adding support for a full scripting language seems like a bad idea. However, lit also has to meet our testing needs and in its current form, we're missing a direct way to support our uses for defect report testing in C and C++. Because there are thousands of defect reports in C++ alone, we group the tests together in blocks of roughly 100 tests within a single file. However, we need to run those tests semi-hermetically because we do not want an earlier test failure to impact the diagnostics for later tests (this causes us to miss diagnostics that would otherwise be issued and can hide compiler or testing bugs). Additionally, each of these tests needs to be run in all of the various language modes we support and we'd like it to be easy to add new language standard modes to the set without having to update each test file directly. The proposed support in this patch will help solve those problems in a way that I think is pretty straightforward yet powerful enough to be used to solve testing problems for other projects as well (such as the issue flang is having that @MaskRay pointed out).

[awarzynski]

In D154987#4531662, @jhenderson wrote:

lit already has the ability to execute python scripts (call %python),

I forgot about that, thanks for the reminder! This means that "python super powers" that I was concerned about earlier are already available to us. Please ignore my earlier concerns regarding that part.

I think it would be worth outlining in this patch why simply calling an external python script (which could actually be embedded within the file) is insufficient for the motivating case.

+1 Also, is this change meant to replace %python? That would be very nice and would also help maintain consistency within our tests. Otherwise, we will have "yet another way" to run Python.

What lit internals do you mean? Do you mean it enables tests to directly access functions and symbols within lit's implementation? This patch specifically avoids that. Access to lit-provided features is via a carefully defined lit object in python.

@jdenny, thanks for clarifying. I assumed (incorrectly) that the intention was to open-up all of LIT's internals to people writing tests. That's not the case. My bad for not reading the summary beforehand.

The proposed support in this patch will help solve those problems in a way that I think is pretty straightforward yet powerful enough to be used to solve testing problems for other projects as well (such as the issue flang is having that @MaskRay pointed out).

IMHO, we should refrain from making assumptions about any potential refactoring that _might_ happen following this change. Unless there are volunteers to implement them. I am referring specifically to Flang:

• updating ~700 tests is not a small task,
• those tests were ported from Bash to Python (via %python) to enable better testing on Windows,
• I am not aware of any limitation of the current approach (with %python) that active Flang developers would like to have solved,
• those tests don't use FileCheck (previous discussion on that) or any other "typical" LLVM infra/tooling (i.e. there's relatively little alignment with other parts of LLVM).

It was one of my colleagues who ported those Flang tests from Bash to Python - that's how I know the context.

[jdenny]

Rebased. Extended docs with comparison to using %python, as requested.

[jdenny]

Thanks for the additional comments from everyone.

In D154987#4508046, @ldionne wrote:

It looks like there's no reliable way to test what I was trying to test, and that test isn't *that* important anyway. I'll remove it since it was broken in the first place, so just rebase your patch. Sorry for the trouble.

No worries, and thanks for the fix. I finally got around to the rebase.

In D154987#4539229, @awarzynski wrote:

In D154987#4531662, @jhenderson wrote:

I think it would be worth outlining in this patch why simply calling an external python script (which could actually be embedded within the file) is insufficient for the motivating case.

+1

I just extended TestingGuide.rst. Let me know if it's not what you are looking for.

Also, is this change meant to replace %python? That would be very nice and would also help maintain consistency within our tests. Otherwise, we will have "yet another way" to run Python.

I'm not sure we can prevent every subproject from defining its own %python lit substitution in its test suite configuration. I'd like to think many would eventually choose the PYTHON directive instead.

[jdenny]

This patch will eventually need to be updated for the changes in D156954, but I haven't gotten there yet.

[jdenny]

Rebase onto D156954 and update to its execution trace style.

[jhenderson]

In D154987#4559192, @jdenny wrote:

Thanks for the additional comments from everyone.

In D154987#4508046, @ldionne wrote:

It looks like there's no reliable way to test what I was trying to test, and that test isn't *that* important anyway. I'll remove it since it was broken in the first place, so just rebase your patch. Sorry for the trouble.

No worries, and thanks for the fix. I finally got around to the rebase.

In D154987#4539229, @awarzynski wrote:

In D154987#4531662, @jhenderson wrote:

I think it would be worth outlining in this patch why simply calling an external python script (which could actually be embedded within the file) is insufficient for the motivating case.

+1

I just extended TestingGuide.rst. Let me know if it's not what you are looking for.

Thanks. In my case at least, I was just referring to enhancing the commit description or even just posting a review comment with the explanation. Enhancing the documentation works for me though.

llvm/docs/TestingGuide.rst

348

[jdenny]

Rebase onto recent D156954 changes to address libcxx pre-commit CI failure.

In docs, apply suggested typo correction, and wordsmith recent text a little.

[jdenny]

In D154987#4568886, @jhenderson wrote:

In D154987#4539229, @awarzynski wrote:

In D154987#4531662, @jhenderson wrote:

I think it would be worth outlining in this patch why simply calling an external python script (which could actually be embedded within the file) is insufficient for the motivating case.

+1

I just extended TestingGuide.rst. Let me know if it's not what you are looking for.

Thanks. In my case at least, I was just referring to enhancing the commit description or even just posting a review comment with the explanation. Enhancing the documentation works for me though.

Thanks for taking a look (and the typo correction). Hopefully having it in the docs will help if this point continues to be raised, and it might help users choose between %python and PYTHON.

[jdenny]

Rebase onto recent D156954 changes to address another libcxx pre-commit CI failure.

[jdenny]

Replace <lit test> with the actual test file name when compiling PYTHON lines. This info helps the python interpreter produce more informative diagnostics, including not only the test file name but also sometimes full PYTHON lines quoted from the test file. After a future patch, I expect it to also help when setting breakpoints while using a python debugger on lit test files.

Also, set __file__ (normally defined in python scripts) to the test file name.

[jdenny]

Rebased onto PR #66408, which includes recent changes from main. Within TestRunner.py, this rebase required moving runShScript and PythonDirectiveLitAPI into runOnce to make the per-run --per-test-coverage file index visible to lit.run(cmd) calls. Extended tests to cover that case.

Applied python formatting suggestions from darker.

[jdenny]

Now that the major prerequisites have landed, it would be good to resume our discussion of the PYTHON directive. My recollection is that most people who commented were happy with the concept.

@awarzynski I recall you had some concerns and that we addressed some. What is your current opinion?

[awarzynski]

In D154987#4652527, @jdenny wrote:

@awarzynski I recall you had some concerns and that we addressed some. What is your current opinion?

Sorry for delay, I tend to focus on GitHub these days.

I'm OK with this change. Based on the feedback, this is clearly desirable by various folks. And great deal of effort has gone into preparing LIT for this PR, so I wouldn't want to block it. While I'm still concerned about extending LIT like this, there's an option for people _not to use_ the new directive.

As always with this sort of changes, I suggest "advertising" on Discourse to make sure that people are aware what's coming. Also, is any other sub-project beyond libcxx intending to use this?

[Endill]

In D154987#4652870, @awarzynski wrote:

Also, is any other sub-project beyond libcxx intending to use this?

Some of my work on Clang C++ DR test suite is blocked on this (https://reviews.llvm.org/D148433). I've been preparing bits and pieces that play along with this feature for quite a while.

[jdenny]

In D154987#4652870, @awarzynski wrote:

Sorry for delay, I tend to focus on GitHub these days.

No worries. I'm a bit distracted at the moment anyway.

I'm OK with this change. Based on the feedback, this is clearly desirable by various folks. And great deal of effort has gone into preparing LIT for this PR, so I wouldn't want to block it.

Thanks for your understanding.

While I'm still concerned about extending LIT like this, there's an option for people _not to use_ the new directive.

For people who are not interested in the PYTHON directive, I hope they will at least benefit from the debugger option that D158954 builds on top for the sake of lit tests (whether or not those tests actually use PYTHON directives explicitly). After all the recent discussion of copying and pasting shell code from lit's output to a terminal window and hoping that terminal's shell processes them correctly, and after my own experience trying to replicate environment variables and investigating lit substitution issues, I imagine a proper debugger that can step through RUN lines and examine/adjust state would be a welcome improvement.

As always with this sort of changes, I suggest "advertising" on Discourse to make sure that people are aware what's coming.

As you suggested, this new feature is opt-in. It is not meant to change behavior for existing lit tests. For that reason, I think it's less important to write a general RFC before landing than in the case of, for example, the recent lit debug output changes. I'm inclined to land this one first (after a technical review) and advertise afterward so that folks like @Endill can move forward. That would also permit others to more easily try out the new feature once we do advertise. We can always tweak things afterward as the need arises. Is that a reasonable plan?

[jdenny]

Rebased onto PR #68496, which includes the version of PR #67898 that landed. Thus, this patch no longer includes changes from those PRs.

Fixed an omission of -triple in the example appearing in the commit log and in TestingGuide.rst.

libcxx/utils/libcxx/test/format.py

def _executeScriptInternal(test, litConfig, commands):
     Returns (stdout, stderr, exitCode, timeoutInfo, parsedCommands)
 
     TODO: This really should be easier to access from Lit itself
     """
     parsedCommands = parseScript(test, preamble=commands)
 
     _, tmpBase = _getTempPaths(test)
     execDir = os.path.dirname(test.getExecPath())
+    try:
         res = lit.TestRunner.executeScriptInternal(
             test, litConfig, tmpBase, parsedCommands, execDir, debug=False
         )
-    if isinstance(res, lit.Test.Result):  # Handle failure to parse the Lit test
-        res = ("", res.output, 127, None)
+    except lit.TestRunner.ScriptFatal as e:
+        res = ("", str(e), 127, None)
     (out, err, exitCode, timeoutInfo) = res
 
     return (out, err, exitCode, timeoutInfo, parsedCommands)
 
 
 def parseScript(test, preamble):
     """
     Extract the script from a test, with substitutions applied.

llvm/docs/TestingGuide.rst

 how to run the tests. This file is just Python code and thus is very
 flexible, but we've standardized it for the LLVM regression tests. If
 you're adding a directory of tests, just copy ``lit.local.cfg`` from
 another directory to get running. The standard ``lit.local.cfg`` simply
 specifies which files to look in for tests. Any directory that contains
 only directories does not need the ``lit.local.cfg`` file. Read the :doc:`Lit
 documentation <CommandGuide/lit>` for more information.
 
-Each test file must contain lines starting with "RUN:" that tell :program:`lit`
-how to run it. If there are no RUN lines, :program:`lit` will issue an error
-while running a test.
+Each test file must contain lines with "RUN:" or "PYTHON:" that tell
+:program:`lit` how to run it, as described in the following sections. If there
+are no such lines, :program:`lit` will issue an error while running a test.
+
+Put related tests into a single file rather than having a separate file per
+test. Check if there are files already covering your feature and consider
+adding your code there instead of creating a new file.
+
+RUN lines
+---------
 
 RUN lines are specified in the comments of the test program using the
 keyword ``RUN`` followed by a colon, and lastly the command (pipeline)
 to execute. Together, these lines form the "script" that :program:`lit`
 executes to run the test case. The syntax of the RUN lines is similar to a
 shell's syntax for pipelines including I/O redirection and variable
 substitution. However, even though these lines may *look* like a shell
 script, they are not. RUN lines are interpreted by :program:`lit`.

 everything enclosed as one value.
 
 In general, you should strive to keep your RUN lines as simple as possible,
 using them only to run tools that generate textual output you can then examine.
 The recommended way to examine output to figure out if the test passes is using
 the :doc:`FileCheck tool <CommandGuide/FileCheck>`. *[The usage of grep in RUN
 lines is deprecated - please do not send or commit patches that use it.]*
 
-Put related tests into a single file rather than having a separate file per
-test. Check if there are files already covering your feature and consider
-adding your code there instead of creating a new file.
+PYTHON lines
+------------
+
+While RUN lines are great for specifying LLVM tool command lines, its simple
+shell-like functionality can prove insufficient for some testing logic. For such
+cases, PYTHON lines offer the full power of python scripting, including loops,
+functions, and modules, as illustrated in the examples below.
+
+**lit.run(cmd):**
+
+In order to move a RUN line into, for example, a python control structure like a
+loop, you must convert it to a ``lit.run(cmd)`` call in a PYTHON line. For
+example:
+
+.. code-block:: llvm
+
+    ; PYTHON: for triple in ("x86_64-apple-darwin10.6.0", "x86_64-unknown-linux-gnu"):
+    ; PYTHON:     lit.run(f"""
+    ; PYTHON:         %clang_cc1 -verify -fopenmp -fopenmp-version=51
+    ; PYTHON:                    -triple {triple} -emit-llvm -o - %s |
+    ; PYTHON:           FileCheck %s
+    ; PYTHON:     """)
+
+``cmd`` in ``PYTHON: lit.run(cmd)`` behaves very much like it does in ``RUN:
+cmd``. That is, :program:`lit` performs the same substitutions, parses the same
+shell-like syntax, and then executes the resulting commands. Of course, a
+difference is that python string syntax must be followed in PYTHON lines. For
+example, in a python f-string, a :program:`lit` substitution like ``%{foo}``
+must be escaped as ``%{{foo}}``.
+
+**Python functions:**
+
+The ability to write loops in PYTHON lines makes it easy to fall into the trap
+of creating long loop nests to achieve test coverage for all points in a large
+problem space. Especially given that :program:`lit` serially executes all
+commands in a single test file (:program:`lit` parallelism is only across test
+files), this approach can produce very slow tests. Unless full coverage is
+critical, it might be better to specify only a cross-section of such a problem
+space. Python functions can be useful for this purpose. For example, we might
+extend the above example with SIMD testing but choose to do so for only one
+triple:
+
+.. code-block:: llvm
+
+    ; PYTHON: def check(cflags, fcprefix):
+    ; PYTHON:     lit.run(f"""
+    ; PYTHON:         %clang_cc1 -verify -fopenmp -fopenmp-version=51
+    ; PYTHON:                    {cflags} -emit-llvm -o - %s |
+    ; PYTHON:           FileCheck -check-prefix={fcprefix} %s
+    ; PYTHON:     """)
+    ; PYTHON:
+    ; PYTHON: check("-triple x86_64-apple-darwin10.6.0 -fopenmp-simd", "SIMD")
+    ; PYTHON: check("-triple x86_64-apple-darwin10.6.0", "NO-SIMD")
+    ; PYTHON: check("x86_64-unknown-linux-gnu", "NO-SIMD")
+
+**Python symbol scope:**
+
+The above example defines a new python symbol, the function ``check``. As in a
+python script, symbols defined in a PYTHON line are visible in all later PYTHON
+lines in the same test file. They are not visible in other test files, which is
+especially important given that :program:`lit` test file execution order is
+usually non-deterministic.
+
+**Importing python modules:**
+
+You might decide that some python symbols need to be shared across a set of test
+files. As usual in python, you can define such symbols in a python module (but
+see `config.prologue`_ below for an alternative approach if you need to share
+them across an entire test directory). For example, if the module file
+``exampleModule.py`` is located in the same directory as your test files, you
+can import it as follows:
+
+.. code-block:: llvm
+
+    ; PYTHON: import sys
+    ; PYTHON: sys.path.append(lit.expand("%S"))
+    ; PYTHON: import exampleModule
+
+**lit.expand(text):**
+
+``lit.expand(text)`` performs the same substitutions that :program:`lit`
+performs in RUN lines and in `lit.run(cmd)`. In the above example, it retrieves
+the value of :program:`lit`'s ``%S`` substitution, which is the test case's
+source directory, so that ``exampleModule.py`` can be located.
+
+**Exposing the lit object to an imported module:**
+
+As usual in python, an imported module does not automatically have access to
+symbols in the scope of the importer. In the above example, ``exampleModule.py``
+does not have access to the ``lit`` object. However, it needs access if it
+defines the ``check`` function above.
+
+One solution is to declare ``lit`` within the module and then assign it in a
+PYTHON line. For example, the contents of ``exampleModule.py`` would be as
+follows:
+
+.. code-block:: python
+
+    lit = None
+
+    def check(cflags, fcprefix):
+        lit.run(f"""
+            %clang_cc1 -verify -fopenmp -fopenmp-version=51
+                       {cflags} -emit-llvm -o - %s |
+              FileCheck -check-prefix={fcprefix} %s
+        """)
+
+Then your test's PYTHON lines would import ``exampleModule.py`` as follows:
+
+.. code-block:: llvm
+
+    ; PYTHON: import sys
+    ; PYTHON: sys.path.append(lit.expand("%S"))
+    ; PYTHON: import exampleModule
+    ; PYTHON: exampleModule.lit = lit
+
+.. _config.prologue:
+
+**config.prologue:**
+
+If ``exampleModule.py`` is needed in every test file in a test directory, the
+above import code might be cumbersome to repeat and maintain as a prologue in
+them all, especially if it evolves over time to include other modules or to
+perform some additional initialization.
+
+To avoid this problem, you can set ``config.prologue`` in a lit configuration
+file. Its value must be the name of a python file, whose contents :program:`lit`
+handles as if they appear in PYTHON lines at the start of every test file. Thus,
+any python symbols it defines are seen throughout every test file, and it has
+access to the ``lit`` object. For example, you might decide to rename
+``exampleModule.py`` to something like ``lit.prologue.py``, set
+``config.prologue`` to that file, and thus avoid writing the code for importing
+a module and exposing the ``lit`` object.
+
+**lit.has(feature):**
+
+To determine if ``feature`` appears in ``config.available_features`` (see
+`Constraining test execution`_ and `%if` under `Substitutions`_), you can call
+`lit.has(feature)`.
+
+**Mixing PYTHON lines and other directives:**
+
+It is fine to mix RUN lines and PYTHON lines in the same test file.
+:program:`lit` executes them in the order in which they appear. A possible way
+for them to interact is via temporary files (e.g., see ``%t`` in
+`Substitutions`_).
+
+It is also fine to mix PYTHON lines with DEFINE and REDEFINE lines, which assign
+:program:`lit` substitutions (see `Test-specific substitutions`_).
+`lit.expand(text)` and `lit.run(cmd)` will see the latest substitution values
+just as RUN lines do.
+
+The ability to mix PYTHON lines with other directives is helpful for at least
+two reasons. First, it facilitates extending existing test files with PYTHON
+lines without having to replace all RUN, DEFINE, and REDEFINE lines. Second,
+even for new tests, when a simple ``RUN: cmd`` would be sufficient, it is
+probably best not to complicate it with python syntax by converting it to
+``PYTHON: lit.run(cmd)``.
+
+**Python blocks and indentation:**
+
+:program:`lit` parses and executes all consecutive PYTHON lines as a single
+PYTHON block. For :program:`lit`, consecutive just means there are no other
+:program:`lit` directives (e.g., RUN lines) in between, but :program:`lit`
+discards other lines as inert text. The python code in a PYTHON block must be
+syntactically complete. For example, a PYTHON block cannot end with ``PYTHON: if
+cond:`` no matter what appears at the start of the next PYTHON block.
+
+Any whitespace immediately after ``PYTHON:`` on the initial line of a PYTHON
+block must also appear immediately after ``PYTHON:`` on all non-initial lines in
+the block. :program:`lit` trims that whitespace from every line before passing
+it to the python interpreter. Thus, non-initial lines can contain additional
+indentation according to the rules of python syntax.
+
+**Calling python from RUN lines:**
+
+In some tests, you might find techniques for calling python code from
+a RUN line rather than from a PYTHON line.  In particular, a test
+might call the python interpreter (usually specified via a ``%python``
+or ``%{python}`` :program:`lit` substitution) on an external python
+file.  A test might use the ``split-file`` utility (see `Extra files`_
+below) to embed that python file directly into the test file if the
+python code is sufficiently concise and does not need to be reused
+across multiple tests.
+
+A disadvantage of such techniques is that they do not have access to
+the ``lit`` object exposed to PYTHON lines.  For example, to move a
+shell command from a RUN line into a control structure like a loop or
+function without using ``lit.run(cmd)`` as in the examples above, the
+test author can instead pass the shell command to python's own
+``subprocess.run`` function.  To make any required :program:`lit`
+substitutions available, they can pass them as command-line arguments
+on the python interpreter RUN line, keeping in mind that substitutions
+are sometimes complex and that proper shell quoting must be applied.
+It is the test author's job to decide how to capture stdout, stderr,
+exit status, etc. from the shell command and present them for
+debugging when the test fails.  In contrast, ``lit.run(cmd)`` called
+from a PYTHON line handles all these issues already in a manner that
+is consistent with :program:`lit`'s handling of RUN lines.
 
 Generating assertions in regression tests
 -----------------------------------------
 
 Some regression test cases are very large and complex to write/update by hand.
 In that case to reduce the human work we can use the scripts available in
 llvm/utils/ to generate the assertions.
 
- Lit configuration files (e.g., ``lit.cfg`` or ``lit.local.cfg``) can define
   substitutions for all tests in a test directory.  They do so by extending the
   substitution list, ``config.substitutions``.  Each item in the list is a tuple
   consisting of a pattern and its replacement, which lit applies using python's
   ``re.sub`` function.
 - To define substitutions within a single test file, lit supports the
   ``DEFINE:`` and ``REDEFINE:`` directives, described in detail below.  So that
   they have no effect on other test files, these directives modify a copy of the
   substitution list that is produced by lit configuration files.
+- While simple substitution definitions can be useful for avoiding repeated
+  code, complex substitution relationships can be hard to understand and
+  maintain.  Consider using python functions instead (see `PYTHON lines`_).
 
 For example, the following directives can be inserted into a test file to define
 ``%{cflags}`` and ``%{fcflags}`` substitutions with empty initial values, which
 serve as the parameters of another newly defined ``%{check}`` substitution:
 
 .. code-block:: llvm
 
     ; DEFINE: %{cflags} =

llvm/utils/lit/lit/TestRunner.py

 from __future__ import absolute_import
+import contextlib
 import errno
+import inspect
 import io
 import itertools
 import getopt
 import os, signal, subprocess, sys
 import re
 import stat
 import pathlib
 import platform
 import shlex
 import shutil
 import tempfile
 import threading
+import traceback
 
 import io
 
 try:
     from StringIO import StringIO
 except ImportError:
     from io import StringIO
 
 from lit.ShCommands import GlobItem, Command
 import lit.ShUtil as ShUtil
 import lit.Test as Test
 import lit.util
 from lit.util import to_bytes, to_string, to_unicode
 from lit.BooleanExpression import BooleanExpression
 
 
 class InternalShellError(Exception):
     def __init__(self, command, message):
         self.command = command
         self.message = message
 
 
+class ScriptFatal(Exception):
+    """
+    A script had a fatal error such that there's no point in retrying.  The
+    message has not been emitted on stdout or stderr but is instead included in
+    this exception.
+    """
+
+    def __init__(self, message):
+        super().__init__(message)
+
+
+class ScriptFail(Exception):
+    """
+    A script failed, but it might by worth retrying.  Any failure message has
+    already been emitted on stdout or stderr.
+    """
+
+    def __init__(self, exitCode, timeoutInfo):
+        self.exitCode = exitCode
+        self.timeoutInfo = timeoutInfo
+        self.status = Test.FAIL if timeoutInfo is None else Test.TIMEOUT
+
+
 kIsWindows = platform.system() == "Windows"
 
 # Don't use close_fds on Windows.
 kUseCloseFDs = not kIsWindows
 
 # Use temporary files to replace /dev/null on Windows.
 kAvoidDevNull = kIsWindows
 kDevNull = "/dev/null"
 
 # A regex that matches %dbg(ARG), which lit inserts at the beginning of each
 # run command pipeline such that ARG specifies the pipeline's source line
 # number.  lit later expands each %dbg(ARG) to a command that behaves as a null
 # command in the target shell so that the line number is seen in lit's verbose
 # mode.
 #
 # This regex captures ARG.  ARG must not contain a right parenthesis, which
 # terminates %dbg.  ARG must not contain quotes, in which ARG might be enclosed
 # during expansion.
 #
 # COMMAND that follows %dbg(ARG) is also captured. COMMAND can be
 # empty as a result of conditinal substitution.
-kPdbgRegex = "%dbg\\(([^)'\"]*)\\)(.*)"
+#
+# An empty '%dbg()' means sufficient debug info was already printed.
+kPdbgRegex = "%dbg\\(([^)'\"]*)\\)((?:.|\\n)*)"
 
 
 def buildPdbgCommand(msg, cmd):
     res = f"%dbg({msg}) {cmd}"
     assert re.fullmatch(
         kPdbgRegex, res
     ), f"kPdbgRegex expected to match actual %dbg usage: {res}"
     return res

 #
 # If debug is False (set by some custom lit test formats that call this
 # function), out contains only stdout from the script, err contains only stderr
 # from the script, and there is no execution trace.
 def executeScriptInternal(test, litConfig, tmpBase, commands, cwd, debug=True):
     cmds = []
     for i, ln in enumerate(commands):
         # Within lit, we try to always add '%dbg(...)' to command lines in order
-        # to maximize debuggability.  However, custom lit test formats might not
-        # always add it, so add a generic debug message in that case.
+        # to maximize debuggability, or we add '%dbg()' to indicate a debug
+        # message was already printed in the execution trace.  However, custom
+        # lit test formats might not always add it, so add a generic debug
+        # message in that case.
         match = re.fullmatch(kPdbgRegex, ln)
         if match:
             dbg = match.group(1)
             command = match.group(2)
         else:
             dbg = "command line"
             command = ln
         if debug:
-            ln = f"@echo '# {dbg}' "
+            ln = ""
             if command:
-                ln += f"&& @echo {shlex.quote(command.lstrip())} && {command}"
+                if dbg:  # non-empty %dbg(...)
+                    ln += f"@echo '# {dbg}' && "
+                ln += f"@echo {shlex.quote(command.lstrip())} && {command}"
             else:
-                ln += "has no command after substitutions"
+                if not dbg:  # empty %dbg()
+                    dbg = "command line"
+                ln += f"@echo '# {dbg} has no command after substitutions'"
         else:
             ln = command
         try:
             cmds.append(
                 ShUtil.ShParser(ln, litConfig.isWindows, test.config.pipefail).parse()
             )
         except:
-            return lit.Test.Result(
-                Test.FAIL, f"shell parser error on {dbg}: {command.lstrip()}\n"
-            )
+            raise ScriptFatal(
+                f"shell parser error on {dbg}: {command.lstrip()}\n"
+            ) from None
 
     cmd = cmds[0]
     for c in cmds[1:]:
         cmd = ShUtil.Seq(cmd, "&&", c)
 
     results = []
     timeoutInfo = None
     try:
def add_continuation(self, line_number, keyword, line):
         assert False, "expected method to be called on derived class"
 
     def needs_continuation(self):
         """
         Does this directive require a continuation line?
 
         '\' is documented as indicating a line continuation even if whitespace
         separates it from the newline.  It looks like a line continuation, and
-        it would be confusing if it didn't behave as one.
+        it would be confusing if it didn't behave as one.  (However, for python
+        directives, we just let the python compiler decide how to handle this
+        case.)
         """
         assert False, "expected method to be called on derived class"
 
     def get_location(self):
         """
         Get a phrase describing the line or range of lines so far included by
         this directive and any line continuations.
         """
def adjust_substitutions(self, substitutions):
                 f"does not have the pattern specified by '{self.keyword}' "
                 f"directive {self.get_location()}\n"
                 f"Expected pattern: {self.name}"
                 f"{existing_res}"
             )
         substitutions[existing[0]] = (self.name, value_repl)
 
 
+class PythonDirective(ExpandableScriptDirective):
+    """
+    A lit directive taking a python statement.  For example,
+    'PYTHON: lit.run(cmd)'.
+
+    indent: The indentation found on the first line and expected on continuation
+        lines.  It must be stripped for python to compile the code.
+    body: The uncompiled code accumulated so far from the directive and its
+        continuation lines.
+    """
+
+    def __init__(self, start_line_number, end_line_number, keyword, line):
+        super().__init__(start_line_number, end_line_number, keyword)
+        # Add blank lines so python diagnostics produce correct line numbers.
+        self.body = (start_line_number - 1) * "\n"
+        # Determine the indentation and remove it so that it's valid python.
+        line_stripped = line.lstrip()
+        self.indent = line[0 : len(line) - len(line_stripped)]
+        self.body += line_stripped + "\n"
+
+    def add_continuation(self, line_number, keyword, line):
+        if keyword != self.keyword:
+            return False
+        # Add blank lines so python diagnostics produce correct line numbers.
+        assert (
+            self.end_line_number < line_number
+        ), "expected monotonically increasing line number"
+        self.body += (line_number - self.end_line_number - 1) * "\n"
+        self.end_line_number = line_number
+        # Remove common leading indentation so it's valid python.
+        if not line.startswith(self.indent):
+            raise ValueError(
+                f"'{self.keyword}' directive continuation line has indentation "
+                f"that is inconsistent with previous lines"
+            )
+        self.body += line[len(self.indent) :] + "\n"
+        return True
+
+    def needs_continuation(self):
+        # Unlike lit directives that always mark a continued line with a
+        # trailing '\\', sometimes you cannot tell whether a python directive is
+        # continued until you see the indentation of the next python directive.
+        # In general, lit shouldn't reimplement python's line continuation and
+        # block syntax handling.  Thus, lit just concatenates all consecutive
+        # python directives and compiles them together even if they could be
+        # compiled separately.
+        return False
+
+    @staticmethod
+    def executePython(what, filename, pythonCode, python_dict, log):
+        """
+        Execute python code passed as an argument.
+        """
+        python_dict["__file__"] = filename = os.path.abspath(filename)
+        try:
+            c = compile(pythonCode, filename, "exec")
+        except Exception as e:
+            # Report the exception without this stack frame.  Bothering the lit
+            # user with lit internals is just a distraction.
+            tb = e.__traceback__.tb_next
+            traceText = "".join(traceback.format_exception(None, e, tb))
+            raise ScriptFatal(f"error compiling {what}:\n{traceText}")
+        log(f"# executed {what}\n")
+        outIO = StringIO()
+        errIO = StringIO()
+        traceText = ""
+        try:
+            with contextlib.redirect_stdout(outIO), contextlib.redirect_stderr(errIO):
+                exec(c, python_dict)
+        except ScriptFail:
+            raise
+        except Exception as e:
+            # Report the exception without this stack frame.  Bothering the lit
+            # user with lit internals is just a distraction.
+            tb = e.__traceback__.tb_next
+            traceText = "".join(traceback.format_exception(None, e, tb))
+            raise ScriptFail(1, None)
+        finally:
+            outText = outIO.getvalue()
+            errText = errIO.getvalue() + traceText
+            log(formatOutput(f"stdout from {what}", outText))
+            log(formatOutput(f"stderr from {what}", errText))
+
+    def execute(self, filename, python_dict, log):
+        """
+        Execute the python code in this directive.
+        """
+        self.executePython(
+            f"'{self.keyword}' directive {self.get_location()}",
+            filename,
+            self.body,
+            python_dict,
+            log,
+        )
+
+
 def applySubstitutions(script, substitutions, conditions={}, recursion_limit=None):
     """
     Apply substitutions to the script.  Allow full regular expression syntax.
     Replace each matching occurrence of regular expression pattern a with
     substitution b in line ln.
 
     If a substitution expands into another substitution, it is expanded
     recursively until the line has no more expandable substitutions. If
def applySubstitutions(script, substitutions, conditions={}, recursion_limit=None):
     output = []
     for directive in script:
         if isinstance(directive, SubstDirective):
             directive.adjust_substitutions(substitutions)
         else:
             if isinstance(directive, CommandDirective):
                 line = directive.command
             else:
-                # Can come from preamble_commands.
+                # Can come from preamble_commands or from lit.expand.
                 assert isinstance(directive, str)
                 line = directive
             output.append(unescapePercents(process(line)))
 
     return output
 
 
 class ParserKind(object):
     """
     An enumeration representing the style of an integrated test keyword or
     command.
 
     TAG: A keyword taking no value. Ex 'END.'
     COMMAND: A keyword taking a list of shell commands. Ex 'RUN:'
     LIST: A keyword taking a comma-separated list of values.
     BOOLEAN_EXPR: A keyword taking a comma-separated list of
         boolean expressions. Ex 'XFAIL:'
     INTEGER: A keyword taking a single integer. Ex 'ALLOW_RETRIES:'
     CUSTOM: A keyword with custom parsing semantics.
     DEFINE: A keyword taking a new lit substitution definition. Ex
         'DEFINE: %{name}=value'
     REDEFINE: A keyword taking a lit substitution redefinition. Ex
         'REDEFINE: %{name}=value'
+    PYTHON: A keyword taking a python statement. Ex 'PYTHON: lit.run(cmd)'
     """
 
     TAG = 0
     COMMAND = 1
     LIST = 2
     BOOLEAN_EXPR = 3
     INTEGER = 4
     CUSTOM = 5
     DEFINE = 6
     REDEFINE = 7
+    PYTHON = 8
 
     @staticmethod
     def allowedKeywordSuffixes(value):
         return {
             ParserKind.TAG: ["."],
             ParserKind.COMMAND: [":"],
             ParserKind.LIST: [":"],
             ParserKind.BOOLEAN_EXPR: [":"],
             ParserKind.INTEGER: [":"],
             ParserKind.CUSTOM: [":", "."],
             ParserKind.DEFINE: [":"],
             ParserKind.REDEFINE: [":"],
+            ParserKind.PYTHON: [":"],
         }[value]
 
     @staticmethod
     def str(value):
         return {
             ParserKind.TAG: "TAG",
             ParserKind.COMMAND: "COMMAND",
             ParserKind.LIST: "LIST",
             ParserKind.BOOLEAN_EXPR: "BOOLEAN_EXPR",
             ParserKind.INTEGER: "INTEGER",
             ParserKind.CUSTOM: "CUSTOM",
             ParserKind.DEFINE: "DEFINE",
             ParserKind.REDEFINE: "REDEFINE",
+            ParserKind.PYTHON: "PYTHON",
         }[value]
 
 
 class IntegratedTestKeywordParser(object):
     """A parser for LLVM/Clang style integrated test scripts.
 
     keyword: The keyword to parse for. It must end in either '.' or ':'.
     kind: An value of ParserKind.
def __init__(self, keyword, kind, parser=None, initial_value=None):
         elif kind == ParserKind.DEFINE:
             self.parser = lambda line_number, line, output: self._handleSubst(
                 line_number, line, output, self.keyword, new_subst=True
             )
         elif kind == ParserKind.REDEFINE:
             self.parser = lambda line_number, line, output: self._handleSubst(
                 line_number, line, output, self.keyword, new_subst=False
             )
+        elif kind == ParserKind.PYTHON:
+            self.parser = lambda line_number, line, output: self._handlePython(
+                line_number, line, output, self.keyword
+            )
         else:
             raise ValueError("Unknown kind '%s'" % kind)
 
     def parseLine(self, line_number, line):
         try:
             self.parsed_lines += [(line_number, line)]
             self.value = self.parser(line_number, line, self.value)
         except ValueError as e:
def _handleSubst(cls, line_number, line, output, keyword, new_subst):
             return output
         if output is None:
             output = []
         output.append(
             SubstDirective(line_number, line_number, keyword, new_subst, line)
         )
         return output
 
+    @classmethod
+    def _handlePython(cls, line_number, line, output, keyword):
+        """A parser for PYTHON type keywords"""
+        if output and output[-1].add_continuation(line_number, keyword, line):
+            return output
+        if output is None:
+            output = []
+        output.append(PythonDirective(line_number, line_number, keyword, line))
+        return output
+
 
 def _parseKeywords(sourcepath, additional_parsers=[], require_script=True):
     """_parseKeywords
 
     Scan an LLVM/Clang style integrated test script and extract all the lines
     pertaining to a special parser. This includes 'RUN', 'XFAIL', 'REQUIRES',
     'UNSUPPORTED', 'ALLOW_RETRIES', 'END', 'DEFINE', 'REDEFINE', as well as
     other specified custom parsers.
builtin_parsers = [
         IntegratedTestKeywordParser("REQUIRES:", ParserKind.BOOLEAN_EXPR),
         IntegratedTestKeywordParser("UNSUPPORTED:", ParserKind.BOOLEAN_EXPR),
         IntegratedTestKeywordParser("ALLOW_RETRIES:", ParserKind.INTEGER),
         IntegratedTestKeywordParser("END.", ParserKind.TAG),
         IntegratedTestKeywordParser("DEFINE:", ParserKind.DEFINE, initial_value=script),
         IntegratedTestKeywordParser(
             "REDEFINE:", ParserKind.REDEFINE, initial_value=script
         ),
+        IntegratedTestKeywordParser("PYTHON:", ParserKind.PYTHON, initial_value=script),
     ]
     keyword_parsers = {p.keyword: p for p in builtin_parsers}
 
     # Install user-defined additional parsers.
     for parser in additional_parsers:
         if not isinstance(parser, IntegratedTestKeywordParser):
             raise ValueError(
                 "Additional parser must be an instance of "
def _parseKeywords(sourcepath, additional_parsers=[], require_script=True):
     ):
         parser = keyword_parsers[command_type]
         parser.parseLine(line_number, ln)
         if command_type == "END." and parser.getValue() is True:
             break
 
     # Verify the script contains a run line.
     if require_script and not any(
-        isinstance(directive, CommandDirective) for directive in script
+        isinstance(directive, CommandDirective)
+        or isinstance(directive, PythonDirective)
+        for directive in script
     ):
-        raise ValueError("Test has no 'RUN:' line")
+        raise ValueError("Test has no 'RUN:' line or 'PYTHON:' line")
 
     # Check for unterminated run or subst lines.
     #
     # If, after a line continuation for one kind of directive (e.g., 'RUN:',
     # 'DEFINE:', 'REDEFINE:') in script, the next directive in script is a
     # different kind, then the '\\' remains on the former, and we report it
     # here.
     for directive in script:
try:
         parsed = _parseKeywords(
             test.getSourcePath(), additional_parsers, require_script
         )
     except ValueError as e:
         return lit.Test.Result(Test.UNRESOLVED, str(e))
     script = parsed["RUN:"] or []
     assert parsed["DEFINE:"] == script
     assert parsed["REDEFINE:"] == script
+    assert parsed["PYTHON:"] == script
     test.xfails += parsed["XFAIL:"] or []
     test.requires += parsed["REQUIRES:"] or []
     test.unsupported += parsed["UNSUPPORTED:"] or []
     if parsed["ALLOW_RETRIES:"]:
         test.allowed_retries = parsed["ALLOW_RETRIES:"][0]
 
     # Enforce REQUIRES:
     missing_required_features = test.getMissingRequiredFeatures()
if not test.isWithinFeatureLimits():
             Test.UNSUPPORTED,
             "Test does not require any of the features "
             "specified in limit_to_features: %s" % msg,
         )
 
     return script
 
 
-def _runShTest(test, litConfig, useExternalSh, script, tmpBase):
+# This is used in the lit test suite to check that symbols from lit's
+# implementation are not accidentally made available to 'PYTHON:' directives.
+def pythonDirectiveInaccessibleFunction():
+    assert False, "should be inaccessible from 'PYTHON:' directives"
+
+
+# script must be a list in which each item is either a (1) string that is a
+# shell command or (2) an ExpandableScriptDirective object, which might produce
+# shell commands.
+#
+# If substitutionApplier is not specified, lit substitutions in shell commands
+# are treated as plain text.  Otherwise substitutionApplier must be a function
+# that accepts a list of strings and returns the same list with lit
+# substitutions expanded.
+def _runShTest(
+    test,
+    litConfig,
+    useExternalSh,
+    script,
+    tmpBase,
+    substitutionApplier=lambda x: x,
+):
+    # Make one attempt to run the test.
     def runOnce(execdir):
-        # script is modified below (for litConfig.per_test_coverage, and for
-        # %dbg expansions).  runOnce can be called multiple times, but applying
-        # the modifications multiple times can corrupt script, so always modify
-        # a copy.
-        scriptCopy = script[:]
+        # Counter for litConfig.per_test_coverage file names.
+        perTestCoverageIndex = 0
+
+        # scriptPart has the same constraints as script for _runShTest except
+        # scriptPart cannot contain python directives.
+        def runShScript(scriptPart, addStdout, addStderr):
+            scriptPart = substitutionApplier(scriptPart)
+            # If there were only substitution directives, there's nothing left now.
+            if not scriptPart:
+                return
             # Set unique LLVM_PROFILE_FILE for each run command
             if litConfig.per_test_coverage:
                 # Extract the test case name from the test object, and remove the
                 # file extension.
-            test_case_name = test.path_in_suite[-1]
-            test_case_name = test_case_name.rsplit(".", 1)[0]
-            coverage_index = 0  # Counter for coverage file index
-            for i, ln in enumerate(scriptCopy):
+                testCaseName = test.path_in_suite[-1]
+                testCaseName = testCaseName.rsplit(".", 1)[0]
+                for i, ln in enumerate(scriptPart):
                     match = re.fullmatch(kPdbgRegex, ln)
                     if match:
                         dbg = match.group(1)
                         command = match.group(2)
                     else:
                         command = ln
-                profile = f"{test_case_name}{coverage_index}.profraw"
-                coverage_index += 1
+                    nonlocal perTestCoverageIndex
+                    profile = f"{testCaseName}{perTestCoverageIndex}.profraw"
+                    perTestCoverageIndex += 1
                     command = f"export LLVM_PROFILE_FILE={profile}; {command}"
                     if match:
                         command = buildPdbgCommand(dbg, command)
-                scriptCopy[i] = command
-
+                    scriptPart[i] = command
+            # Execute the remaining shell commands.
             if useExternalSh:
-            res = executeScript(test, litConfig, tmpBase, scriptCopy, execdir)
+                res = executeScript(test, litConfig, tmpBase, scriptPart, execdir)
             else:
-            res = executeScriptInternal(test, litConfig, tmpBase, scriptCopy, execdir)
-        if isinstance(res, lit.Test.Result):
-            return res
-
+                res = executeScriptInternal(
+                    test, litConfig, tmpBase, scriptPart, execdir
+                )
             out, err, exitCode, timeoutInfo = res
-        if exitCode == 0:
-            status = Test.PASS
-        else:
-            if timeoutInfo is None:
-                status = Test.FAIL
+            addStdout(out)
+            addStderr(err)
+            if exitCode != 0:
+                raise ScriptFail(exitCode, timeoutInfo)
+
+        class PythonDirectiveLitAPI(object):
+            """
+            Class for the 'lit' object accessible in 'PYTHON:' directives and in
+            config.prologue.
+            """
+
+            # All members of PythonDirectiveLitAPI are exposed within 'PYTHON:'
+            # directives.  Symbols only required for lit iternals should be declared
+            # outside it.
+
+            # TODO: In the future, extend the 'lit' API to be able to *write*
+            # substitutions?
+
+            @staticmethod
+            def has(feature):
+                """
+                Check if config.available_features indicates a feature is enabled.
+                """
+                return feature in test.config.available_features
+
+            @staticmethod
+            def expand(text):
+                """
+                Expand substitutions in 'text' with their most recent values (from,
+                for example, the most recent 'DEFINE:' and 'REDEFINE:' directives),
+                and return the result.
+                """
+                return substitutionApplier([text])[0]
+
+            @staticmethod
+            def run(cmd):
+                """
+                Execute 'cmd' as if it appeared in a 'RUN:' directive.
+                """
+                msg = ""
+                # In the execution trace, report the call stack for this lit.run
+                # call.  However, skip this stack frame, and then climb the call
+                # stack, adding each frame, until we've reached this file again,
+                # and skip the rest.  That is, bothering the lit user with lit
+                # internals is just a distraction.
+                for caller in inspect.stack()[1:]:
+                    if caller.filename == __file__:
+                        break
+                    loc = f"{caller.filename}:{caller.lineno}"
+                    if not msg:
+                        msg += f"# lit.run called from {loc}"
+                    else:
+                        msg += "\n"
+                        msg += f"#         called from {loc}"
+                print(msg)
+                runShScript(
+                    # Empty "%dbg()" as the above suffices.
+                    [buildPdbgCommand("", cmd)],
+                    lambda s: print(s, end=""),
+                    lambda s: print(s, end="", file=sys.stderr),
+                )
+
+        out = err = ""
+
+        def addStdout(text):
+            nonlocal out
+            out += text
+
+        def addStderr(text):
+            nonlocal err
+            err += text
+
+        python_dict = dict()
+        python_dict["lit"] = PythonDirectiveLitAPI
+        try:
+            # Execute the python prologue, if any.
+            prologue = test.config.prologue
+            if prologue:
+                with open(prologue) as f:
+                    what = f"config.prologue='{prologue}'"
+                    PythonDirective.executePython(
+                        what, prologue, f.read(), python_dict, addStdout
+                    )
+            # Execute the script.
+            scriptRemaining = script
+            while scriptRemaining:
+                # Execute all leading python directives.
+                for pythonDirIndex, pythonDir in enumerate(scriptRemaining):
+                    if not isinstance(pythonDir, PythonDirective):
+                        break
+                    pythonDir.execute(test.getSourcePath(), python_dict, addStdout)
                 else:
-                status = Test.TIMEOUT
-        return out, err, exitCode, timeoutInfo, status
+                    scriptRemaining = []
+                    break
+                scriptRemaining = scriptRemaining[pythonDirIndex:]
+                # Extract all directives before the next python directive into
+                # a script fragment.
+                pythonDirIndex = next(
+                    (
+                        i
+                        for i, d in enumerate(scriptRemaining)
+                        if isinstance(d, PythonDirective)
+                    ),
+                    len(scriptRemaining),
+                )
+                scriptPart = scriptRemaining[:pythonDirIndex]
+                scriptRemaining = scriptRemaining[pythonDirIndex:]
+                # In that script fragment, execute substitution directives and
+                # expand substitutions in shell commands, and then execute the
+                # resulting shell script.
+                runShScript(scriptPart, addStdout, addStderr)
+                # Repeat until nothing's left in the original script.
+        except ScriptFatal as e:
+            out += f"# " + "\n# ".join(str(e).splitlines()) + "\n"
+            return out, err, 1, None, Test.UNRESOLVED
+        except ScriptFail as e:
+            return out, err, e.exitCode, e.timeoutInfo, e.status
+        return out, err, 0, None, Test.PASS
 
     # Create the output directory if it does not already exist.
     lit.util.mkdir_p(os.path.dirname(tmpBase))
 
     # Re-run failed tests up to test.allowed_retries times.
     execdir = os.path.dirname(test.getExecPath())
     attempts = test.allowed_retries + 1
     for i in range(attempts):
         res = runOnce(execdir)
-        if isinstance(res, lit.Test.Result):
-            return res
-
         out, err, exitCode, timeoutInfo, status = res
         if status != Test.FAIL:
             break
 
     # If we had to run the test more than once, count it as a flaky pass. These
     # will be printed separately in the test summary.
     if i > 0 and status == Test.PASS:
         status = Test.FLAKYPASS
if litConfig.noExecute:
         return lit.Test.Result(Test.PASS)
 
     tmpDir, tmpBase = getTempPaths(test)
     substitutions = list(extra_substitutions)
     substitutions += getDefaultSubstitutions(
         test, tmpDir, tmpBase, normalize_slashes=useExternalSh
     )
     conditions = {feature: True for feature in test.config.available_features}
-    script = applySubstitutions(
-        script,
+
+    def substitutionApplier(lines):
+        return applySubstitutions(
+            lines,
             substitutions,
             conditions,
             recursion_limit=test.config.recursiveExpansionLimit,
         )
 
-    return _runShTest(test, litConfig, useExternalSh, script, tmpBase)
+    return _runShTest(
+        test,
+        litConfig,
+        useExternalSh,
+        script,
+        tmpBase,
+        substitutionApplier=substitutionApplier,
+    )

llvm/utils/lit/lit/TestingConfig.py

):
         self.pipefail = pipefail
         self.standalone_tests = standalone_tests
         # This list is used by TestRunner.py to restrict running only tests that
         # require one of the features in this list if this list is non-empty.
         # Configurations can set this list to restrict the set of tests to run.
         self.limit_to_features = set(limit_to_features)
         self.parallelism_group = parallelism_group
         self._recursiveExpansionLimit = None
+        self.prologue = None
 
     @property
     def recursiveExpansionLimit(self):
         return self._recursiveExpansionLimit
 
     @recursiveExpansionLimit.setter
     def recursiveExpansionLimit(self, value):
         if value is not None and not isinstance(value, int):

llvm/utils/lit/tests/Inputs/per-test-coverage-by-lit-cfg/per-test-coverage-by-lit-cfg.py

 # Check that the environment variable is set correctly
 # RUN: %{python} %s | FileCheck -DINDEX=1 %s
-# RUN: %{python} %s | FileCheck -DINDEX=2 %s
+# PYTHON: lit.run("%{python} %s | FileCheck -DINDEX=2 %s")
+# RUN: %{python} %s | FileCheck -DINDEX=3 %s
 
 # Python script to read the environment variable
 # and print its value
 import os
 
 llvm_profile_file = os.environ.get('LLVM_PROFILE_FILE')
 print(llvm_profile_file)
 
 # CHECK: per-test-coverage-by-lit-cfg[[INDEX]].profraw

llvm/utils/lit/tests/Inputs/per-test-coverage/per-test-coverage.py

 # Check that the environment variable is set correctly
 # RUN: %{python} %s | FileCheck -DINDEX=1 %s
-# RUN: %{python} %s | FileCheck -DINDEX=2 %s
+# PYTHON: lit.run("%{python} %s | FileCheck -DINDEX=2 %s")
+# RUN: %{python} %s | FileCheck -DINDEX=3 %s
 
 # Python script to read the environment variable
 # and print its value
 import os
 
 llvm_profile_file = os.environ.get('LLVM_PROFILE_FILE')
 print(llvm_profile_file)
 
 # CHECK: per-test-coverage[[INDEX]].profraw

llvm/utils/lit/tests/Inputs/shtest-define/errors/no-run.txt

 # DEFINE and REDEFINE are not sufficient.  There must be a RUN.
 
 # DEFINE: %{local:echo}=foo
 # REDEFINE: %{global:echo}=bar
 
-# CHECK: Test has no '{{RUN}}:' line
+# CHECK: Test has no '{{RUN}}:' line or '{{PYTHON}}:' line
 
 # CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-python/errors/README.md

+One goal of the tests in this directory is to check that diagnostics from the
+python interpreter are faithfully copied to the lit user.  However, verifying
+the exact text of those diagnostics makes the tests brittle across python
+versions.
+
+Please follow the guide below when adding new tests, and update it if you find
+additional issues.  If you think of a better strategy for checking lit's
+handling of python diagnostics, please propose it.
+
+So far in the tests we have written, the following diagnostic text seems to be
+stable across python versions, and it seems worthwhile to verify that it is
+copied faithfully to the lit user:
+
+  - The traceback.
+  - When the erroneous python statement contains only one line, it is the last
+    line that is quoted from the PYTHON block.
+  - The presence of a caret (`^`) after the last line quoted from the PYTHON
+    block.
+  - The exception, if raised directly by our test code.
+
+The following have proven unstable:
+
+  - The last line that is quoted from the PYTHON block when the erroneous
+    python statement contains multiple lines.
+  - The number of context lines quoted from the PYTHON block.  Of course, if
+    the PYTHON block has only one line, that number is stably zero.
+  - The indentation of the last line quoted from the PYTHON block.  In
+    particular, we have found that, independently of lit, at least python 3.8.0
+    sometimes botches the indentation relative to other quoted lines and the
+    caret, making the diagnostic confusing.
+  - The indentation of the caret, including where it points in the last quoted
+    line.
+  - The exception, including its kind (e.g., `SyntaxError: invalid syntax`), if
+    not raised directly by our test code.
+  - In a compilation error, whether a line quoted from a PYTHON block includes
+    only the python code or the full source line including the `# PYTHON: `
+    prefix.

llvm/utils/lit/tests/Inputs/shtest-python/errors/incomplete-python.txt

+# Incomplete python code is a special error case that some tools for compiling
+# python (e.g., code.compile_command) leave to the caller.  Whether or not lit
+# uses such tools now, make sure lit doesn't start to miss this case.
+
+# PYTHON: if True: #LN
+
+#       CHECK:Command Output (stdout):
+#  CHECK-NEXT:--
+#  CHECK-NEXT:# error compiling '{{PYTHON}}:' directive at line [[#LN: @LINE - 4]]:
+#  CHECK-NEXT:#   File "[[TEST_FILE]]", line [[#LN]]
+#  CHECK-NEXT:# {{ *(# PYTHON[:] )?}}if True: #LN
+#  CHECK-NEXT:# {{ *}}^
+#  CHECK-NEXT:# {{.*Error.*}}
+# CHECK-EMPTY:
+#  CHECK-NEXT:--
+
+# CHECK:  Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-python/errors/inconsistent-indent-2-lines.txt

+# PYTHON:  3 + \
+# PYTHON: 4
+
+#      CHECK:'{{PYTHON}}:' directive continuation line has indentation that is inconsistent with previous lines
+# CHECK-NEXT:in {{PYTHON}}: directive on test line 2
+
+# CHECK:  Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-python/errors/inconsistent-indent-3-lines.txt

+# PYTHON: 3 + \
+# PYTHON: 4 *  \
+# PYTHON:5
+
+#      CHECK:'{{PYTHON}}:' directive continuation line has indentation that is inconsistent with previous lines
+# CHECK-NEXT:in {{PYTHON}}: directive on test line 3
+
+# CHECK:  Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-python/errors/internal-api-inaccessible.txt

+# pythonDirectiveInaccessibleFunction is defined within lit's implementation
+# (specifically for the purposes of the test).  It and other lit implementation
+# symbols should not be accessible to PYTHON directives.
+
+# PYTHON: pythonDirectiveInaccessibleFunction() # LN
+
+#       CHECK:Command Output (stdout):
+#  CHECK-NEXT:--
+#  CHECK-NEXT:# executed '{{PYTHON}}:' directive at line [[#LN: @LINE - 4]]
+#  CHECK-NEXT:# .---stderr from '{{PYTHON}}:' directive at line [[#LN]]
+#  CHECK-NEXT:# | Traceback{{.*}}:
+#  CHECK-NEXT:# |   File "[[TEST_FILE]]", line [[#LN]], in <module>
+#  CHECK-NEXT:# | {{ *}}# {{PYTHON}}: pythonDirectiveInaccessibleFunction() # LN
+#  CHECK-NEXT:# | {{.*Error.*pythonDirectiveInaccessibleFunction.*}}
+#  CHECK-NEXT:# `-----------------------------
+# CHECK-EMPTY:
+#  CHECK-NEXT:--
+
+# CHECK:  Failed: 1

llvm/utils/lit/tests/Inputs/shtest-python/errors/lit.prologue.py

+import inspect, sys
+
+print("config.prologue writes to stdout")
+print("config.prologue writes to stderr", file=sys.stderr)
+line = str(inspect.stack()[0].lineno + 1)
+raise Exception(f"exception in config.prologue at line {line}")

llvm/utils/lit/tests/Inputs/shtest-python/errors/python-syntax-error-1-line.txt

+# PYTHON:+
+
+#       CHECK:Command Output (stdout):
+#  CHECK-NEXT:--
+#  CHECK-NEXT:# error compiling '{{PYTHON}}:' directive at line 1:
+#  CHECK-NEXT:#   File "[[TEST_FILE]]", line 1
+#  CHECK-NEXT:# {{ *(# PYTHON[:])?}}+
+#  CHECK-NEXT:# {{ *}}^
+#  CHECK-NEXT:# {{.*Error.*}}
+# CHECK-EMPTY:
+#  CHECK-NEXT:--
+
+# CHECK:  Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-python/errors/python-syntax-error-2-lines.txt

+# PYTHON:3 +   \
+# PYTHON:  * 4
+
+#       CHECK:Command Output (stdout):
+#  CHECK-NEXT:--
+#  CHECK-NEXT:# error compiling '{{PYTHON}}:' directive from line 1 to 2:
+#  CHECK-NEXT:#   File "[[TEST_FILE]]", line 2
+#       CHECK:# {{ *(# PYTHON[:]  )?}}* 4
+#  CHECK-NEXT:# {{ *}}^
+#  CHECK-NEXT:# {{.*Error.*}}
+# CHECK-EMPTY:
+#  CHECK-NEXT:--
+
+# CHECK:  Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-python/errors/python-syntax-error-3-lines.txt

+# PYTHON:3 +   \
+# PYTHON:  4 * \
+# PYTHON:  / 5
+
+# Some versions of python (e.g., 3.8.0) botch the indentation on line
+# continuations, so we accept any indentation there for now.
+
+#       CHECK:Command Output (stdout):
+#  CHECK-NEXT:--
+#  CHECK-NEXT:# error compiling '{{PYTHON}}:' directive from line 1 to 3:
+#  CHECK-NEXT:#   File "[[TEST_FILE]]", line 3
+#       CHECK:# {{ *(# PYTHON[:]  )?}}/ 5
+#  CHECK-NEXT:# {{ *}}^
+#  CHECK-NEXT:# {{.*Error.*}}
+# CHECK-EMPTY:
+#  CHECK-NEXT:--
+
+# CHECK:  Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-python/errors/python-syntax-error-leading-indent.txt

+# PYTHON:  3 +   \
+# PYTHON:    4 * \
+# PYTHON:    / 5
+
+# Some versions of python (e.g., 3.8.0) botch the indentation on line
+# continuations, so we accept any indentation there for now.
+
+#       CHECK:Command Output (stdout):
+#  CHECK-NEXT:--
+#  CHECK-NEXT:# error compiling '{{PYTHON}}:' directive from line 1 to 3:
+#  CHECK-NEXT:#   File "[[TEST_FILE]]", line 3
+#       CHECK:# {{ *(# PYTHON[:]    )?}}/ 5
+#  CHECK-NEXT:# {{ *}}^
+#  CHECK-NEXT:# {{.*Error.*}}
+# CHECK-EMPTY:
+#  CHECK-NEXT:--
+
+# CHECK:  Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-python/errors/trace-compile-error.txt

+# Check that the execution trace is correct when a PYTHON directive fails to
+# compile.  Specifically, make sure the exception is written at the end of the
+# execution trace and doesn't prevent stdout and stderr from prior RUN
+# directives and python code from appearing.
+
+# PYTHON: import sys                                        #LN
+# PYTHON: print("PYTHON writes to stdout")                  #LN + 1
+# PYTHON: print("PYTHON writes to stderr", file=sys.stderr) #LN + 2
+#    RUN: %{python} %S/../write-to-stdout-and-stderr.py
+# PYTHON: +
+
+#       CHECK:Command Output (stdout):
+#  CHECK-NEXT:--
+#  CHECK-NEXT:# executed '{{PYTHON}}:' directive from line [[#LN : @LINE - 8]] to [[# @LINE - 6]]
+#  CHECK-NEXT:# .---stdout from '{{PYTHON}}:' directive from line [[#LN]] to [[#LN + 2]]
+#  CHECK-NEXT:# | PYTHON writes to stdout
+#  CHECK-NEXT:# `-----------------------------
+#  CHECK-NEXT:# .---stderr from '{{PYTHON}}:' directive from line [[#LN]] to [[#LN + 2]]
+#  CHECK-NEXT:# | PYTHON writes to stderr
+#  CHECK-NEXT:# `-----------------------------
+#  CHECK-NEXT:# {{RUN}}: at line [[#LN + 3]]
+#  CHECK-NEXT:{{.*/python.*}} {{.*}}/write-to-stdout-and-stderr.py
+#  CHECK-NEXT:# executed command: {{.*/python.*}} {{.*}}/write-to-stdout-and-stderr.py
+#  CHECK-NEXT:# .---command stdout------------
+#  CHECK-NEXT:# | a line on stdout
+#  CHECK-NEXT:# `-----------------------------
+#  CHECK-NEXT:# .---command stderr------------
+#  CHECK-NEXT:# | a line on stderr
+#  CHECK-NEXT:# `-----------------------------
+#  CHECK-NEXT:# error compiling '{{PYTHON}}:' directive at line [[#LN + 4]]:
+#  CHECK-NEXT:#   File "[[TEST_FILE]]", line [[#LN + 4]]
+#  CHECK-NEXT:# {{ *(# PYTHON[:] )?}}+
+#  CHECK-NEXT:# {{ *}}^
+#  CHECK-NEXT:# {{.*Error.*}}
+# CHECK-EMPTY:
+#  CHECK-NEXT:--
+
+# CHECK:  Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-python/errors/trace-exception.txt

+# Check that the execution trace is correct when a PYTHON directive raises an
+# exception.  Specifically, make sure the exception is written at the end of the
+# associated stderr and doesn't prevent prior stdout and stderr from appearing.
+# Also, make sure non-directive lines mixed in with the PYTHON directives
+# doesn't throw off the line numbers in the python traceback.
+
+# PYTHON: import sys                                        #LN_START
+
+# Non-directive lines might be used for formatting or comments like this one.
+# PYTHON: print("PYTHON writes to stdout")
+# PYTHON: print("PYTHON writes to stderr", file=sys.stderr)
+#
+#
+# PYTHON: def fn():
+# PYTHON:     raise Exception("fail")
+#
+# PYTHON: fn()                                              #LN_END
+
+#       CHECK:Command Output (stdout):
+#  CHECK-NEXT:--
+#  CHECK-NEXT:# executed '{{PYTHON}}:' directive from line [[#LN_START: @LINE - 14]] to [[#LN_END: @LINE - 4]]
+#  CHECK-NEXT:# .---stdout from '{{PYTHON}}:' directive from line [[#LN_START]] to [[#LN_END]]
+#  CHECK-NEXT:# | PYTHON writes to stdout
+#  CHECK-NEXT:# `-----------------------------
+#  CHECK-NEXT:# .---stderr from '{{PYTHON}}:' directive from line [[#LN_START]] to [[#LN_END]]
+#  CHECK-NEXT:# | PYTHON writes to stderr
+#  CHECK-NEXT:# | Traceback{{.*}}:
+#  CHECK-NEXT:# |   File "[[TEST_FILE]]", line [[#LN_END]], in <module>
+#  CHECK-NEXT:# | {{ *}}# {{PYTHON}}: fn() {{ *}} #LN_END
+#  CHECK-NEXT:# |   File "[[TEST_FILE]]", line [[#LN_END - 2]], in fn
+#  CHECK-NEXT:# | {{ *}}# {{PYTHON}}:     raise Exception("fail")
+#  CHECK-NEXT:# | Exception: fail
+#  CHECK-NEXT:# `-----------------------------
+# CHECK-EMPTY:
+#  CHECK-NEXT:--
+
+# CHECK:  Failed: 1

llvm/utils/lit/tests/Inputs/shtest-python/errors/trace-prologue-exception.txt

+# Check that the execution trace is correct when config.prologue raises an
+# exception.  Specifically, make sure the exception is written at the end of the
+# associated stderr and doesn't prevent prior stdout and stderr from
+# config.prologue from appearing.  However, no PYTHON directive should execute
+# or produce any output.
+
+# PYTHON: print("PYTHON writes to stdout")                  #LN
+# PYTHON: print("PYTHON writes to stderr", file=sys.stderr) #LN + 2
+#
+#      CHECK:Command Output (stdout):
+# CHECK-NEXT:--
+#  CHECK-NEXT:# executed config.prologue='[[PROLOGUE:.*/lit.prologue.py]]'
+#  CHECK-NEXT:# .---stdout from config.prologue='[[PROLOGUE]]'
+#  CHECK-NEXT:# | config.prologue writes to stdout
+#  CHECK-NEXT:# `-----------------------------
+#  CHECK-NEXT:# .---stderr from config.prologue='[[PROLOGUE]]'
+#  CHECK-NEXT:# | config.prologue writes to stderr
+#  CHECK-NEXT:# | Traceback{{.*}}:
+#  CHECK-NEXT:# |   File "[[PROLOGUE]]", line [[#LOC_LN:]], in <module>
+#  CHECK-NEXT:# |     raise Exception({{.*}})
+#  CHECK-NEXT:# | Exception: exception in config.prologue at line [[#LOC_LN]]
+#  CHECK-NEXT:# `-----------------------------
+# CHECK-EMPTY:
+#  CHECK-NEXT:--
+
+# CHECK:  Failed: 1

llvm/utils/lit/tests/Inputs/shtest-python/errors/trace-run-compile-error.txt

+# Check that the execution trace is correct when a RUN directive fails to
+# compile.  Specifically, make sure the error is written at the end of the
+# execution trace and doesn't prevent stdout and stderr from prior RUN
+# directives and python code from appearing.
+#
+# Before PYTHON directives were introduced to lit, this scenario was an
+# impossible case.  There were no PYTHON directives to break RUN directives into
+# multiple scripts, so all RUN directives were compiled before any were
+# executed, so there could not have been prior stdout or stderr.
+
+#    RUN: %{python} %S/../write-to-stdout-and-stderr.py
+# PYTHON: import sys                                        #LN + 1
+# PYTHON: print("PYTHON writes to stdout")                  #LN + 2
+# PYTHON: print("PYTHON writes to stderr", file=sys.stderr) #LN + 3
+#    RUN: &&
+
+#       CHECK:Command Output (stdout):
+#  CHECK-NEXT:--
+#  CHECK-NEXT:# {{RUN}}: at line [[#LN: @LINE - 8]]
+#  CHECK-NEXT:{{.*/python.*}} {{.*}}/write-to-stdout-and-stderr.py
+#  CHECK-NEXT:# executed command: {{.*/python.*}} {{.*}}/write-to-stdout-and-stderr.py
+#  CHECK-NEXT:# .---command stdout------------
+#  CHECK-NEXT:# | a line on stdout
+#  CHECK-NEXT:# `-----------------------------
+#  CHECK-NEXT:# .---command stderr------------
+#  CHECK-NEXT:# | a line on stderr
+#  CHECK-NEXT:# `-----------------------------
+#  CHECK-NEXT:# executed '{{PYTHON}}:' directive from line [[#LN + 1]] to [[#LN + 3]]
+#  CHECK-NEXT:# .---stdout from '{{PYTHON}}:' directive from line [[#LN + 1]] to [[#LN + 3]]
+#  CHECK-NEXT:# | PYTHON writes to stdout
+#  CHECK-NEXT:# `-----------------------------
+#  CHECK-NEXT:# .---stderr from '{{PYTHON}}:' directive from line [[#LN + 1]] to [[#LN + 3]]
+#  CHECK-NEXT:# | PYTHON writes to stderr
+#  CHECK-NEXT:# `-----------------------------
+#  CHECK-NEXT:# shell parser error on RUN: at line [[#LN + 4]]: &&
+# CHECK-EMPTY:
+#  CHECK-NEXT:--
+
+# CHECK:  Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-python/exampleModule.py

+# The importer should assign this to the 'lit' object, or uses of 'lit' below
+# will fail.
+lit = None
+
+import inspect
+
+
+def helloWorldFromLitRun():
+    loc = f"{inspect.stack()[0].filename}:{inspect.stack()[0].lineno + 1}"
+    lit.run(f"echo hello world from lit.run at {loc}")
+
+
+def goodbyeWorldFromLitRun():
+    loc = f"{inspect.stack()[0].filename}:{inspect.stack()[0].lineno + 1}"
+    lit.run(f"echo goodbye world from lit.run at {loc}")

llvm/utils/lit/tests/Inputs/shtest-python/file.txt

+# Check that __file__ is defined in PYTHON lines.
+
+# PYTHON: print(f"test file <{__file__}>")
+
+# CHECK:# | test file <[[TEST_FILE]]>

llvm/utils/lit/tests/Inputs/shtest-python/has.txt

+# PYTHON: assert lit.has('enabledFeature')
+# PYTHON: assert not lit.has('disabledFeature')
+
+# CHECK:  Passed: 1

llvm/utils/lit/tests/Inputs/shtest-python/import.txt

+# For safety, lit doesn't automatically add %S to sys.path.
+#
+# PYTHON: try:
+# PYTHON:     import exampleModule
+# PYTHON: except ModuleNotFoundError:
+# PYTHON:     pass
+# PYTHON: else:
+# PYTHON:     raise Exception("error: sys.path already contains %S")
+
+# Adding %S to sys.path is easy enough from a PYTHON directive.
+#
+# PYTHON: import sys
+# PYTHON: sys.path.append(lit.expand('%S'))
+# PYTHON: import exampleModule
+
+# lit doesn't automatically expose the 'lit' object to an imported module, which
+# might have nothing to do with lit.
+#
+# PYTHON: try:
+# PYTHON:     exampleModule.helloWorldFromLitRun()
+# PYTHON: except AttributeError:
+# PYTHON:     pass
+# PYTHON: else:
+# PYTHON:     raise Exception("error: imported module already sees 'lit'")
+
+# Every function in an imported module could be written to accept the 'lit'
+# object as a parameter, but it seems easier to just expose it once via a PYTHON
+# directive as follows.
+#
+# PYTHON: exampleModule.lit = lit
+# PYTHON: exampleModule.helloWorldFromLitRun()
+# PYTHON: exampleModule.goodbyeWorldFromLitRun()
+#
+# CHECK:# | # | hello world from lit.run at {{.*}}
+# CHECK:# | # | goodbye world from lit.run at {{.*}}
+
+# In summary, here's the boilerplate to set up use of exampleModule in a single
+# test:
+#
+#   import sys
+#   sys.path.append(lit.expand('%S'))
+#   import exampleModule
+#   exampleModule.lit = lit
+#
+# If you have to do this in every test in a test suite, it's easier to use
+# config.prologue.  See the test ./prologue.txt for an example.
+
+# CHECK:  Passed: 1

llvm/utils/lit/tests/Inputs/shtest-python/lit.cfg

+import lit.formats
+
+config.name = "shtest-python"
+config.suffixes = [".txt"]
+# Use lit's internal shell to avoid shell portability issues within RUN lines
+# (e.g., for 'echo' commands in Windows).  Those issues should be orthogonal to
+# the python directive behavior we are trying to test.
+config.test_format = lit.formats.ShTest(execute_external=False)
+config.test_source_root = None
+config.test_exec_root = None
+config.substitutions.append(("%{python}", f'"{sys.executable}"'))
+
+config.available_features.add("enabledFeature")
+
+prologue = lit_config.params.get("addPrologue", None)
+if prologue:
+    config.prologue = os.path.join(os.path.dirname(__file__), prologue)

llvm/utils/lit/tests/Inputs/shtest-python/lit.prologue.py

+# If lit.cfg sets config.prologue to this file, this file executes immediately
+# before the first PYTHON directive almost if it appears in a PYTHON directive.
+# One notable difference is __file__ is set to the absolute location of this
+# file to help with imports.
+
+import os, sys
+
+sys.path.append(os.path.dirname(__file__))
+
+import exampleModule
+
+exampleModule.lit = lit  # so the module can access the lit object
+
+# The following helps us test the execution trace of config.prologue.
+
+import inspect
+
+print("config.prologue writes to stdout")
+print("config.prologue writes to stderr", file=sys.stderr)
+lit.run("true")  # writes nothing to stdout or stderr
+loc = str(inspect.stack()[0].lineno + 1)
+lit.run("echo config.prologue lit.run writes to stdout at line " + loc)
+loc = str(inspect.stack()[0].lineno + 1)
+lit.run("%{python} %S/write-to-stderr.py && echo at line " + loc)
+loc = str(inspect.stack()[0].lineno + 1)
+lit.run("%{python} %S/write-to-stdout-and-stderr.py && echo at line " + loc)
+
+
+# fmt: off
+def localFnA():
+    loc = str(inspect.stack()[0].lineno + 1)
+    lit.run("echo config.prologue lit.run from func at line " + loc)
+def localFnB():
+    localFnA()  # must be at above lit.run line number + 2
+localFnB()  # must be at above lit.run line number + 3
+# fmt: on
+exampleModule.helloWorldFromLitRun()
+
+del inspect, os, sys  # optional: just avoids unnecessary pollution across tests

llvm/utils/lit/tests/Inputs/shtest-python/no-shell-commands.txt

+# Check the case where the entire test file executes python but no shell
+# commands.  This should be fine.
+
+# PYTHON: pass
+
+#       CHECK:Command Output (stdout):
+#  CHECK-NEXT:--
+#  CHECK-NEXT:# executed '{{PYTHON}}:' directive at line [[# @LINE - 4]]
+# CHECK-EMPTY:
+#  CHECK-NEXT:--
+
+# CHECK:  Passed: 1

llvm/utils/lit/tests/Inputs/shtest-python/prologue.txt

+# To avoid the boilerplate discussed in the test ./import.txt to set up use of a
+# module, this test depends on lit.cfg's config.prologue.
+#
+# That config.prologue has writes to stdout/stderr so we can check when it's
+# executed:
+# - It should be executed before any RUN or PYTHON directive.  This test checks
+#   the case when the first directive is a RUN directive.  ./trace.txt checks
+#   the case when the first directive is a PYTHON directive.
+# - It should be executed only once even if there are multiple PYTHON blocks
+#   and regardless of the number of lines in those blocks.
+
+#    RUN: echo after config.prologue executes
+# PYTHON: exampleModule.helloWorldFromLitRun()
+#    RUN: echo start new python block
+# PYTHON: exampleModule.helloWorldFromLitRun()
+# PYTHON: exampleModule.goodbyeWorldFromLitRun()
+
+# CHECK-NOT:config.prologue writes
+#     CHECK:# | config.prologue writes to stdout
+# CHECK-NOT:config.prologue writes
+#     CHECK:# | config.prologue writes to stderr
+# CHECK-NOT:config.prologue writes
+#     CHECK:# | after config.prologue executes
+# CHECK-NOT:config.prologue writes
+#     CHECK:# | # | hello world from lit.run at {{.*}}
+# CHECK-NOT:config.prologue writes
+#     CHECK:# | # | hello world from lit.run at {{.*}}
+# CHECK-NOT:config.prologue writes
+#     CHECK:# | # | goodbye world from lit.run at {{.*}}
+# CHECK-NOT:config.prologue writes
+
+# CHECK:  Passed: 1

llvm/utils/lit/tests/Inputs/shtest-python/shell-affects-python.txt

+# Check that the effect of a RUN directive or a lit.run call in a PYTHON
+# directive can be seen by later python code in PYTHON directives.  That is, the
+# role of PYTHON directives is more than just calling lit.run to insert
+# additional shell commands while building the RUN directives' shell script for
+# later execution.  Instead, PYTHON directives must execute as part of that
+# script.
+
+#    RUN: mkdir -p %t
+# PYTHON: lit.run('touch %t/python-different-block')
+#    RUN: touch %t/run
+
+# PYTHON: import os
+# PYTHON: lit.run('touch %t/python-same-block')
+# PYTHON: for file in sorted(os.listdir(lit.expand('%t'))):
+# PYTHON:   print(file)
+
+# CHECK-COUNT-3:{{.*}} executed command: touch {{.*}}
+#  CHECK-NEXT:# | python-different-block
+#  CHECK-NEXT:# | python-same-block
+#  CHECK-NEXT:# | run
+#  CHECK-NEXT:# `--{{-*}}
+
+# CHECK:  Passed: 1

llvm/utils/lit/tests/Inputs/shtest-python/substs-affects-python.txt

+# Check that the effect of DEFINE/REDEFINE directives can be seen by later
+# PYTHON directives' lit.expand or lit.run.
+
+# PYTHON: def run():
+# PYTHON:     lit.run('echo %{greeting} from lit.run')
+# PYTHON:     greeting = lit.expand('%{greeting}')
+# PYTHON:     print(f"{greeting} from lit.expand")
+
+# DEFINE: %{greeting} = hello
+# PYTHON: run()
+
+# REDEFINE: %{greeting} = goodbye
+# PYTHON: run()
+
+# CHECK-NOT:{{^(hello|goodbye)}}
+#     CHECK:# | # | hello from lit.run
+# CHECK-NOT:{{^(hello|goodbye)}}
+#     CHECK:# | hello from lit.expand
+# CHECK-NOT:{{^(hello|goodbye)}}
+#     CHECK:# | # | goodbye from lit.run
+# CHECK-NOT:{{^(hello|goodbye)}}
+#     CHECK:# | goodbye from lit.expand
+# CHECK-NOT:{{^(hello|goodbye)}}
+
+# CHECK:  Passed: 1

llvm/utils/lit/tests/Inputs/shtest-python/trace.txt

+# The execution trace is captured as stdout under the header "Command Output
+# (stdout)".  This test uses lit's internal shell, which doesn't use the header
+# "Command Output (stderr)".  (When using an external shell, stderr from shell
+# commands is handled differently.)
+#
+#      CHECK:Command Output (stdout):
+# CHECK-NEXT:--
+
+# Execution traces behave as follows.  First, a comment is added to indicate the
+# start of execution of each of the following, and that comment is followed by a
+# header for all captured stdout and a separate header for all captured stderr,
+# if any:
+#
+# - config.prologue.
+# - PYTHON directive block (includes any consecutive PYTHON directives).
+# - lit.run call.  Its trace appears in the stdout of the associated
+#   config.prologue or PYTHON directive block.  Its starting comment includes
+#   locations from the entire call stack up to but not including code in lit's
+#   implementation.
+# - RUN directive (plus line continuations).  Those are checked in detail in
+#   other tests.
+# - Each shell command.  Its trace appears in the stdout of the associated
+#   lit.run call or RUN directive.
+#
+# Execution traces when python code fails in some what are checked in
+# ./errors/trace-*.txt.
+
+# Check the execution trace details described above for config.prologue.
+#
+#  CHECK-NEXT:# executed config.prologue='[[PROLOGUE_FILE:.*/lit.prologue.py]]'
+#  CHECK-NEXT:# .---stdout from config.prologue='[[PROLOGUE_FILE]]'
+#  CHECK-NEXT:# | config.prologue writes to stdout
+#  CHECK-NEXT:# | # lit.run called from [[PROLOGUE_FILE]]:[[#]]
+#  CHECK-NEXT:# | true
+#  CHECK-NEXT:# | # executed command: true
+#  CHECK-NEXT:# | # lit.run called from [[PROLOGUE_FILE]]:[[#LOC_LN:]]
+#  CHECK-NEXT:# | echo config.prologue lit.run writes to stdout at line [[#LOC_LN]]
+#  CHECK-NEXT:# | # executed command: echo config.prologue lit.run writes to stdout at line [[#LOC_LN]]
+#  CHECK-NEXT:# | # .---command stdout------------
+#  CHECK-NEXT:# | # | config.prologue lit.run writes to stdout at line [[#LOC_LN]]
+#  CHECK-NEXT:# | # `-----------------------------
+#  CHECK-NEXT:# | # lit.run called from [[PROLOGUE_FILE]]:[[#LOC_LN:]]
+#  CHECK-NEXT:# | {{.*/python.*}} {{.*}}/write-to-stderr.py && echo at line [[#LOC_LN]]
+#  CHECK-NEXT:# | # executed command: {{.*/python.*}} {{.*}}/write-to-stderr.py
+#  CHECK-NEXT:# | # .---command stderr------------
+#  CHECK-NEXT:# | # | a line on stderr
+#  CHECK-NEXT:# | # `-----------------------------
+#  CHECK-NEXT:# | # executed command: echo at line [[#LOC_LN]]
+#  CHECK-NEXT:# | # .---command stdout------------
+#  CHECK-NEXT:# | # | at line [[#LOC_LN]]
+#  CHECK-NEXT:# | # `-----------------------------
+#  CHECK-NEXT:# | # lit.run called from [[PROLOGUE_FILE]]:[[#LOC_LN:]]
+#  CHECK-NEXT:# | {{.*/python.*}} {{.*}}/write-to-stdout-and-stderr.py && echo at line [[#LOC_LN]]
+#  CHECK-NEXT:# | # executed command: {{.*/python.*}} {{.*}}/write-to-stdout-and-stderr.py
+#  CHECK-NEXT:# | # .---command stdout------------
+#  CHECK-NEXT:# | # | a line on stdout
+#  CHECK-NEXT:# | # `-----------------------------
+#  CHECK-NEXT:# | # .---command stderr------------
+#  CHECK-NEXT:# | # | a line on stderr
+#  CHECK-NEXT:# | # `-----------------------------
+#  CHECK-NEXT:# | # executed command: echo at line [[#LOC_LN]]
+#  CHECK-NEXT:# | # .---command stdout------------
+#  CHECK-NEXT:# | # | at line [[#LOC_LN]]
+#  CHECK-NEXT:# | # `-----------------------------
+#  CHECK-NEXT:# | # lit.run called from [[PROLOGUE_FILE]]:[[#LOC_LN:]]
+#  CHECK-NEXT:# | #         called from [[PROLOGUE_FILE]]:[[#LOC_LN + 2]]
+#  CHECK-NEXT:# | #         called from [[PROLOGUE_FILE]]:[[#LOC_LN + 3]]
+#  CHECK-NEXT:# | echo config.prologue lit.run from func at line [[#LOC_LN]]
+#  CHECK-NEXT:# | # executed command: echo config.prologue lit.run from func at line [[#LOC_LN]]
+#  CHECK-NEXT:# | # .---command stdout------------
+#  CHECK-NEXT:# | # | config.prologue lit.run from func at line [[#LOC_LN]]
+#  CHECK-NEXT:# | # `-----------------------------
+#  CHECK-NEXT:# | # lit.run called from [[LOC_FILE:[^:]*]]:[[#LOC_LN:]]
+#  CHECK-NEXT:# | #         called from [[PROLOGUE_FILE]]:[[#]]
+#  CHECK-NEXT:# | echo hello world from lit.run at [[LOC_FILE]]:[[#LOC_LN]]
+#  CHECK-NEXT:# | # executed command: echo hello world from lit.run at [[LOC_FILE]]:[[#LOC_LN]]
+#  CHECK-NEXT:# | # .---command stdout------------
+#  CHECK-NEXT:# | # | hello world from lit.run at [[LOC_FILE]]:[[#LOC_LN]]
+#  CHECK-NEXT:# | # `-----------------------------
+#  CHECK-NEXT:# `-----------------------------
+#  CHECK-NEXT:# .---stderr from config.prologue='[[PROLOGUE_FILE]]'
+#  CHECK-NEXT:# | config.prologue writes to stderr
+#  CHECK-NEXT:# `-----------------------------
+
+# Repeat the above execution trace checks but now for a PYTHON directive block.
+#
+# Thus, this test checks the case when the first RUN or PYTHON directive is a
+# PYTHON directive.  ./prologue.txt checks the case when it's a RUN directive.
+#
+# PYTHON: import sys                                            #LN_PYTHON
+# PYTHON: print("PYTHON writes to stdout")                      #LN_PYTHON + 1
+# PYTHON: print("PYTHON writes to stderr", file=sys.stderr)     #LN_PYTHON + 2
+# PYTHON: lit.run("true")                                       #LN_PYTHON + 3
+# PYTHON: lit.run("echo PYTHON lit.run writes to stdout")       #LN_PYTHON + 4
+# PYTHON: lit.run("%{python} %S/write-to-stderr.py")            #LN_PYTHON + 5
+# PYTHON: lit.run("%{python} %S/write-to-stdout-and-stderr.py") #LN_PYTHON + 6
+# PYTHON: def localFnA():                                       #LN_PYTHON + 7
+# PYTHON:     lit.run("echo lit.run from func")                 #LN_PYTHON + 8
+# PYTHON: def localFnB():                                       #LN_PYTHON + 9
+# PYTHON:     localFnA()                                        #LN_PYTHON + 10
+# PYTHON: localFnB()                                            #LN_PYTHON + 11
+# PYTHON: exampleModule.helloWorldFromLitRun()                  #LN_PYTHON + 12
+#
+#  CHECK-NEXT:# executed '{{PYTHON}}:' directive from line [[#LN_PYTHON : @LINE - 14]] to [[# @LINE - 2]]
+#  CHECK-NEXT:# .---stdout from '{{PYTHON}}:' directive from line [[#LN_PYTHON]] to [[#LN_PYTHON + 12]]
+#  CHECK-NEXT:# | PYTHON writes to stdout
+#  CHECK-NEXT:# | # lit.run called from [[TEST_FILE]]:[[#LN_PYTHON + 3]]
+#  CHECK-NEXT:# | true
+#  CHECK-NEXT:# | # executed command: true
+#  CHECK-NEXT:# | # lit.run called from [[TEST_FILE]]:[[#LN_PYTHON + 4]]
+#  CHECK-NEXT:# | echo PYTHON lit.run writes to stdout
+#  CHECK-NEXT:# | # executed command: echo PYTHON lit.run writes to stdout
+#  CHECK-NEXT:# | # .---command stdout------------
+#  CHECK-NEXT:# | # | PYTHON lit.run writes to stdout
+#  CHECK-NEXT:# | # `-----------------------------
+#  CHECK-NEXT:# | # lit.run called from [[TEST_FILE]]:[[#LN_PYTHON + 5]]
+#  CHECK-NEXT:# | {{.*/python.*}} {{.*}}/write-to-stderr.py
+#  CHECK-NEXT:# | # executed command: {{.*/python.*}} {{.*}}/write-to-stderr.py
+#  CHECK-NEXT:# | # .---command stderr------------
+#  CHECK-NEXT:# | # | a line on stderr
+#  CHECK-NEXT:# | # `-----------------------------
+#  CHECK-NEXT:# | # lit.run called from [[TEST_FILE]]:[[#LN_PYTHON + 6]]
+#  CHECK-NEXT:# | {{.*/python.*}} {{.*}}/write-to-stdout-and-stderr.py
+#  CHECK-NEXT:# | # executed command: {{.*/python.*}} {{.*}}/write-to-stdout-and-stderr.py
+#  CHECK-NEXT:# | # .---command stdout------------
+#  CHECK-NEXT:# | # | a line on stdout
+#  CHECK-NEXT:# | # `-----------------------------
+#  CHECK-NEXT:# | # .---command stderr------------
+#  CHECK-NEXT:# | # | a line on stderr
+#  CHECK-NEXT:# | # `-----------------------------
+#  CHECK-NEXT:# | # lit.run called from [[TEST_FILE]]:[[#LN_PYTHON + 8]]
+#  CHECK-NEXT:# | #         called from [[TEST_FILE]]:[[#LN_PYTHON + 10]]
+#  CHECK-NEXT:# | #         called from [[TEST_FILE]]:[[#LN_PYTHON + 11]]
+#  CHECK-NEXT:# | echo lit.run from func
+#  CHECK-NEXT:# | # executed command: echo lit.run from func
+#  CHECK-NEXT:# | # .---command stdout------------
+#  CHECK-NEXT:# | # | lit.run from func
+#  CHECK-NEXT:# | # `-----------------------------
+#  CHECK-NEXT:# | # lit.run called from [[LOC_FILE:[^:]*]]:[[#LOC_LN:]]
+#  CHECK-NEXT:# | #         called from [[TEST_FILE]]:[[#LN_PYTHON + 12]]
+#  CHECK-NEXT:# | echo hello world from lit.run at [[LOC_FILE]]:[[#LOC_LN]]
+#  CHECK-NEXT:# | # executed command: echo hello world from lit.run at [[LOC_FILE]]:[[#LOC_LN]]
+#  CHECK-NEXT:# | # .---command stdout------------
+#  CHECK-NEXT:# | # | hello world from lit.run at [[LOC_FILE]]:[[#LOC_LN]]
+#  CHECK-NEXT:# | # `-----------------------------
+#  CHECK-NEXT:# `-----------------------------
+#  CHECK-NEXT:# .---stderr from '{{PYTHON}}:' directive from line [[#LN_PYTHON]] to [[#LN_PYTHON + 12]]
+#  CHECK-NEXT:# | PYTHON writes to stderr
+#  CHECK-NEXT:# `-----------------------------
+
+# This splits apart a PYTHON directive block.  That facilitates writing this
+# test, but it also checks that lit produces separate comments for separate
+# PYTHON directive blocks.
+#
+#         RUN: true
+#  CHECK-NEXT:# {{RUN}}: at line [[# @LINE - 1]]
+#  CHECK-NEXT:true
+#  CHECK-NEXT:# executed command: true
+
+# If there's only stdout from a PYTHON directive block, it should not print the
+# stderr header.  This is also checks the trace when the block contains only one
+# PYTHON directive.
+#
+# PYTHON: print("PYTHON writes only to stdout")
+#
+#  CHECK-NEXT:# executed '{{PYTHON}}:' directive at line [[#LN_PYTHON_STDOUT: @LINE - 2]]
+#  CHECK-NEXT:# .---stdout from '{{PYTHON}}:' directive at line [[#LN_PYTHON_STDOUT]]
+#  CHECK-NEXT:# | PYTHON writes only to stdout
+#  CHECK-NEXT:# `-----------------------------
+
+# Split PYTHON directive blocks so we can check the trace separately.
+#
+#         RUN: true
+#  CHECK-NEXT:# {{RUN}}: at line [[# @LINE - 1]]
+#  CHECK-NEXT:true
+#  CHECK-NEXT:# executed command: true
+
+# If there's only stderr from a PYTHON directive block, it should not print the
+# stdout header.
+#
+# PYTHON: print("PYTHON writes only to stderr", file=sys.stderr)
+#
+#  CHECK-NEXT:# executed '{{PYTHON}}:' directive at line [[#LN_PYTHON_STDERR: @LINE - 2]]
+#  CHECK-NEXT:# .---stderr from '{{PYTHON}}:' directive at line [[#LN_PYTHON_STDERR]]
+#  CHECK-NEXT:# | PYTHON writes only to stderr
+#  CHECK-NEXT:# `-----------------------------
+
+# Split PYTHON directive blocks so we can check the trace separately.
+#
+#         RUN: true
+#  CHECK-NEXT:# {{RUN}}: at line [[# @LINE - 1]]
+#  CHECK-NEXT:true
+#  CHECK-NEXT:# executed command: true
+
+# If there's no stdout or stderr, it shouldn't print either header.
+#
+# PYTHON: pass
+#
+#  CHECK-NEXT:# executed '{{PYTHON}}:' directive at line [[#LN_PYTHON_STDERR: @LINE - 2]]
+
+# Split PYTHON directive blocks so we can check the trace separately.
+#
+#         RUN: true
+#  CHECK-NEXT:# {{RUN}}: at line [[# @LINE - 1]]
+#  CHECK-NEXT:true
+#  CHECK-NEXT:# executed command: true
+
+# Check that reported line numbers are correct when a PYTHON directive block has
+# non-directive lines mixed in.
+#
+# PYTHON: lit.run("true") # LN_PYTHON_SEP_START
+
+# PYTHON: lit.run("true")
+# non-blank line
+# PYTHON: lit.run("true")
+#         non-blank line
+#
+# non-blank line
+# PYTHON: lit.run("true") # LN_PYTHON_SEP_END
+#
+#  CHECK-NEXT:# executed '{{PYTHON}}:' directive from line [[#LN_PYTHON_SEP_START: @LINE - 10]] to [[#LN_PYTHON_SEP_END: @LINE - 2]]
+#  CHECK-NEXT:# .---stdout from '{{PYTHON}}:' directive from line [[#LN_PYTHON_SEP_START]] to [[#LN_PYTHON_SEP_END]]
+#  CHECK-NEXT:# | # lit.run called from [[TEST_FILE]]:[[#LN_PYTHON_SEP_START]]
+#  CHECK-NEXT:# | true
+#  CHECK-NEXT:# | # executed command: true
+#  CHECK-NEXT:# | # lit.run called from [[TEST_FILE]]:[[#LN_PYTHON_SEP_START + 2]]
+#  CHECK-NEXT:# | true
+#  CHECK-NEXT:# | # executed command: true
+#  CHECK-NEXT:# | # lit.run called from [[TEST_FILE]]:[[#LN_PYTHON_SEP_START + 4]]
+#  CHECK-NEXT:# | true
+#  CHECK-NEXT:# | # executed command: true
+#  CHECK-NEXT:# | # lit.run called from [[TEST_FILE]]:[[#LN_PYTHON_SEP_START + 8]]
+#  CHECK-NEXT:# | true
+#  CHECK-NEXT:# | # executed command: true
+#  CHECK-NEXT:# `-----------------------------
+
+# End of "Command Output (stdout)".
+#
+# CHECK-EMPTY:
+#  CHECK-NEXT:--
+
+# CHECK:  Passed: 1

llvm/utils/lit/tests/Inputs/shtest-python/write-to-stderr.py

+#!/usr/bin/env python
+
+import sys
+
+sys.stderr.write("a line on stderr\n")
+sys.stderr.flush()

llvm/utils/lit/tests/Inputs/shtest-python/write-to-stdout-and-stderr.py

+#!/usr/bin/env python
+
+import sys
+
+sys.stdout.write("a line on stdout\n")
+sys.stdout.flush()
+
+sys.stderr.write("a line on stderr\n")
+sys.stderr.flush()

llvm/utils/lit/tests/lit.cfg

 # This test suite calls %{lit} to test lit's behavior for the sample test
 # suites in %{inputs}.  This test suite's results are then determined in part
 # by %{lit}'s textual output, which includes the output of FileCheck calls
 # within %{inputs}'s test suites.  Thus, %{lit} clears environment variables
 # that can affect FileCheck's output.  It also includes "--order=lexical -j1"
 # to ensure predictable test order, as it is often required for FileCheck
 # matches.
 config.substitutions.append(("%{inputs}", "Inputs"))
+config.substitutions.append(("%{inputs-absolute}",
+                             os.path.join(config.test_source_root, "Inputs")))
 config.substitutions.append(("%{lit}", "%{lit-no-order-opt} --order=lexical"))
 config.substitutions.append(
     (
         "%{lit-no-order-opt}",
         "{env} %{{python}} {lit} -j1".format(
             env="env -u FILECHECK_OPTS", lit=os.path.join(lit_path, "lit.py")
         ),
     )

llvm/utils/lit/tests/per-test-coverage-by-lit-cfg.py

 # CHECK-NEXT: --
 #      CHECK: export
 #      CHECK: LLVM_PROFILE_FILE=per-test-coverage-by-lit-cfg0.profraw
 #      CHECK: per-test-coverage-by-lit-cfg.py
 #      CHECK: {{RUN}}: at line 2
 #      CHECK: export
 #      CHECK: LLVM_PROFILE_FILE=per-test-coverage-by-lit-cfg1.profraw
 #      CHECK: per-test-coverage-by-lit-cfg.py
-#      CHECK: {{RUN}}: at line 3
+#      CHECK: {{RUN}}: at line 4
 #      CHECK: export
-#      CHECK: LLVM_PROFILE_FILE=per-test-coverage-by-lit-cfg2.profraw
+#      CHECK: LLVM_PROFILE_FILE=per-test-coverage-by-lit-cfg3.profraw
 #      CHECK: per-test-coverage-by-lit-cfg.py

llvm/utils/lit/tests/per-test-coverage.py

 # CHECK-NEXT: --
 #      CHECK: export
 #      CHECK: LLVM_PROFILE_FILE=per-test-coverage0.profraw
 #      CHECK: per-test-coverage.py
 #      CHECK: {{RUN}}: at line 2
 #      CHECK: export
 #      CHECK: LLVM_PROFILE_FILE=per-test-coverage1.profraw
 #      CHECK: per-test-coverage.py
-#      CHECK: {{RUN}}: at line 3
+#      CHECK: {{RUN}}: at line 4
 #      CHECK: export
-#      CHECK: LLVM_PROFILE_FILE=per-test-coverage2.profraw
+#      CHECK: LLVM_PROFILE_FILE=per-test-coverage3.profraw
 #      CHECK: per-test-coverage.py

llvm/utils/lit/tests/shtest-keyword-parse-errors.py

 # RUN: not %{lit} -vv %{inputs}/shtest-keyword-parse-errors > %t.out
 # RUN: FileCheck -input-file %t.out %s
 #
 # END.
 
 # CHECK: Testing: 3 tests
 
 # CHECK-LABEL: UNRESOLVED: shtest-keyword-parse-errors :: empty.txt
-# CHECK:       {{^}}Test has no 'RUN:' line{{$}}
+# CHECK:       {{^}}Test has no 'RUN:' line or 'PYTHON:' line{{$}}
 
 # CHECK-LABEL: UNRESOLVED: shtest-keyword-parse-errors :: multiple-allow-retries.txt
 # CHECK:       {{^}}Test has more than one ALLOW_RETRIES lines{{$}}
 
 # CHECK-LABEL: UNRESOLVED: shtest-keyword-parse-errors :: unterminated-run.txt
 # CHECK:       {{^}}Test has unterminated 'RUN:' directive (with '\') at line 1{{$}}

llvm/utils/lit/tests/shtest-python.py

+# We're using PYTHON to help us write tests for PYTHON.
+
+# RUN: echo "-- Available Tests --" > %t.tests.actual.txt
+
+# DEFINE: %{my-inputs} = %{inputs-absolute}/shtest-python
+
+# PYTHON: def runTest(test, litPre="", prologue=None):
+# PYTHON:     litCmd = litPre + " %{lit} -va"
+# PYTHON:     if prologue:
+# PYTHON:         litCmd += " -DaddPrologue=" + prologue
+# PYTHON:     testAbs = "%{my-inputs}/" + test
+# PYTHON:     lit.run(f"""
+# PYTHON:         {litCmd} {testAbs} 2>&1 |
+# PYTHON:           FileCheck {testAbs} -strict-whitespace -match-full-lines
+# PYTHON:                               -dump-input-filter=all -vv -color
+# PYTHON:                               -DTEST_FILE='{testAbs}' &&
+# PYTHON:         echo "  shtest-python :: {test}" >> %t.tests.actual.txt
+# PYTHON:     """)
+# PYTHON: def runPass(test, prologue=None):
+# PYTHON:     runTest(test, prologue=prologue)
+# PYTHON: def runFail(test, prologue=None):
+# PYTHON:     runTest(test, litPre="not", prologue=prologue)
+
+# PYTHON: runFail("errors/incomplete-python.txt")
+# PYTHON: runFail("errors/inconsistent-indent-2-lines.txt")
+# PYTHON: runFail("errors/inconsistent-indent-3-lines.txt")
+# PYTHON: runFail("errors/internal-api-inaccessible.txt")
+# PYTHON: runFail("errors/python-syntax-error-1-line.txt")
+# PYTHON: runFail("errors/python-syntax-error-2-lines.txt")
+# PYTHON: runFail("errors/python-syntax-error-3-lines.txt")
+# PYTHON: runFail("errors/python-syntax-error-leading-indent.txt")
+# PYTHON: runFail("errors/trace-compile-error.txt")
+# PYTHON: runFail("errors/trace-exception.txt")
+# PYTHON: runFail("errors/trace-prologue-exception.txt",
+# PYTHON:         prologue="errors/lit.prologue.py")
+# PYTHON: runFail("errors/trace-run-compile-error.txt")
+
+# PYTHON: runPass("file.txt")
+# PYTHON: runPass("has.txt")
+# PYTHON: runPass("import.txt")
+# PYTHON: runPass("no-shell-commands.txt")
+# PYTHON: runPass("prologue.txt", prologue="lit.prologue.py")
+# PYTHON: runPass("shell-affects-python.txt")
+# PYTHON: runPass("substs-affects-python.txt")
+# PYTHON: runPass("trace.txt", prologue="lit.prologue.py")
+
+# Make sure we didn't forget to run something.
+#
+# RUN: %{lit} --show-tests %{my-inputs} > %t.tests.expected.txt
+# RUN: diff -u -w %t.tests.expected.txt %t.tests.actual.txt

llvm/utils/lit/tests/shtest-shell.py

 # CHECK: not-a-real-command
 # CHECK: # .---command stderr{{-*}}
 # CHECK: # | 'not-a-real-command': command not found
 # CHECK: # error: command failed with exit status: 127
 # CHECK: ***
 
 # FIXME: The output here sucks.
 #
-# CHECK: FAIL: shtest-shell :: error-1.txt
+# CHECK: UNRESOLVED: shtest-shell :: error-1.txt
 # CHECK: *** TEST 'shtest-shell :: error-1.txt' FAILED ***
 # CHECK: shell parser error on RUN: at line 3: echo "missing quote
 # CHECK: ***
 
 # CHECK: FAIL: shtest-shell :: error-2.txt
 # CHECK: *** TEST 'shtest-shell :: error-2.txt' FAILED ***
 # CHECK: Unsupported redirect:
 # CHECK: ***

 # CHECK-NEXT: # | {{.b.a.r.}}
 # CHECK-NEXT: # | {{.b.a.z.$}}
 # CHECK-NEXT: # `---{{-*}}
 #  CHECK-NOT: error
 #      CHECK: false
 #      CHECK: ***
 
 # CHECK: PASS: shtest-shell :: valid-shell.txt
-# CHECK: Failed Tests (39)
+# CHECK: Unresolved Tests (1)
+# CHECK: Failed Tests (38)