[lit] Implement DEFINE and REDEFINE directives

[jdenny]

Summary

These directives define per-test lit substitutions. The concept was
discussed at
https://discourse.llvm.org/t/iterating-lit-run-lines/62596/10.

For example, the following directives can be inserted into a test file
to define %{cflags} and %{fcflags} substitutions with empty
initial values, which serve as the parameters of another newly defined
%{check} substitution:

// DEFINE: %{cflags} =
// DEFINE: %{fcflags} =

// DEFINE: %{check} = %clang_cc1 %{cflags} -emit-llvm -o - %s | \ 
// DEFINE:            FileCheck %{fcflags} %s

The following directives then redefine the parameters before each use
of %{check}:

// REDEFINE: %{cflags} = -foo
// REDEFINE: %{fcflags} = -check-prefix=FOO
// RUN: %{check}

// REDEFINE: %{cflags} = -bar
// REDEFINE: %{fcflags} = -check-prefix=BAR
// RUN: %{check}

Of course, %{check} would typically be more elaborate, increasing
the benefit of the reuse.

One issue is that the strings DEFINE: and REDEFINE: already appear
in 5 tests. This patch adjusts those tests not to use those strings.
Our prediction is that, in the vast majority of cases, if a test
author mistakenly uses one of those strings for another purpose, the
text appearing after the string will not happen to have the syntax
required for these directives. Thus, the test author will discover
the mistake immediately when lit reports the syntax error.

This patch also expands the documentation on existing lit substitution
behavior.

[jhenderson]

I haven't got the time to review this right now, but I wanted to say a big thank you for working on this. I've wanted the ability to specify my own substitutions within a test for a long time now. I experimented with a CHECK-DEFINE pattern for FileCheck, since I only needed it in the context of FileCheck check patterns, but this will do the job perfectly: we can use it in conjunction with FileCheck's -D option for crazy shenanigans like reusing the same string in both positive and negative checks, or swapping around the meaning of certain variables (by naming them differently). As a bonus of this design, it also could be used to rerun the same FileCheck invocation, but with different sets of prefixes, without having to repeat everything else.

One question I do have: why do we need both DEFINE and REDEFINE? I personally would think that just DEFINE would be enough, with it implicitly replacing any existing substitution. Or is it something to do with substitution order?

[jdenny]

In D132513#3744850, @jhenderson wrote:

I haven't got the time to review this right now, but I wanted to say a big thank you for working on this.

Sure. Thanks for the quick response.

I experimented with a CHECK-DEFINE pattern for FileCheck, since I only needed it in the context of FileCheck check patterns, but this will do the job perfectly:

Makes sense.

One question I do have: why do we need both DEFINE and REDEFINE? I personally would think that just DEFINE would be enough, with it implicitly replacing any existing substitution. Or is it something to do with substitution order?

I agree that DEFINE would be enough for the core functionality. However, I find that debugging complex substitution expansions can sometimes be painful. I split DEFINE into two directives in order to improve readability and facilitate debugging, as follows.

I work with test suites where lit configuration files define substitutions in terms of other substitutions. Let's say a lit configuration file defines a substitution foo such that its value contains a substitution bar. Now let's say you have a test file that uses foo in a RUN line, and you don't realize that foo uses bar or that bar is a thing at all. However, bar seems like a nice name for a per-test substitution that you want, so you define bar using DEFINE. If DEFINE doesn't complain that bar is already defined, you've corrupted the RUN line, and you'll be lucky if the RUN line fails so you have the opportunity to debug and eventually figure out your mistake. On the other hand, if DEFINE does complain, you will immediately discover your mistake.

Another scenario is where you intend to redefine a substitution but mistype it. Such things do happen: think of all the ways people have tried to diagnose typos in FileCheck prefixes. Again, hopefully the next RUN line will fail and you'll eventually figure out your mistake... or maybe it won't fail given that the intended substitution's stale value didn't produce a failure in previous RUN lines. On the other hand, if REDEFINE complains that the substitution is not already defined, you will immediately discover your mistake.

Finally, I find that substitution expansion order can be confusing at times. This patch tries to address that confusion with a definition-before-reference rule. However, if the only directive is DEFINE, then how do you tell where the original definitions are? Sure, you could look for the first DEFINE for a substitution, but what if the first DEFINE in the file isn't the first definition because there's a definition in the lit configuration file? I think it's easier to understand expansion order if DEFINE always sets substitution expansion order and REDEFINE never does.

[MaskRay]

Thanks for implementing this! The design looks good to me.
I think it is powerful enough for my impending use case which is quite straightforward to reason about, without making the lit domain specific language too complex.

One question I do have: why do we need both DEFINE and REDEFINE? I personally would think that just DEFINE would be enough, with it implicitly replacing any existing substitution. Or is it something to do with substitution order?

I agree that DEFINE would be enough for the core functionality. However, I find that debugging complex substitution expansions can sometimes be painful. I split DEFINE into two directives in order to improve readability and facilitate debugging, as follows.

Having the DEFINE/REDEFINE distinction looks good to me. If a substitution is defined in multiple places, knowing where they are redefined helps (like variable declarations and assignments. Python/Ruby made it difficult.)

llvm/utils/lit/tests/Inputs/shtest-define/errors/assignment/before-name.txt

9

delete blank line

llvm/utils/lit/tests/Inputs/shtest-define/errors/assignment/braces-empty.txt

2

space after DEFINE:

[jdenny]

In D132513#3763209, @MaskRay wrote:

Thanks for implementing this! The design looks good to me.
I think it is powerful enough for my impending use case which is quite straightforward to reason about, without making the lit domain specific language too complex.

Great! For others, that discussion is at https://discourse.llvm.org/t/lit-run-a-run-line-multiple-times-with-different-replacements/64932/15.

Having the DEFINE/REDEFINE distinction looks good to me. If a substitution is defined in multiple places, knowing where they are redefined helps (like variable declarations and assignments.

Good analogy. Perhaps I can work it into the documentation.

Thanks for the review. I'll apply your suggestions on the next update.

[jdenny]

• Fixed 5 existing tests that use the strings DEFINE: and REDEFINE:.
• Applied reviewer suggestions in tests.
• Added declaration/assignment analogy to docs.
• Rebased.

[awarzynski]

Just a drive through review from me. Thanks for working on this!

llvm/docs/TestingGuide.rst

624–627

Why delete "This can be used in tests with multiple RUN lines, which reference test file's line numbers"?

696–698

What happens if these remain undefined? I couldn't find this documented or tested.

llvm/utils/lit/tests/Inputs/shtest-define/errors/continuation/unterminated-define-bad-redefine.txt

14

FIXME

llvm/utils/lit/tests/Inputs/shtest-define/errors/continuation/unterminated-define-redefine.txt

14

FIXME

llvm/utils/lit/tests/Inputs/shtest-define/errors/continuation/unterminated-define-run.txt

10

FIXME

llvm/utils/lit/tests/Inputs/shtest-define/errors/continuation/unterminated-define.txt

9

FIXME

llvm/utils/lit/tests/Inputs/shtest-define/lit.cfg

48

FIXME

[jdenny]

In D132513#3769267, @awarzynski wrote:

Just a drive through review from me. Thanks for working on this!

Thanks for the review.

llvm/docs/TestingGuide.rst

624–627

It seems misleading to me: these substitutions can be used even if there's only one RUN line. Beyond that issue, I'm not sure what the sentence says that the first sentence doesn't already say more clearly.

If I've misunderstood the sentence and lost an important point, please let me know. However, if we keep it, it might need to be generalized to make sense for the new directives.

696–698

If, for example, %{triple} is left undefined, then the REDEFINE: directive for %{triple} below will report an error. This error is documented under the REDEFINE: documentation entry below, and llvm/utils/lit/tests/Inputs/shtest-define/errors/defined-check/redefine-none.txt tests that behavior. Is that what you're looking for?

llvm/utils/lit/tests/Inputs/shtest-define/errors/continuation/unterminated-define-redefine.txt

14

Ah, I thought I had fixed all these. Will do in my next update. Thanks.

[jhenderson]

I've done a bit of a review. Not looked at the testing yet, and I can't say to have fully grasped the details of everything that's going on in the python at this point.

llvm/docs/TestingGuide.rst

901

Nit: I believe it should be "use-case" (hyphenated).

llvm/utils/lit/lit/TestRunner.py

1259

Nit: I'd probably put blank lines between functions, as in some cases, it's easy to miss that you've entered a new function.

1270–1272

How come this only applies to substitutions and not commands?

1280

Is there a test to show that you can have substitutions containing = characters in the value? (I skimmed briefly, but there's a lot of tests, so I may not have spotted it).

1301

Not sure if it's worth caring about, but normally _ would be a valid leading character in an identifier in C and Python, so not permitting it here might cause a little confusion.

1304

"must must" -> "must" (and presumably fix a corresponding test).

1564

dir is a built-in python command, so I'd pick a different name. I also find it a confusing name anyway, because I'd expand it to "directory" not "directive". Any reason not to just name it "directive"?

1571

Nit

1591

Nit: double space after period is inconsistent with the rest of this comment.

1719

Nit: whilst moving this, might as well add the missing trailing full stop.

1836–1838

Ditto comments re. dir name.

1846

Ditto.

[jdenny]

Apply reviewer suggestions:

• Accepted leading _ for name in %{name}. Test what chars are accepted in name.
• Test that = is accepted in a value.
• Fix various typographical and naming issues.

[jdenny]

In D132513#3769641, @jhenderson wrote:

I've done a bit of a review. Not looked at the testing yet, and I can't say to have fully grasped the details of everything that's going on in the python at this point.

Thanks for the review so far.

llvm/docs/TestingGuide.rst

901

Interesting. I didn't recall it was ever written that way. https://www.merriam-webster.com/dictionary/use%20case claims that's less common. https://en.wikipedia.org/wiki/Use_case spells it without the hyphen. A couple greps in llvm-project.git:

$ git grep -i 'use-case' | wc
     47     509    5139
$ git grep -i 'use\s*case' | wc
    276    3310   30237

The hyphen makes sense to me if the phrase is being used as an adjective, but here's it a noun.

llvm/utils/lit/lit/TestRunner.py

1270–1272

_handleCommand already does the rstrip, so the effect is the same. Would it help if I moved that logic into CommandDirective?

1280

Good point. Added one.

1301

For the leading character, I was following the FileCheck prefix documentation. It turns out that doesn't match the FileCheck prefix implementation.

OK, I've extended it to accept the leading _, and I've added a test that all the above characters are indeed accepted.

1304

Good catch. Fixed.

The associated tests check the diagnostic only up to the word "malformed", so there's nothing to update. That was my thinking when I wrote them: don't maintain the verbose description of the syntax repeatedly in the test suite. I'm generally not very consistent about how much of a diagnostic to replicate in a test suite, so let me know if you think I should include a different amount in this case.

1564

Good point. Done.

[awarzynski]

llvm/docs/TestingGuide.rst

624–627

these substitutions can be used even if there's only one RUN line.

That's true, but do they make sense if there's only one RUN line? I don't think so 🤔 . I couldn't find a single example in tree though :( I wrote one myself:

one two three


; RUN: head -%(line+8) %s | tail +%(line+8) | FileCheck %s --check-prefix=BOTTOM
; RUN: head -%(line+7) %s | tail +%(line+7) | FileCheck %s --check-prefix=BOTTOM
; RUN: head -%(line-5) %s | tail +%(line-5) | FileCheck %s --check-prefix=TOP

; TOP: one two three
; BOTTOM: five six seven


five six seven

To me that sentence helps understand the rationale for having %(line+<number>)/%(line-<number>) (as opposed to plain %(line)). Without it I'm just not sure what those substitutions are for.

696–698

Is that what you're looking for?

No :) By "undefined" I meant "empty". My bad, sorry! Basically, I had this scenario in mind:

; DEFINE: %{triple} =
; DEFINE: %{cflags} =
; DEFINE: %{fc-prefix} =

 ; DEFINE: %{check} =                                                  \
 ; DEFINE:   %clang_cc1 -verify -fopenmp -fopenmp-version=51 %{cflags} \
 ; DEFINE:              -triple %{triple} -emit-llvm -o - %s |         \
 ; DEFINE:     FileCheck -check-prefix=%{fc-prefix} %s

What should happen here?

If, for example, %{triple} is left undefined, then the REDEFINE: directive for %{triple} below will report an error.

So this is allowed:

; DEFINE: %{triple} =
; REDEFINE: %{triple} =

and this is not allowed (assuming that there's no DEFINE: %{triple} in any of the local config files):

;  REDEFINE: %{triple} =

Makes sense!

llvm/utils/lit/tests/Inputs/shtest-define/lit.cfg

15

22–23

Why not?

28–29

And I guess that such usage should be discouraged?

33–34

Why not? And what's the difference between %{global:multiple-once-exact} and <%{global:multiple-once-exact}>?

[jdenny]

Applied reviewer suggestions:

• Rewrote examples to avoid confusion over unusable initial substitution values.
• Improved some comments in the test suite.
• Fixed a typo.

llvm/docs/TestingGuide.rst

624–627

I have not found examples of line number substitutions either. I even grepped through the LLVM history.

To me, saying line number substitutions don't make sense unless there are multiple RUN lines suggests that the RUN lines would refer to each other's line numbers. When is that useful?

I can imagine a RUN line referring to another part of the file, such as source code that is expected to produce a diagnostic including a line number. Only one RUN line is required for that to make sense. For example:

<text>foo</txt>
# RUN: tool-under-test %s 2>&1 | diag-check-tool --line-number=%(line-1) 'error: unmatched <text>'

But we normally use FileCheck or clang's -verify to verify output, and they have their own line number support, so the above use case probably doesn't come up in practice.

Does the example from your comment above make sense with a single RUN line? I think so. In that example, the line number substitutions are used to refer to other parts of the test file, but not specifically to the other RUN lines. One RUN line is enough for them to be useful. Did I misunderstand the intent?

If you feel the sentence in question describes a real use case, I might just add the phrase "For example," before it so at least it doesn't imply that's the only use case.

(By the way, I do show an example of using non-relative line number substitutions in D133172.)

696–698

; DEFINE: %{triple} =
; DEFINE: %{cflags} =
; DEFINE: %{fc-prefix} =

 ; DEFINE: %{check} =                                                  \
 ; DEFINE:   %clang_cc1 -verify -fopenmp -fopenmp-version=51 %{cflags} \
 ; DEFINE:              -triple %{triple} -emit-llvm -o - %s |         \
 ; DEFINE:     FileCheck -check-prefix=%{fc-prefix} %s

What should happen here?

I think I get the point now: the initial values for %{triple} and %{fc-prefix} are unusable, so they must be redefined before %{check} is used in a RUN line, or the RUN line will malfunction I see how that quirk is confusing for an introductory example, so I've rewritten the example so that empty values work fine. I've also made it so every use of %{check} redefines every parameter substitution rather than just some, and I think that makes the example easier to read too. Let me know if it's still confusing.

If, for example, %{triple} is left undefined, then the REDEFINE: directive for %{triple} below will report an error.

So this is allowed:

; DEFINE: %{triple} =
; REDEFINE: %{triple} =

and this is not allowed (assuming that there's no DEFINE: %{triple} in any of the local config files):

;  REDEFINE: %{triple} =

Makes sense!

Right.

llvm/utils/lit/tests/Inputs/shtest-define/lit.cfg

22–23

I extended the comments here to explain. TestingGuide.rst documents the restriction generally.

28–29

I would say so. If you use %{global:multiple-exact} somewhere, which value do you expect it to have?

33–34

Why not?

I extended the comments here to explain. TestingGuide.rst documents the restriction generally.

And what's the difference between %{global:multiple-once-exact} and <%{global:multiple-once-exact}>?

I'm not sure I understand the question. One pattern simply contains the other. Can you rephrase?

[jdenny]

llvm/utils/lit/tests/Inputs/shtest-define/examples/param-subst.txt

2

I mean to keep this example in sync with TestingGuide.rst, at least until the patch lands. Hopefully I'll get to that tomorrow.

[jdenny]

Updated the example in the test suite to match the one in the documentation.

[awarzynski]

Thanks for addressing my comments! Here's a few more from me.

I've only skimmed through your changes in "TestRunner.py". I will try to take another look later. Unless @jhenderson approves in the meantime :)

llvm/docs/TestingGuide.rst

624–627

If you feel the sentence in question describes a real use case, I might just add the phrase "For example," before it so at least it doesn't imply that's the only use case.

I don't feel too strongly about this. To me it sounds like we are both unsure about the rationale for having %(line) there in the first place. For this reason, I would suggest either leaving that sentence intact or removing support for %(line) altogether (after having this proposed and accepted on Discourse). However, ....

(By the way, I do show an example of using non-relative line number substitutions in D133172.)

... there is going to be an in-tree example soon :)

I'm happy with whatever you decide here. Personally I'd consider removing %(line), but that's beyond the scope of this patch!

696–698

I've rewritten the example so that empty values work fine.

Nice, this is much clearer now. Thank you!

llvm/utils/lit/lit/TestRunner.py

1221

Docstring? Same for other classes/method.

1247

ScriptDirective.add_continuation takes only 3 arguments.

1315

[nit] You could use f-strings instead.

llvm/utils/lit/tests/Inputs/shtest-define/lit.cfg

28–29

My point is that this particular example could suggest that this usage is fine. I would add a comment explaining that this is bad practice and is only used here to facilitate testing.

[jdenny]

Applied reviewer suggestions:

• Restored some existing %(line) documentation.
• Added python docstrings.
• Fixed parameter list on add_continuation, which was inconsistent between ScriptDirective and derived classes.
• Switched to python f-strings.
• Clarified that some substitutions in test code are discouraged.

llvm/docs/TestingGuide.rst

624–627

If you feel the sentence in question describes a real use case, I might just add the phrase "For example," before it so at least it doesn't imply that's the only use case.

I don't feel too strongly about this. To me it sounds like we are both unsure about the rationale for having %(line) there in the first place.

Agreed.

For this reason, I would suggest either leaving that sentence intact

I've restored the sentence with "For example".

or removing support for %(line) altogether (after having this proposed and accepted on Discourse). However, ....

(By the way, I do show an example of using non-relative line number substitutions in D133172.)

... there is going to be an in-tree example soon :)

I'm happy with whatever you decide here. Personally I'd consider removing %(line), but that's beyond the scope of this patch!

I don't want to remove %(line). I think the use case presented in D133172 has value. I currently use %(line) in that way in my downstream work. I imagine that usage will find its way upstream after these patches land.

Thanks for the discussion. Maybe someone who knows the history will see this and answer our questions.

696–698

Great. Thanks for the feedback. I like the result better too.

llvm/utils/lit/lit/TestRunner.py

1221

Done.

1247

Thanks for catching that. Fixed.

1315

I wasn't aware f-strings. Nice. I've changed all code modified/added by this patch to use those instead of format. Thanks.

llvm/utils/lit/tests/Inputs/shtest-define/lit.cfg

28–29

Good point. I've added a comment before all three of these sets of confusing definitions.

[awarzynski]

Thanks for adding comments in the Python script, that helps a lot! I've only skimmed through for now and have one small comment about the overall approach. I will try to take a look again later today.

llvm/utils/lit/lit/TestRunner.py

1223–1224

Through this class you are effectively re-introducing the notion of "script directives" in LIT, but this class is only meant for RUN, DEFINE and REDEFINE, right? What about XFAIL, REQUIRES or UNSUPPORTED?

If this is only meant for a sub-set of LIT directives, I suggest:

• renaming the class as e.g. OrderedClassDirective (so that it is clear that this does not represent any LIT directive)
• adding an assert in the constructor to make sure that self.keyword is one of RUN, DEFINE or REDEFINE.

An assert will document the code in a way that cannot be ignored by developers who decide to extend this class later ;-)

[jdenny]

llvm/utils/lit/lit/TestRunner.py

1223–1224

Through this class you are effectively re-introducing the notion of "script directives" in LIT, but this class is only meant for RUN, DEFINE and REDEFINE, right? What about XFAIL, REQUIRES or UNSUPPORTED?

Thanks for asking this. I agree the terminology here is confusing.

Even without this patch, "script" is overloaded to mean *at least* three different things in TestRunner.py:

1. The original integrated test script that has all the directives. For example, parseIntegratedTestScriptCommands.
2. The script containing the RUN directives extracted from 1. For example, the script variable in _parseKeywords, the require_script parameter to _parseKeywords and parseIntegratedTestScript, the script parameter to applySubstitutions.
3. The shell script produced after expanding substitutions in 2. For example, executeScriptInternal, executeScript.

With this patch, 2 also includes DEFINE/REDEFINE directives. I chose ScriptDirective to refer to directives in 2 because that's the usage of "script" I saw most often in the code I was modifying.

Should we try to eliminate this overload generally in TestRunner.py? Here are some possible names:

1. IntegratedTestScript
2. UnexpandedScript
3. ShellScript

Maybe there are better names. What do you think?

If this is only meant for a sub-set of LIT directives, I suggest:

• renaming the class as e.g. OrderedClassDirective (so that it is clear that this does not represent any LIT directive)

I'd like to pick something that will fit the name we give script 2 above. If script 2 is OrderedScript, then OrderedDirective works. However, script 3 is also ordered. Script 1 is ordered even though the order doesn't matter for some parts. I think we need something more distinct. Whatever names we choose, I'll be sure to improve my python documentation accordingly (e.g., don't emphasize order so much).

• adding an assert in the constructor to make sure that self.keyword is one of RUN, DEFINE or REDEFINE.

An assert will document the code in a way that cannot be ignored by developers who decide to extend this class later ;-)

Do you want asserts for the CommandDirective and SubstDirective constructors as well? Of course, SubstDirective.adjust_substitutions already asserts about keywords because its behavior is already known to vary based on the keyword.

[awarzynski]

llvm/utils/lit/lit/TestRunner.py

1223–1224

GENERAL DISCUSSION

Thanks for asking this. I agree the terminology here is confusing.

This is way deeper than I expected, thanks for the thorough explanation!

Should we try to eliminate this overload generally in TestRunner.py?

Yes, but probably in a different patch. This one is already quite long. Below are some of my thoughts based on your explanation.

Here are some possible names:

IntegratedTestScript
UnexpandedScript
ShellScript

Maybe there are better names. What do you think?

All in all, I would consider doing some refactoring it in two steps (based on my understanding, which might be incorrect):

1. Clearly define what LIT directives are and what directives are supported (i.e. RUN, XFAIL, REQUIRES ...) . Ideally, every class that manipulates them should clearly define what directives it will process. In the case of ScriptDirective, it's RUN, DEFINE and REDEFINE, right?
2. Hopefully, Step 1. will be a good source of inspiration for renaming things. That's when I would try to fix what "script" means in various classes.

I'm suggesting Step 1. as from your description it sounds like "script" in TestRunner.py refers to the command that follows a directive. At least in it's most general form. That wasn't clear to me at all.

As for Step 2., how about the following classification:

• RawScript (base class),
• ExpandableScript (specialization of RawScript, with substitutions either to be expanded or already expanded).

RENAMING

The refactoring discussed above should happen in a separate patch IMO. In this patch I would just focus on renaming ScriptDirective.

I'd like to pick something that will fit the name we give script 2 above. If script 2 is OrderedScript, then OrderedDirective works. However, script 3 is also ordered.

So what's unique about script 2 then? From your description, it sounds like OrderedScriptWithSubstitutions. Or something similar.

Do you want asserts for the CommandDirective and SubstDirective constructors as well?

Yes, it sounds like these should be only used with specific directives and IMO that should be very clear from the implementation.

Of course, SubstDirective.adjust_substitutions already asserts about keywords

True, but that's quite far from object creation. To me it sounds like SubstDirective should only work for specific directives. You can be strict about it and enforce it during object creation (which sends very clear message to class users and/or other developers) or defer it until that assumption becomes vital. Both approaches will work. I'm merely suggesting to express the intent more explicitly.

1388

I would move this assert to the constructor. In here you can add something like this instead:

if self.keyword != 'REDEFINE:':
  raise ValueError(f"Unsupported directive {self.keyword})")

[jdenny]

• ScriptDirective -> ExpandableScriptDirective, and updated python docs accordingly.
• Fixed and tested support for custom keywords in the case of DEFINE: and REDEFINE:

llvm/utils/lit/lit/TestRunner.py

1223–1224

GENERAL DISCUSSION

Should we try to eliminate this overload generally in TestRunner.py?

Yes, but probably in a different patch.

Agreed.

All in all, I would consider doing some refactoring it in two steps (based on my understanding, which might be incorrect):

I agree that steps like these would improve TestRunner.py. Implementing them myself is more than I want to sign up for right now, but I would be happy to review patches. I would like to finish the DEFINE/REDEFINE series first though.

In the case of ScriptDirective, it's RUN, DEFINE and REDEFINE, right?

Right.

it sounds like "script" in TestRunner.py refers to the command that follows a directive. At least in it's most general form. That wasn't clear to me at all.

I might be missing your point here, but that description doesn't match my view. The first two script forms I identified (the original test file, and the one with only RUN/DEFINE/REDEFINE directives) both have to distinguish directives by the directive keyword. Only the third script form has just shell commands and thus no directive keywords.

As for Step 2., how about the following classification:

• RawScript (base class),

Might be fine. We could discuss more in a separate patch to implement the refactoring.

• ExpandableScript (specialization of RawScript, with substitutions either to be expanded or already expanded).

ExpandableScript works for me.

RENAMING

The refactoring discussed above should happen in a separate patch IMO. In this patch I would just focus on renaming ScriptDirective.

Agreed.

I'd like to pick something that will fit the name we give script 2 above. If script 2 is OrderedScript, then OrderedDirective works. However, script 3 is also ordered.

So what's unique about script 2 then? From your description, it sounds like OrderedScriptWithSubstitutions. Or something similar.

I need to think more about whether the "ordered" property is really the right emphasis here. In my mind, any "script" is an ordered sequence, and google just showed me definitions of "script" that agree, so OrderedScript sounds redundant. Moreover, maybe directives not included in this script form will one day (or already do?) care about order in some way I haven't imagined.

You suggested ExpandableScript above. A consistent name for the directive would then be ExpandableScriptDirective. I think that succinctly captures which stage of the transformation this script and its directives represents. I went ahead and implemented that and updated the python documentation to reflect that name.

Maybe there are better names. I think we both have suggested there's a larger discussion and mostly orthogonal patch to work on, and the naming could be revisited then. Is that ok for now?

Do you want asserts for the CommandDirective and SubstDirective constructors as well?

Yes, it sounds like these should be only used with specific directives and IMO that should be very clear from the implementation.

I tried, but a test failure reminded me that some test suites define their own keywords in place of RUN:. See llvm/utils/lit/tests/unit/TestRunner.py. What if someone wants to do that for DEFINE: or REDEFINE:? I'm not sure whether anyone ever will, but it would be more consistent to permit it.

In other words, we need to go the other way: less strictness about keywords.

I implemented that by extending SubstDirective with a new field indicating whether it defines or redefines. Now, adjust_substitutions doesn't need to check the directive's keyword at all.

I also minimally extended llvm/utils/lit/tests/unit/TestRunner.py to check that custom keywords for DEFINE: and REDEFINE: work as expected. I could probably test much more here, but I haven't found much evidence that keyword customization is used much, so I'm not sure exhaustive testing is worthwhile.

1388

I've removed the assert entirely, as discussed in the other comment.

[jdenny]

Made a small improvement to the testing I just added.

[jdenny]

Addressed a forgotten reviewer comment: coded the handling of whitespace after \ more consistently. Tested it more carefully.

llvm/utils/lit/lit/TestRunner.py

1270–1272

This comment was originally on the python documentation in SubstDirective.needs_continuation.

I've now addressed it by moving the rstrip call from _handleCommand to CommandDirective so that handling of whitespace stripping is consistently enscapsulated in the SubstDirective and CommandDirective classes. I've moved the python documentation to the parent class's needs_continuation.

[awarzynski]

Implementing them myself is more than I want to sign up for right now

Totally understandable!

it sounds like "script" in TestRunner.py refers to the command that follows a directive. At least in it's most general form. That wasn't clear to me at all.

I might be missing your point here, but that description doesn't match my view.

Sounds like our mental models are slightly different. This is the breakdown that derived from our discussion:

; RUN: <script>

<test input>

; <expected output>

To me, RUN: <script> is a "LIT directive" followed by a "shell script" (that may or may not be expandable). I feel that refactoring TestRunner.py to be strict about this distinction would make a lot of things easier. But perhaps I'm oversimplifying this? Also, another patch, another time :)

What if someone wants to do that for DEFINE: or REDEFINE:? I'm not sure whether anyone ever will, but it would be more consistent to permit it.

That sounds like a bit "exotic" use case that complicates a fair few things here, but I agree that consistency is key. Thanks for trying!

Having read and understood the code (thanks for bearing with me and all the extra context), I've skimmed through the documentation again. I've made a couple of minor suggestions, but nothing serious. It reads really well for me. Basically, I'm suggesting keeping the discussion on recursiveExpansionLimit and how it affects DEFINE and REDEFINE separate (perhaps a paragraph on "advanced usage" or "fine details").

Tl;Dr
I won't have any more comments after this. This is a very well written, thoroughly tested and documented patch. Many thanks for all the hard work! It's been a rather long discussion and I really appreciate you seeing this through. This is going to make our testing infrastructure much more powerful and I suspect that LLVM folks will be adopting this in their testing soon :)

llvm/docs/TestingGuide.rst

680

Hm, in practice it's "lit.cfg.py" and "lit.local.cfg.py", but all LIT docs seem to drop the extension 🤔 .

898

[nit] I'd be tempted to add "(e.g. lit.cfg or lit.local.cfg)" To avoid any potential ambiguity.

902–906

"confusing" and "insufficient" are a bit ambiguous and vague. Sometimes less is more and I would skip this point. Instead, I would mention above (after "There's a simple way to remember the required definition order in a test file: define a substitution before you refer to it.") that "This behavior can be overridden by specifying recursiveExpansionLimit). An example would also be super helpful! You could use the contents of "recursiveExpansionLimit.txt".

907–909

923–924

[nit] I would remove this part to keep this short and to the point

[jdenny]

In D132513#3782848, @awarzynski wrote:

Sounds like our mental models are slightly different. This is the breakdown that derived from our discussion:

It sounds like this is just a difference in the way we're using the terminology. Clear terminology is indeed important, but...

Also, another patch, another time :)

.... yes, I think it's reasonable to defer.

Basically, I'm suggesting keeping the discussion on recursiveExpansionLimit and how it affects DEFINE and REDEFINE separate (perhaps a paragraph on "advanced usage" or "fine details").

Thanks for those comments. I'll pursue them soon.

I won't have any more comments after this.

But feel free if something comes up. Once you've hit accept, I'll likely stall for about a week anyway to give other reviewers a chance to catch up.

This is a very well written, thoroughly tested and documented patch. Many thanks for all the hard work! It's been a rather long discussion and I really appreciate you seeing this through. This is going to make our testing infrastructure much more powerful and I suspect that LLVM folks will be adopting this in their testing soon :)

Thanks for saying so, and thank you for your review, which has definitely improved the patch. I appreciate thorough reviews, especially on patches affecting LLVM's testing infrastructure as they can easily impact all LLVM developers.

llvm/docs/TestingGuide.rst

680

Hm, in practice it's "lit.cfg.py" and "lit.local.cfg.py", but all LIT docs seem to drop the extension 🤔 .

All four are used. Lit even permits some customization (see --config-prefix). So these are just common examples.

680

You suggested dropping "test/" here. I assume "test/" was meant to hint that lit.cfg is typically at the root directory of a test suite. Maybe we should add a corresponding hint for lit.local.cfg, as in my above suggestion. Does that help?

923–924

By "this part", do you mean "even if `recursiveExpansionLimit` is not used" or the whole sentence starting with "Thus"?

[awarzynski]

llvm/docs/TestingGuide.rst

680

In general, I feel that adding "test/" might be a bit confusing. What about test sub-directories like "<project-root-dir>/test/Lowering/" in which case "Lowering/lit.local.cfg" could be more accurate than "test/lit.local.cfg". That's why I would drop "test/" in "test/lit.local.cfg". If you do want to keep it, then ...

Maybe we should add a corresponding hint for lit.local.cfg, as in my above suggestion.

+ 1

923–924

Yes, "even if `recursiveExpansionLimit`". I was hoping that Phabricator would highlight it.

[jdenny]

Addressed reviewer concerns:

• Relocated and expanded recursiveExpansionLimit documentation with an example in terms of DEFINE/REDEFINE.
• Trimmed other discussions of recursiveExpansionLimit.
• Drop "test/" from "(e.g., test/lit.cfg or lit.local.cfg)".

llvm/docs/TestingGuide.rst

898

There are now several mentions of lit configuration files between here and the original description, which has "(e.g., lit.cfg or lit.local.cfg)". To me, it seems arbitrary to give examples at this particular mention. See how it reads to you now. If you still feel it would be helpful, I'll add it.

902–906

Instead, I would mention above (after "There's a simple way to remember the required definition order in a test file: define a substitution before you refer to it.") that "This behavior can be overridden by specifying recursiveExpansionLimit).

Because this bulleted list is meant to be a general description of substitution behavior, I feel it's better to keep this point here than to complicate the introductory example with it. However, I did abbreviate the description and append it to the bullet that describes the default single pass over the substitution list, where it seems like a natural fit.

An example would also be super helpful! You could use the contents of "recursiveExpansionLimit.txt".

Done. I'm using that to replace the original recursiveExpansionLimit documentation.

[awarzynski]

LGTM!

Great job, thank you!

[jhenderson]

I've run out of time to look at this further for now, but hopefully should be able to go through the rest of the testing later in the week.

llvm/docs/TestingGuide.rst

693–694

The part from "as well as" doesn't read well for me. I think it's probably clearer to rephrase as "... to define %{cflags} and %{fcflags} substitutions with empty initial values, which serve as the parameters of another newly-defined %{check} substitution:". WDYT?

776–777

On first read, I thought the reference to patterns containing %{name} meant something like %{foo} couldn't be referenced with a definition/redefinition if %{afooa} existed. Is it actually meant to be referring to something like %{a%{foo}a}? If so, I feel like an example here could be useful. If not (i.e. %{foo} can't be used if e.g. %{afooa} is defined), I find this to be a problem, as I think it is not unreasonable to have e.g. `%{check} and %{check2} in the same substitution set.

796–797

Would it be clearer here to refer to the config files, since I believe that's the only way to specify an "illegal" pattern? Something like "However, substitution patterns defined in configuration files are not restricted to this form...".

822–823

I'm not sure the bit about "against the advice" makes a huge amount of sense, since the previous section explained the situation and what happens, I thought? I'd just omit the bit between the commas, personally.

832–833

I'd suggest replacing the first "Because" with something else, e.g. "Since", to avoid the double because in this sentence.

857

llvm/utils/lit/lit/TestRunner.py

1277

Are there any cases where this isn't a RUN directive? Similarly, are there any cases where a RUN directive isn't this? If not to both, I'd consider renaming to RunDirective so that it marries up with the actual name used in the test files.

1302

I'd suggest this slight addition.

1329

I think this is strictly more correct, which could be important in some edge cases, I guess.

llvm/utils/lit/tests/Inputs/shtest-define/errors/continuation/unterminated-run-redefine.txt

4–5

There's nothing particularly wrong with this, but why the two run lines in this case (unlike the define case)?

[jdenny]

Addressed reviewer comments:

• Applied various wording improvements in the documentation.
• Adjusted a test for consistency.

[jdenny]

@awarzynski, @jhenderson: Thanks!

llvm/docs/TestingGuide.rst

776–777

On first read, I thought the reference to patterns containing %{name} meant something like %{foo} couldn't be referenced with a definition/redefinition if %{afooa} existed. Is it actually meant to be referring to something like %{a%{foo}a}?

The full %{name} is the substitution's pattern, so the latter is what I meant.

If so, I feel like an example here could be useful.

Done.

796–797

The other possibility I was thinking of is a substitution always defined by lit itself. For example, it's best not to define a substitution whose pattern starts with %s.

That's relevant to the first part of the bullet (why braces are required here). However, as far as I know, it's not currently relevant to the second part (I know of no existing lit-defined substitution whose pattern would overlap with the braced form). Lit evolves constantly, so that could change. For example, what if someone thinks %{name([0-9]*)} is a reasonable pattern (where parens are capturing)?

I've expanded "not defined by these directives" to mention both sources of substitutions. Let me know if that's not clear enough.

822–823

I'm not sure the bit about "against the advice" makes a huge amount of sense, since the previous section explained the situation and what happens, I thought? I'd just omit the bit between the commas, personally.

822–823

Done.

I was trying to make sure the reader knows this example is not the recommended way to write directives. Instead, I've added a comment next to the directives, which should help anyone skimming the documentation for examples and not reading the text.

llvm/utils/lit/lit/TestRunner.py

1277

Are there any cases where this isn't a RUN directive?

Yes. See MY_RUN in llvm/utils/lit/tests/Inputs/testrunner-custom-parsers/test.txt and llvm/utils/lit/tests/unit/TestRunner.py.

I chose "CommandDirective" for consistency with _handleCommand and ParserKind.COMMAND.

Similarly, are there any cases where a RUN directive isn't this?

I don't think so.

llvm/utils/lit/tests/Inputs/shtest-define/errors/continuation/unterminated-run-redefine.txt

4–5

I've forgotten. I think it was a lazy way to make sure this check is performed when continuation lines are themselves continued.

For now, I've made them consistent. I'll think about the right way to check that case separately.

[jdenny]

llvm/docs/TestingGuide.rst

768

Ah, I forgot brackets are special in patterns. I'll escape them.

[jhenderson]

Mostly just comment nitpicks from me. I don't really have time to think about test coverage, so that's about it from me, aside from any follow-up comments.

llvm/utils/lit/lit/TestRunner.py

1277

To clarify, I meant, any cases other than if someone specifies a different name for the "RUN" directive explicitly. Given that that is a specific overriding of the default behaviour, I think my suggestion still stands, but up to you (I don't feel strongly about it).

llvm/utils/lit/tests/Inputs/shtest-define/expansion-order.txt

2

Not sure I follow the bit after the colon? Are you saying that %{global:greeting} is the parameter for %{global:echo}? If so, I'd just move the name to between "the" and "parameter" earlier in the line.

llvm/utils/lit/tests/Inputs/shtest-define/ws-and-continuations.txt

2

Just wondering if it's worth a test case with two adjacent substitutions without separating whitespace? I.e. something like %{foo}%{bar}? I'd expect both to expand, with no separating whitespace added. Not sure where the test case would belong.

9

Nit: whitespace is usually all one word, right?

19

In this comment you're being inconsistent :)

69

What's with the "..."?

llvm/utils/lit/tests/shtest-define.py

12–13

Nit: a slightly more natural flow would be to have the record-test substitution after the run-test one, since that is the order they're used in.

[jdenny]

Addressed remaining comments:

• Fixed an improperly escaped pattern in the docs.
• Extended tests a little, polished their comments, etc.

I also extended ws-and-continuations.txt with a couple checks that somehow ended up in D133172 but make sense here.

llvm/utils/lit/lit/TestRunner.py

1277

I'm not sure if it matters to your point, but MY_RUN doesn't override RUN. It's an additional keyword parsed as a COMMAND.

In any case, I feel more comfortable using the term Command because that's the precedent already set by _handleCommand and ParserKind.COMMAND. If we want to rethink that terminology, I feel like that should be a separate patch.

llvm/utils/lit/tests/Inputs/shtest-define/errors/continuation/unterminated-run-redefine.txt

4–5

I decided it's not worthwhile to roughly double the number of unterminated tests in order to check every combination of conditions. Instead, I added only two to offer some assurance that unterminated continuation lines are checked for the new directives added by this patch:

• unterminated-define-continuation.txt
• unterminated-redefine-continuation.txt

llvm/utils/lit/tests/Inputs/shtest-define/expansion-order.txt

2

Done.

llvm/utils/lit/tests/Inputs/shtest-define/ws-and-continuations.txt

2

I've added a test to this file.

9

I've seen it written both ways, but I should at least be consistent within the same code. Thanks.

19

Oops. :-)

69

I've reworded.

llvm/utils/lit/tests/shtest-define.py

12–13

Done.

[jhenderson]

LGTM, thanks!

[jdenny]

@jhenderson: Thanks for the review!

All: There has been no discussion so far on the directive name collisions, as described in the review summary:

One issue is that the strings DEFINE: and REDEFINE: already appear in 5 tests. This patch adjusts those tests not to use those strings. Alternatively, maybe lit keywords from now on should start with LIT- to avoid collisions. For this patch, we might use LIT-DEF: and LIT-REDEF:.

If I don't hear any opinions on this within a couple days, I'll go ahead and land the patch with the current names (DEFINE: and REDEFINE:). Thanks.

[jhenderson]

In D132513#3802768, @jdenny wrote:

@jhenderson: Thanks for the review!

All: There has been no discussion so far on the directive name collisions, as described in the review summary:

One issue is that the strings DEFINE: and REDEFINE: already appear in 5 tests. This patch adjusts those tests not to use those strings. Alternatively, maybe lit keywords from now on should start with LIT- to avoid collisions. For this patch, we might use LIT-DEF: and LIT-REDEF:.

If I don't hear any opinions on this within a couple days, I'll go ahead and land the patch with the current names (DEFINE: and REDEFINE:). Thanks.

My personal opinion is that it's fine as long as there is pretty much zero risk of a downstream user running into spurious passes due to the change from a FileCheck directive to a lit command. In 99% of cases, I think it'll likely fail due to an invalid definition, so I think we're good, personally. (Plus strictly, LIT-DEF could already be in use downstream anyway).

[jdenny]

In D132513#3802792, @jhenderson wrote:

My personal opinion is that it's fine as long as there is pretty much zero risk of a downstream user running into spurious passes due to the change from a FileCheck directive to a lit command. In 99% of cases, I think it'll likely fail due to an invalid definition, so I think we're good, personally. (Plus strictly, LIT-DEF could already be in use downstream anyway).

I agree, but I wasn't confident. Thanks for giving a second opinion.

OK, I'll try to land tomorrow provided no one else comments on the patch.

[jdenny]

It was a small battle to get the tests past the windows bots, but they eventually succeeded:

https://lab.llvm.org/buildbot/#/builders/216/builds/10125

I pushed the following additional patches while trying to fix the tests for windows:

• f47a5df92d488bdff418e3b1d249f407885896f2
• 387924b307ae992cd4bf19908bdd5e4833063ce1
• b9735db6464cf0bb4f60077b9f5495c0fa372ecf
• 88f183c0db71579f06cfca606c333bce7084074e

b9735db6464cf0bb4f60077b9f5495c0fa372ecf adds a fixme and workaround for the issue that echo is doubling backslashes under windows. I don't think that's specific to the DEFINE/REDEFINE implementation, and my hunch is that it's not a critical issue. It would take me a while to produce a windows build, so I won't investigate that further anytime soon. If someone else wants to, please feel free.

If anything looks objectionable, I can amend further or revert the whole series.

[awarzynski]

I've skimmed through - these fixes make sense to me. Thank you for the quick summary!

[jdenny]

Thanks for reviewing!

clang/test/CodeGen/attr-noundef.cpp

 // TODO: No structs may currently be marked noundef
 
 namespace check_structs {
 struct Trivial {
   int a;
 };
 Trivial ret_trivial() { return {}; }
 void pass_trivial(Trivial e) {}
-// CHECK-INTEL: [[DEFINE:define( dso_local)?]] i32 @{{.*}}ret_trivial
-// CHECK-AARCH: [[DEFINE:define( dso_local)?]] i32 @{{.*}}ret_trivial
-// CHECK-INTEL: [[DEFINE]] void @{{.*}}pass_trivial{{.*}}(i32 %
-// CHECK-AARCH: [[DEFINE]] void @{{.*}}pass_trivial{{.*}}(i64 %
+// CHECK-INTEL: [[DEF:define( dso_local)?]] i32 @{{.*}}ret_trivial
+// CHECK-AARCH: [[DEF:define( dso_local)?]] i32 @{{.*}}ret_trivial
+// CHECK-INTEL: [[DEF]] void @{{.*}}pass_trivial{{.*}}(i32 %
+// CHECK-AARCH: [[DEF]] void @{{.*}}pass_trivial{{.*}}(i64 %
 
 struct NoCopy {
   int a;
   NoCopy(NoCopy &) = delete;
 };
 NoCopy ret_nocopy() { return {}; }
 void pass_nocopy(NoCopy e) {}
-// CHECK: [[DEFINE]] void @{{.*}}ret_nocopy{{.*}}(%"struct.check_structs::NoCopy"* noalias sret({{[^)]+}}) align 4 %
-// CHECK: [[DEFINE]] void @{{.*}}pass_nocopy{{.*}}(%"struct.check_structs::NoCopy"* noundef %
+// CHECK: [[DEF]] void @{{.*}}ret_nocopy{{.*}}(%"struct.check_structs::NoCopy"* noalias sret({{[^)]+}}) align 4 %
+// CHECK: [[DEF]] void @{{.*}}pass_nocopy{{.*}}(%"struct.check_structs::NoCopy"* noundef %
 
 struct Huge {
   int a[1024];
 };
 Huge ret_huge() { return {}; }
 void pass_huge(Huge h) {}
-// CHECK: [[DEFINE]] void @{{.*}}ret_huge{{.*}}(%"struct.check_structs::Huge"* noalias sret({{[^)]+}}) align 4 %
-// CHECK: [[DEFINE]] void @{{.*}}pass_huge{{.*}}(%"struct.check_structs::Huge"* noundef
+// CHECK: [[DEF]] void @{{.*}}ret_huge{{.*}}(%"struct.check_structs::Huge"* noalias sret({{[^)]+}}) align 4 %
+// CHECK: [[DEF]] void @{{.*}}pass_huge{{.*}}(%"struct.check_structs::Huge"* noundef
 } // namespace check_structs
 
 //************ Passing unions by value
 // No unions may be marked noundef
 
 namespace check_unions {
 union Trivial {
   int a;
 };
 Trivial ret_trivial() { return {}; }
 void pass_trivial(Trivial e) {}
-// CHECK-INTEL: [[DEFINE]] i32 @{{.*}}ret_trivial
-// CHECK-AARCH: [[DEFINE]] i32 @{{.*}}ret_trivial
-// CHECK-INTEL: [[DEFINE]] void @{{.*}}pass_trivial{{.*}}(i32 %
-// CHECK-AARCH: [[DEFINE]] void @{{.*}}pass_trivial{{.*}}(i64 %
+// CHECK-INTEL: [[DEF]] i32 @{{.*}}ret_trivial
+// CHECK-AARCH: [[DEF]] i32 @{{.*}}ret_trivial
+// CHECK-INTEL: [[DEF]] void @{{.*}}pass_trivial{{.*}}(i32 %
+// CHECK-AARCH: [[DEF]] void @{{.*}}pass_trivial{{.*}}(i64 %
 
 union NoCopy {
   int a;
   NoCopy(NoCopy &) = delete;
 };
 NoCopy ret_nocopy() { return {}; }
 void pass_nocopy(NoCopy e) {}
-// CHECK: [[DEFINE]] void @{{.*}}ret_nocopy{{.*}}(%"union.check_unions::NoCopy"* noalias sret({{[^)]+}}) align 4 %
-// CHECK: [[DEFINE]] void @{{.*}}pass_nocopy{{.*}}(%"union.check_unions::NoCopy"* noundef %
+// CHECK: [[DEF]] void @{{.*}}ret_nocopy{{.*}}(%"union.check_unions::NoCopy"* noalias sret({{[^)]+}}) align 4 %
+// CHECK: [[DEF]] void @{{.*}}pass_nocopy{{.*}}(%"union.check_unions::NoCopy"* noundef %
 } // namespace check_unions
 
 //************ Passing `this` pointers
 // `this` pointer must always be defined
 
 namespace check_this {
 struct Object {
   int data[];

 namespace check_vecs {
 typedef int __attribute__((vector_size(12))) i32x3;
 i32x3 ret_vec() {
   return {};
 }
 void pass_vec(i32x3 v) {
 }
 
-// CHECK: [[DEFINE]] noundef <3 x i32> @{{.*}}ret_vec{{.*}}()
-// CHECK-INTEL: [[DEFINE]] void @{{.*}}pass_vec{{.*}}(<3 x i32> noundef %
-// CHECK-AARCH: [[DEFINE]] void @{{.*}}pass_vec{{.*}}(<4 x i32> %
+// CHECK: [[DEF]] noundef <3 x i32> @{{.*}}ret_vec{{.*}}()
+// CHECK-INTEL: [[DEF]] void @{{.*}}pass_vec{{.*}}(<3 x i32> noundef %
+// CHECK-AARCH: [[DEF]] void @{{.*}}pass_vec{{.*}}(<4 x i32> %
 } // namespace check_vecs
 
 //************ Passing exotic types
 // Function/Array pointers, Function member / Data member pointers, nullptr_t, ExtInt types
 
 namespace check_exotic {
 struct Object {
   int mfunc();
_BitInt(3) ret_BitInt() {
   return 0;
 }
 void pass_BitInt(_BitInt(3) e) {
 }
 void pass_large_BitInt(_BitInt(127) e) {
 }
 
 // Pointers to arrays/functions are always noundef
-// CHECK: [[DEFINE]] noundef [32 x i32]* @{{.*}}ret_arrptr{{.*}}()
-// CHECK: [[DEFINE]] noundef i32 (i32)* @{{.*}}ret_fnptr{{.*}}()
+// CHECK: [[DEF]] noundef [32 x i32]* @{{.*}}ret_arrptr{{.*}}()
+// CHECK: [[DEF]] noundef i32 (i32)* @{{.*}}ret_fnptr{{.*}}()
 
 // Pointers to members are never noundef
-// CHECK: [[DEFINE]] i64 @{{.*}}ret_mdptr{{.*}}()
-// CHECK-INTEL: [[DEFINE]] { i64, i64 } @{{.*}}ret_mfptr{{.*}}()
-// CHECK-AARCH: [[DEFINE]] [2 x i64] @{{.*}}ret_mfptr{{.*}}()
+// CHECK: [[DEF]] i64 @{{.*}}ret_mdptr{{.*}}()
+// CHECK-INTEL: [[DEF]] { i64, i64 } @{{.*}}ret_mfptr{{.*}}()
+// CHECK-AARCH: [[DEF]] [2 x i64] @{{.*}}ret_mfptr{{.*}}()
 
 // nullptr_t is never noundef
-// CHECK: [[DEFINE]] i8* @{{.*}}ret_npt{{.*}}()
-// CHECK: [[DEFINE]] void @{{.*}}pass_npt{{.*}}(i8* %
+// CHECK: [[DEF]] i8* @{{.*}}ret_npt{{.*}}()
+// CHECK: [[DEF]] void @{{.*}}pass_npt{{.*}}(i8* %
 
 // TODO: for now, ExtInt is only noundef if it is sign/zero-extended
-// CHECK-INTEL: [[DEFINE]] noundef signext i3 @{{.*}}ret_BitInt{{.*}}()
-// CHECK-AARCH: [[DEFINE]] i3 @{{.*}}ret_BitInt{{.*}}()
-// CHECK-INTEL: [[DEFINE]] void @{{.*}}pass_BitInt{{.*}}(i3 noundef signext %
-// CHECK-AARCH: [[DEFINE]] void @{{.*}}pass_BitInt{{.*}}(i3 %
-// CHECK-INTEL: [[DEFINE]] void @{{.*}}pass_large_BitInt{{.*}}(i64 %{{.*}}, i64 %
-// CHECK-AARCH: [[DEFINE]] void @{{.*}}pass_large_BitInt{{.*}}(i127 %
+// CHECK-INTEL: [[DEF]] noundef signext i3 @{{.*}}ret_BitInt{{.*}}()
+// CHECK-AARCH: [[DEF]] i3 @{{.*}}ret_BitInt{{.*}}()
+// CHECK-INTEL: [[DEF]] void @{{.*}}pass_BitInt{{.*}}(i3 noundef signext %
+// CHECK-AARCH: [[DEF]] void @{{.*}}pass_BitInt{{.*}}(i3 %
+// CHECK-INTEL: [[DEF]] void @{{.*}}pass_large_BitInt{{.*}}(i64 %{{.*}}, i64 %
+// CHECK-AARCH: [[DEF]] void @{{.*}}pass_large_BitInt{{.*}}(i127 %
 } // namespace check_exotic

clang/test/CodeGen/indirect-noundef.cpp

 // RUN: %clang_cc1 -no-opaque-pointers -x c++ -triple x86_64-unknown-unknown -O0 -emit-llvm -o - %s | FileCheck %s
 // RUN: %clang_cc1 -no-opaque-pointers -x c++ -triple x86_64-unknown-unknown -O0 -emit-llvm -fsanitize-memory-param-retval -o - %s | FileCheck %s
 
 // no-sanitize-memory-param-retval does NOT conflict with enable-noundef-analysis
 // RUN: %clang_cc1 -no-opaque-pointers -x c++ -triple x86_64-unknown-unknown -O0 -emit-llvm -fno-sanitize-memory-param-retval -o - %s | FileCheck %s
 
 union u1 {
   int val;
 };
 
 // CHECK: @indirect_callee_int_ptr = [[GLOBAL:(dso_local )?global]] i32 (i32)*
 int (*indirect_callee_int_ptr)(int);
 // CHECK: @indirect_callee_union_ptr = [[GLOBAL]] i32 (i32)*
 union u1 (*indirect_callee_union_ptr)(union u1);
 
-// CHECK: [[DEFINE:define( dso_local)?]] noundef i32 @{{.*}}indirect_callee_int{{.*}}(i32 noundef %
+// CHECK: [[DEF:define( dso_local)?]] noundef i32 @{{.*}}indirect_callee_int{{.*}}(i32 noundef %
 int indirect_callee_int(int a) { return a; }
-// CHECK: [[DEFINE]] i32 @{{.*}}indirect_callee_union{{.*}}(i32 %
+// CHECK: [[DEF]] i32 @{{.*}}indirect_callee_union{{.*}}(i32 %
 union u1 indirect_callee_union(union u1 a) {
   return a;
 }
 
 int main() {
   // CHECK: call noundef i32 @{{.*}}indirect_callee_int{{.*}}(i32 noundef 0)
   indirect_callee_int(0);
   // CHECK: call i32 @{{.*}}indirect_callee_union{{.*}}(i32 %

clang/test/Preprocessor/init.c

 //
 // RUN: %clang_cc1 -E -dM -ffreestanding -triple=sparc64-none-openbsd < /dev/null | FileCheck -match-full-lines -check-prefix SPARC64-OBSD %s
 // SPARC64-OBSD:#define __INT64_TYPE__ long long int
 // SPARC64-OBSD:#define __INTMAX_C_SUFFIX__ LL
 // SPARC64-OBSD:#define __INTMAX_TYPE__ long long int
 // SPARC64-OBSD:#define __UINTMAX_C_SUFFIX__ ULL
 // SPARC64-OBSD:#define __UINTMAX_TYPE__ long long unsigned int
 //
-// RUN: %clang_cc1 -E -dM -ffreestanding -triple=x86_64-pc-kfreebsd-gnu < /dev/null | FileCheck -match-full-lines -check-prefix KFREEBSD-DEFINE %s
-// KFREEBSD-DEFINE:#define __FreeBSD_kernel__ 1
-// KFREEBSD-DEFINE:#define __GLIBC__ 1
-//
-// RUN: %clang_cc1 -E -dM -ffreestanding -triple=i686-pc-kfreebsd-gnu < /dev/null | FileCheck -match-full-lines -check-prefix KFREEBSDI686-DEFINE %s
-// KFREEBSDI686-DEFINE:#define __FreeBSD_kernel__ 1
-// KFREEBSDI686-DEFINE:#define __GLIBC__ 1
+// RUN: %clang_cc1 -E -dM -ffreestanding -triple=x86_64-pc-kfreebsd-gnu < /dev/null | FileCheck -match-full-lines -check-prefix KFREEBSD-DEF %s
+// KFREEBSD-DEF:#define __FreeBSD_kernel__ 1
+// KFREEBSD-DEF:#define __GLIBC__ 1
+//
+// RUN: %clang_cc1 -E -dM -ffreestanding -triple=i686-pc-kfreebsd-gnu < /dev/null | FileCheck -match-full-lines -check-prefix KFREEBSDI686-DEF %s
+// KFREEBSDI686-DEF:#define __FreeBSD_kernel__ 1
+// KFREEBSDI686-DEF:#define __GLIBC__ 1
 //
 // RUN: %clang_cc1 -x c++ -triple i686-pc-linux-gnu -fobjc-runtime=gcc -E -dM < /dev/null | FileCheck -match-full-lines -check-prefix GNUSOURCE %s
 // RUN: %clang_cc1 -x c++ -triple sparc-rtems-elf -E -dM < /dev/null | FileCheck -match-full-lines -check-prefix GNUSOURCE %s
 // GNUSOURCE:#define _GNU_SOURCE 1
 //
 // Check that the GNUstep Objective-C ABI defines exist and are clamped at the
 // highest supported version.
 // RUN: %clang_cc1 -x objective-c -triple i386-unknown-freebsd -fobjc-runtime=gnustep-1.9 -E -dM < /dev/null | FileCheck -match-full-lines -check-prefix GNUSTEP1 %s

llvm/docs/CommandGuide/lit.rst

  %:T                     On Windows, %/T but a ``:`` is removed if its the second character.
                          Otherwise, %T but with a single leading ``/`` removed.
  ======================= ==============
 
 Other substitutions are provided that are variations on this base set and
 further substitution patterns can be defined by each test module. See the
 modules :ref:`local-configuration-files`.
 
-By default, substitutions are expanded exactly once, so that if e.g. a
-substitution ``%build`` is defined in top of another substitution ``%cxx``,
-``%build`` will expand to ``%cxx`` textually, not to what ``%cxx`` expands to.
-However, if the ``recursiveExpansionLimit`` property of the ``TestingConfig``
-is set to a non-negative integer, substitutions will be expanded recursively
-until that limit is reached. It is an error if the limit is reached and
-expanding substitutions again would yield a different result.
-
 More detailed information on substitutions can be found in the
 :doc:`../TestingGuide`.
 
 TEST RUN OUTPUT FORMAT
 ~~~~~~~~~~~~~~~~~~~~~~
 
 The :program:`lit` output for a test run conforms to the following schema, in
 both short and verbose modes (although in short mode no PASS lines will be

llvm/docs/TestingGuide.rst

 
 ``%if feature %{<if branch>%} %else %{<else branch>%}``
 
  Conditional substitution: if ``feature`` is available it expands to
  ``<if branch>``, otherwise it expands to ``<else branch>``.
  ``%else %{<else branch>%}`` is optional and treated like ``%else %{%}``
  if not present.
 
+``%(line)``, ``%(line+<number>)``, ``%(line-<number>)``
+
+  The number of the line where this substitution is used, with an
+  optional integer offset.  These expand only if they appear
+  immediately in ``RUN:``, ``DEFINE:``, and ``REDEFINE:`` directives.
+  Occurrences in substitutions defined elsewhere are never expanded.
+  For example, this can be used in tests with multiple RUN lines,
+  which reference the test file's line numbers.
+
 **LLVM-specific substitutions:**
 
 ``%shlibext``
    The suffix for the host platforms shared library files. This includes the
    period as the first character.
 
    Example: ``.so`` (Linux), ``.dylib`` (macOS), ``.dll`` (Windows)
 
 ``%exeext``
    The suffix for the host platforms executable files. This includes the
    period as the first character.
 
    Example: ``.exe`` (Windows), empty on Linux.
 
-``%(line)``, ``%(line+<number>)``, ``%(line-<number>)``
-   The number of the line where this substitution is used, with an optional
-   integer offset. This can be used in tests with multiple RUN lines, which
-   reference test file's line numbers.
-
-
 **Clang-specific substitutions:**
 
 ``%clang``
    Invokes the Clang driver.
 
 ``%clang_cpp``
    Invokes the Clang driver for C++.
 

 
 **FileCheck-specific substitutions:**
 
 ``%ProtectFileCheckOutput``
    This should precede a ``FileCheck`` call if and only if the call's textual
    output affects test results.  It's usually easy to tell: just look for
    redirection or piping of the ``FileCheck`` call's stdout or stderr.
 
-To add more substitutions, look at ``test/lit.cfg`` or ``lit.local.cfg``.
+.. _Test-specific substitutions:
+
+**Test-specific substitutions:**
+
+Additional substitutions can be defined as follows:
+
+- Lit configuration files (e.g., ``lit.cfg`` or ``lit.local.cfg``) can define
+  substitutions for all tests in a test directory.  They do so by extending the
+  substitution list, ``config.substitutions``.  Each item in the list is a tuple
+  consisting of a pattern and its replacement, which lit applies using python's
+  ``re.sub`` function.
+- To define substitutions within a single test file, lit supports the
+  ``DEFINE:`` and ``REDEFINE:`` directives, described in detail below.  So that
+  they have no effect on other test files, these directives modify a copy of the
+  substitution list that is produced by lit configuration files.
+
+For example, the following directives can be inserted into a test file to define
+``%{cflags}`` and ``%{fcflags}`` substitutions with empty initial values, which
+serve as the parameters of another newly defined ``%{check}`` substitution:
+
+.. code-block:: llvm
+
+    ; DEFINE: %{cflags} =
+    ; DEFINE: %{fcflags} =
+
+    ; DEFINE: %{check} =                                                  \
+    ; DEFINE:   %clang_cc1 -verify -fopenmp -fopenmp-version=51 %{cflags} \
+    ; DEFINE:              -emit-llvm -o - %s |                           \
+    ; DEFINE:     FileCheck %{fcflags} %s
+
+Alternatively, the above substitutions can be defined in a lit configuration
+file to be shared with other test files.  Either way, the test file can then
+specify directives like the following to redefine the parameter substitutions as
+desired before each use of ``%{check}`` in a ``RUN:`` line:
+
+.. code-block:: llvm
+
+    ; REDEFINE: %{cflags} = -triple x86_64-apple-darwin10.6.0 -fopenmp-simd
+    ; REDEFINE: %{fcflags} = -check-prefix=SIMD
+    ; RUN: %{check}
+
+    ; REDEFINE: %{cflags} = -triple x86_64-unknown-linux-gnu -fopenmp-simd
+    ; REDEFINE: %{fcflags} = -check-prefix=SIMD
+    ; RUN: %{check}
+
+    ; REDEFINE: %{cflags} = -triple x86_64-apple-darwin10.6.0
+    ; REDEFINE: %{fcflags} = -check-prefix=NO-SIMD
+    ; RUN: %{check}
+
+    ; REDEFINE: %{cflags} = -triple x86_64-unknown-linux-gnu
+    ; REDEFINE: %{fcflags} = -check-prefix=NO-SIMD
+    ; RUN: %{check}
+
+Besides providing initial values, the initial ``DEFINE:`` directives for the
+parameter substitutions in the above example serve a second purpose: they
+establish the substitution order so that both ``%{check}`` and its parameters
+expand as desired.  There's a simple way to remember the required definition
+order in a test file: define a substitution before any substitution that might
+refer to it.
+
+In general, substitution expansion behaves as follows:
+
+- Upon arriving at each ``RUN:`` line, lit expands all substitutions in that
+  ``RUN:`` line using their current values from the substitution list.  No
+  substitution expansion is performed immediately at ``DEFINE:`` and
+  ``REDEFINE:`` directives except ``%(line)``, ``%(line+<number>)``, and
+  ``%(line-<number>)``.
+- When expanding substitutions in a ``RUN:`` line, lit makes only one pass
+  through the substitution list by default.  In this case, a substitution must
+  have been inserted earlier in the substitution list than any substitution
+  appearing in its value in order for the latter to expand.  (For greater
+  flexibility, you can enable multiple passes through the substitution list by
+  setting `recursiveExpansionLimit`_ in a lit configuration file.)
+- While lit configuration files can insert anywhere in the substitution list,
+  the insertion behavior of the ``DEFINE:`` and ``REDEFINE:`` directives is
+  specified below and is designed specifically for the use case presented in the
+  example above.
+- Defining a substitution in terms of itself, whether directly or via other
+  substitutions, should be avoided.  It usually produces an infinitely recursive
+  definition that cannot be fully expanded.  It does *not* define the
+  substitution in terms of its previous value, even when using ``REDEFINE:``.
+
+The relationship between the ``DEFINE:`` and ``REDEFINE:`` directive is
+analogous to the relationship between a variable declaration and variable
+assignment in many programming languages:
+
+- ``DEFINE: %{name} = value``
+
+   This directive assigns the specified value to a new substitution whose
+   pattern is ``%{name}``, or it reports an error if there is already a
+   substitution whose pattern contains ``%{name}`` because that could produce
+   confusing expansions (e.g., a lit configuration file might define a
+   substitution with the pattern ``%{name}\[0\]``).  The new substitution is
+   inserted at the start of the substitution list so that it will expand first.
+   Thus, its value can contain any substitution previously defined, whether in
+   the same test file or in a lit configuration file, and both will expand.
+
+- ``REDEFINE: %{name} = value``
+
+   This directive assigns the specified value to an existing substitution whose
+   pattern is ``%{name}``, or it reports an error if there are no substitutions
+   with that pattern or if there are multiple substitutions whose patterns
+   contain ``%{name}``.  The substitution's current position in the substitution
+   list does not change so that expansion order relative to other existing
+   substitutions is preserved.
+
+The following properties apply to both the ``DEFINE:`` and ``REDEFINE:``
+directives:
+
+- **Substitution name**: In the directive, whitespace immediately before or
+  after ``%{name}`` is optional and discarded.  ``%{name}`` must start with
+  ``%{``, it must end with ``}``, and the rest must start with a letter or
+  underscore and contain only alphanumeric characters, hyphens, underscores, and
+  colons.  This syntax has a few advantages:
+
+    - It is impossible for ``%{name}`` to contain sequences that are special in
+      python's ``re.sub`` patterns.  Otherwise, attempting to specify
+      ``%{name}`` as a substitution pattern in a lit configuration file could
+      produce confusing expansions.
+    - The braces help avoid the possibility that another substitution's pattern
+      will match part of ``%{name}`` or vice-versa, producing confusing
+      expansions.  However, the patterns of substitutions defined by lit
+      configuration files and by lit itself are not restricted to this form, so
+      overlaps are still theoretically possible.
+
+- **Substitution value**: The value includes all text from the first
+  non-whitespace character after ``=`` to the last non-whitespace character.  If
+  there is no non-whitespace character after ``=``, the value is the empty
+  string.  Escape sequences that can appear in python ``re.sub`` replacement
+  strings are treated as plain text in the value.
+- **Line continuations**: If the last non-whitespace character on the line after
+  ``:`` is ``\``, then the next directive must use the same directive keyword
+  (e.g., ``DEFINE:``) , and it is an error if there is no additional directive.
+  That directive serves as a continuation.  That is, before following the rules
+  above to parse the text after ``:`` in either directive, lit joins that text
+  together to form a single directive, replaces the ``\`` with a single space,
+  and removes any other whitespace that is now adjacent to that space.  A
+  continuation can be continued in the same manner.  A continuation containing
+  only whitespace after its ``:`` is an error.
+
+.. _recursiveExpansionLimit:
+
+**recursiveExpansionLimit:**
+
+As described in the previous section, when expanding substitutions in a ``RUN:``
+line, lit makes only one pass through the substitution list by default.  Thus,
+if substitutions are not defined in the proper order, some will remain in the
+``RUN:`` line unexpanded.  For example, the following directives refer to
+``%{inner}`` within ``%{outer}`` but do not define ``%{inner}`` until after
+``%{outer}``:
+
+.. code-block:: llvm
+
+    ; By default, this definition order does not enable full expansion.
+
+    ; DEFINE: %{outer} = %{inner}
+    ; DEFINE: %{inner} = expanded
+
+    ; RUN: echo '%{outer}'
+
+``DEFINE:`` inserts substitutions at the start of the substitution list, so
+``%{inner}`` expands first but has no effect because the original ``RUN:`` line
+does not contain ``%{inner}``.  Next, ``%{outer}`` expands, and the output of
+the ``echo`` command becomes:
+
+.. code-block:: shell
+
+    %{inner}
+
+Of course, one way to fix this simple case is to reverse the definitions of
+``%{outer}`` and ``%{inner}``.  However, if a test has a complex set of
+substitutions that can all reference each other, there might not exist a
+sufficient substitution order.
+
+To address such use cases, lit configuration files support
+``config.recursiveExpansionLimit``, which can be set to a non-negative integer
+to specify the maximum number of passes through the substitution list.  Thus, in
+the above example, setting the limit to 2 would cause lit to make a second pass
+that expands ``%{inner}`` in the ``RUN:`` line, and the output from the ``echo``
+command when then be:
+
+.. code-block:: shell
+
+    expanded
+
+To improve performance, lit will stop making passes when it notices the ``RUN:``
+line has stopped changing.  In the above example, setting the limit higher than
+2 is thus harmless.
 
+To facilitate debugging, after reaching the limit, lit will make one extra pass
+and report an error if the ``RUN:`` line changes again.  In the above example,
+setting the limit to 1 will thus cause lit to report an error instead of
+producing incorrect output.
 
 Options
 -------
 
 The llvm lit configuration allows to customize some things with user options:
 
 ``llc``, ``opt``, ...
     Substitute the respective llvm tool name with a custom command line. This

 --------------
 
 To make RUN line writing easier, there are several helper programs. These
 helpers are in the PATH when running tests, so you can just call them using
 their name. For example:
 
 ``not``
    This program runs its arguments and then inverts the result code from it.
    Zero result codes become 1. Non-zero result codes become 0.
 
 To make the output more useful, :program:`lit` will scan
 the lines of the test case for ones that contain a pattern that matches
 ``PR[0-9]+``. This is the syntax for specifying a PR (Problem Report) number
 that is related to the test case. The number after "PR" specifies the
 LLVM Bugzilla number. When a PR number is specified, it will be used in
 the pass/fail reporting. This is useful to quickly get some context when
 a test fails.
 
 Finally, any line that contains "END." will cause the special
 interpretation of lines to terminate. This is generally done right after
 the last RUN: line. This has two side effects:
 
 (a) it prevents special interpretation of lines that are part of the test
     program, not the instructions to the test case, and
 
 (b) it speeds things up for really big test cases by avoiding
     interpretation of the remainder of the file.

llvm/test/tools/llvm-cvtres/help.test

 ; RUN: llvm-cvtres /h > %t
 ; RUN: FileCheck -input-file=%t %s -check-prefix=HELP_TEST
 
 ; HELP_TEST:    OVERVIEW: Resource Converter
 ; HELP_TEST-DAG:  USAGE: llvm-cvtres [options] file...
 ; HELP_TEST-DAG:  OPTIONS:
-; HELP_TEST-NEXT:   /DEFINE:symbol Not implemented
+; HELP_TEST-NEXT:   /{{DEFINE}}:symbol Not implemented
 ; HELP_TEST-NEXT:   /FOLDDUPS: Not implemented
 ; HELP_TEST-NEXT:   /HELP Display available options
 ; HELP_TEST-NEXT:   /MACHINE:{ARM|ARM64|EBC|IA64|X64|X86}
 ; HELP_TEST-NEXT:   Machine architecture
 ; HELP_TEST-DAG:    /NOLOGO Not implemented
 ; HELP_TEST-NEXT:   /OUT:filename Output file
 ; HELP_TEST-NEXT:   /READONLY Not implemented
 ; HELP_TEST-NEXT:   /TIMESTAMP:<value> Timestamp for coff header, defaults to current time
 ; HELP_TEST-NEXT:   /VERBOSE Use verbose output

llvm/test/tools/yaml2obj/ELF/custom-null-section.yaml

- Type:         SHT_NULL
     EntSize:      0x0
     Link:         0
     Info:         0
     Address:      0x0
 
 ## Check we can redefine fields of the first SHT_NULL section.
 
 # RUN: yaml2obj --docnum=3 %s -o %t3
-# RUN: llvm-readelf --sections %t3 | FileCheck %s --check-prefix=REDEFINE
+# RUN: llvm-readelf --sections %t3 | FileCheck %s --check-prefix=REDEF
 
-# REDEFINE:      Section Headers:
-# REDEFINE-NEXT:  [Nr] Name Type Address          Off    Size   ES Flg Lk Inf Al
-# REDEFINE-NEXT:  [ 0] .foo NULL 0000000000000006 000000 000002 03   A 4   5  1
+# REDEF:      Section Headers:
+# REDEF-NEXT:  [Nr] Name Type Address          Off    Size   ES Flg Lk Inf Al
+# REDEF-NEXT:  [ 0] .foo NULL 0000000000000006 000000 000002 03   A 4   5  1
 
 --- !ELF
 FileHeader:
   Class: ELFCLASS64
   Data:  ELFDATA2LSB
   Type:  ET_REL
 Sections:
   - Type:         SHT_NULL

llvm/utils/lit/lit/TestRunner.py

def memoized(x):
             cache[x] = f(x)
         return cache[x]
     return memoized
 
 @_memoize
 def _caching_re_compile(r):
     return re.compile(r)
 
+class ExpandableScriptDirective(object):
+    """
+    Common interface for lit directives for which any lit substitutions must be
+    expanded to produce the shell script.  It includes directives (e.g., 'RUN:')
+    specifying shell commands that might have lit substitutions to be expanded.
+    It also includes lit directives (e.g., 'DEFINE:') that adjust substitutions.
+
+    start_line_number: The directive's starting line number.
+    end_line_number: The directive's ending line number, which is
+        start_line_number if the directive has no line continuations.
+    keyword: The keyword that specifies the directive.  For example, 'RUN:'.
+    """
+
+    def __init__(self, start_line_number, end_line_number, keyword):
+        # Input line number where the directive starts.
+        self.start_line_number = start_line_number
+        # Input line number where the directive ends.
+        self.end_line_number = end_line_number
+        # The keyword used to indicate the directive.
+        self.keyword = keyword
+
+    def add_continuation(self, line_number, keyword, line):
+        """
+        Add a continuation line to this directive and return True, or do nothing
+        and return False if the specified line is not a continuation for this
+        directive (e.g., previous line does not end in '\', or keywords do not
+        match).
+
+        line_number: The line number for the continuation line.
+        keyword: The keyword that specifies the continuation line.  For example,
+            'RUN:'.
+        line: The content of the continuation line after the keyword.
+        """
+        assert False, "expected method to be called on derived class"
+
+    def needs_continuation(self):
+        """
+        Does this directive require a continuation line?
+
+        '\' is documented as indicating a line continuation even if whitespace
+        separates it from the newline.  It looks like a line continuation, and
+        it would be confusing if it didn't behave as one.
+        """
+        assert False, "expected method to be called on derived class"
+
+    def get_location(self):
+        """
+        Get a phrase describing the line or range of lines so far included by
+        this directive and any line continuations.
+        """
+        if self.start_line_number == self.end_line_number:
+            return f'at line {self.start_line_number}'
+        return f'from line {self.start_line_number} to {self.end_line_number}'
+
+class CommandDirective(ExpandableScriptDirective):
+    """
+    A lit directive taking a shell command line.  For example,
+    'RUN: echo hello world'.
+
+    command: The content accumulated so far from the directive and its
+        continuation lines.
+    """
+
+    def __init__(self, start_line_number, end_line_number, keyword, line):
+        super().__init__(start_line_number, end_line_number, keyword)
+        self.command = line.rstrip()
+
+    def add_continuation(self, line_number, keyword, line):
+        if keyword != self.keyword or not self.needs_continuation():
+            return False
+        self.command = self.command[:-1] + line.rstrip()
+        self.end_line_number = line_number
+        return True
+
+    def needs_continuation(self):
+        # Trailing whitespace is stripped immediately when each line is added,
+        # so '\' is never hidden here.
+        return self.command[-1] == '\\'
+
+class SubstDirective(ExpandableScriptDirective):
+    """
+    A lit directive taking a substitution definition or redefinition.  For
+    example, 'DEFINE: %{name} = value'.
+
+    new_subst: True if this directive defines a new substitution.  False if it
+        redefines an existing substitution.
+    body: The unparsed content accumulated so far from the directive and its
+        continuation lines.
+    name: The substitution's name, or None if more continuation lines are still
+        required.
+    value: The substitution's value, or None if more continuation lines are
+        still required.
+    """
+
+    def __init__(self, start_line_number, end_line_number, keyword, new_subst,
+                 line):
+        super().__init__(start_line_number, end_line_number, keyword)
+        self.new_subst = new_subst
+        self.body = line
+        self.name = None
+        self.value = None
+        self._parse_body()
+
+    def add_continuation(self, line_number, keyword, line):
+        if keyword != self.keyword or not self.needs_continuation():
+            return False
+        if not line.strip():
+            raise ValueError("Substitution's continuation is empty")
+        # Append line.  Replace the '\' and any adjacent whitespace with a
+        # single space.
+        self.body = self.body.rstrip()[:-1].rstrip() + ' ' + line.lstrip()
+        self.end_line_number = line_number
+        self._parse_body()
+        return True
+
+    def needs_continuation(self):
+        return self.body.rstrip()[-1:] == '\\'
+
+    def _parse_body(self):
+        """
+        If no more line continuations are required, parse all the directive's
+        accumulated lines in order to identify the substitution's name and full
+        value, and raise an exception if invalid.
+        """
+        if self.needs_continuation():
+            return
+
+        # Extract the left-hand side and value, and discard any whitespace
+        # enclosing each.
+        parts = self.body.split('=', 1)
+        if len(parts) == 1:
+            raise ValueError("Substitution's definition does not contain '='")
+        self.name = parts[0].strip()
+        self.value = parts[1].strip()
+
+        # Check the substitution's name.
+        #
+        # Do not extend this to permit '.' or any sequence that's special in a
+        # python pattern.  We could escape that automatically for
+        # DEFINE/REDEFINE directives in test files.  However, lit configuration
+        # file authors would still have to remember to escape them manually in
+        # substitution names but not in values.  Moreover, the manually chosen
+        # and automatically chosen escape sequences would have to be consistent
+        # (e.g., '\.' vs. '[.]') in order for REDEFINE to successfully redefine
+        # a substitution previously defined by a lit configuration file.  All
+        # this seems too error prone and confusing to be worthwhile.  If you
+        # want your name to express structure, use ':' instead of '.'.
+        #
+        # Actually, '{' and '}' are special if they contain only digits possibly
+        # separated by a comma.  Requiring a leading letter avoids that.
+        if not re.fullmatch(r'%{[_a-zA-Z][-_:0-9a-zA-Z]*}', self.name):
+            raise ValueError(
+                f"Substitution name '{self.name}' is malformed as it must "
+                f"start with '%{{', it must end with '}}', and the rest must "
+                f"start with a letter or underscore and contain only "
+                f"alphanumeric characters, hyphens, underscores, and colons")
+
+    def adjust_substitutions(self, substitutions):
+        """
+        Modify the specified substitution list as specified by this directive.
+        """
+        assert not self.needs_continuation(), \
+               "expected directive continuations to be parsed before applying"
+        value_repl = self.value.replace('\\', '\\\\')
+        existing = [i for i, subst in enumerate(substitutions)
+                    if self.name in subst[0]]
+        existing_res = ''.join("\nExisting pattern: " + substitutions[i][0]
+                               for i in existing)
+        if self.new_subst:
+            if existing:
+                raise ValueError(
+                    f"Substitution whose pattern contains '{self.name}' is "
+                    f"already defined before '{self.keyword}' directive "
+                    f"{self.get_location()}"
+                    f"{existing_res}")
+            substitutions.insert(0, (self.name, value_repl))
+            return
+        if len(existing) > 1:
+            raise ValueError(
+                f"Multiple substitutions whose patterns contain '{self.name}' "
+                f"are defined before '{self.keyword}' directive "
+                f"{self.get_location()}"
+                f"{existing_res}")
+        if not existing:
+            raise ValueError(
+                f"No substitution for '{self.name}' is defined before "
+                f"'{self.keyword}' directive {self.get_location()}")
+        if substitutions[existing[0]][0] != self.name:
+            raise ValueError(
+                f"Existing substitution whose pattern contains '{self.name}' "
+                f"does not have the pattern specified by '{self.keyword}' "
+                f"directive {self.get_location()}\n"
+                f"Expected pattern: {self.name}"
+                f"{existing_res}")
+        substitutions[existing[0]] = (self.name, value_repl)
+
+
 def applySubstitutions(script, substitutions, conditions={},
                        recursion_limit=None):
     """
     Apply substitutions to the script.  Allow full regular expression syntax.
     Replace each matching occurrence of regular expression pattern a with
     substitution b in line ln.
 
     If a substitution expands into another substitution, it is expanded
def processLineToFixedPoint(ln):
         if processed != ln:
             raise ValueError("Recursive substitution of '%s' did not complete "
                              "in the provided recursion limit (%s)" % \
                              (origLine, recursion_limit))
 
         return processed
 
     process = processLine if recursion_limit is None else processLineToFixedPoint
-    
-    return [unescapePercents(process(ln)) for ln in script]
+    output = []
+    for directive in script:
+        if isinstance(directive, SubstDirective):
+            directive.adjust_substitutions(substitutions)
+        else:
+            if isinstance(directive, CommandDirective):
+                line = directive.command
+            else:
+                # Can come from preamble_commands.
+                assert isinstance(directive, str)
+                line = directive
+            output.append(unescapePercents(process(line)))
+    return output
 
 
 class ParserKind(object):
     """
     An enumeration representing the style of an integrated test keyword or
     command.
 
     TAG: A keyword taking no value. Ex 'END.'
     COMMAND: A keyword taking a list of shell commands. Ex 'RUN:'
     LIST: A keyword taking a comma-separated list of values.
     BOOLEAN_EXPR: A keyword taking a comma-separated list of
         boolean expressions. Ex 'XFAIL:'
     INTEGER: A keyword taking a single integer. Ex 'ALLOW_RETRIES:'
     CUSTOM: A keyword with custom parsing semantics.
+    DEFINE: A keyword taking a new lit substitution definition. Ex
+        'DEFINE: %{name}=value'
+    REDEFINE: A keyword taking a lit substitution redefinition. Ex
+        'REDEFINE: %{name}=value'
     """
     TAG = 0
     COMMAND = 1
     LIST = 2
     BOOLEAN_EXPR = 3
     INTEGER = 4
     CUSTOM = 5
+    DEFINE = 6
+    REDEFINE = 7
 
     @staticmethod
     def allowedKeywordSuffixes(value):
         return { ParserKind.TAG:          ['.'],
                  ParserKind.COMMAND:      [':'],
                  ParserKind.LIST:         [':'],
                  ParserKind.BOOLEAN_EXPR: [':'],
                  ParserKind.INTEGER:      [':'],
-                 ParserKind.CUSTOM:       [':', '.']
+                 ParserKind.CUSTOM:       [':', '.'],
+                 ParserKind.DEFINE:       [':'],
+                 ParserKind.REDEFINE:     [':']
                } [value]
 
     @staticmethod
     def str(value):
         return { ParserKind.TAG:          'TAG',
                  ParserKind.COMMAND:      'COMMAND',
                  ParserKind.LIST:         'LIST',
                  ParserKind.BOOLEAN_EXPR: 'BOOLEAN_EXPR',
                  ParserKind.INTEGER:      'INTEGER',
-                 ParserKind.CUSTOM:       'CUSTOM'
+                 ParserKind.CUSTOM:       'CUSTOM',
+                 ParserKind.DEFINE:       'DEFINE',
+                 ParserKind.REDEFINE:     'REDEFINE'
                } [value]
 
 
 class IntegratedTestKeywordParser(object):
     """A parser for LLVM/Clang style integrated test scripts.
 
     keyword: The keyword to parse for. It must end in either '.' or ':'.
     kind: An value of ParserKind.
def __init__(self, keyword, kind, parser=None, initial_value=None):
         elif kind == ParserKind.INTEGER:
             self.parser = self._handleSingleInteger
         elif kind == ParserKind.TAG:
             self.parser = self._handleTag
         elif kind == ParserKind.CUSTOM:
             if parser is None:
                 raise ValueError("ParserKind.CUSTOM requires a custom parser")
             self.parser = parser
+        elif kind == ParserKind.DEFINE:
+            self.parser = lambda line_number, line, output: \
+                                 self._handleSubst(line_number, line, output,
+                                                   self.keyword, new_subst=True)
+        elif kind == ParserKind.REDEFINE:
+            self.parser = lambda line_number, line, output: \
+                                 self._handleSubst(line_number, line, output,
+                                                   self.keyword,
+                                                   new_subst=False)
         else:
             raise ValueError("Unknown kind '%s'" % kind)
 
     def parseLine(self, line_number, line):
         try:
             self.parsed_lines += [(line_number, line)]
             self.value = self.parser(line_number, line, self.value)
         except ValueError as e:
             raise ValueError(str(e) + ("\nin %s directive on test line %d" %
                                        (self.keyword, line_number)))
 
     def getValue(self):
         return self.value
 
     @staticmethod
     def _handleTag(line_number, line, output):
         """A helper for parsing TAG type keywords"""
         return (not line.strip() or output)
 
     @staticmethod
-    def _handleCommand(line_number, line, output, keyword):
-        """A helper for parsing COMMAND type keywords"""
-        # Trim trailing whitespace.
-        line = line.rstrip()
-        # Substitute line number expressions
+    def _substituteLineNumbers(line_number, line):
         line = re.sub(r'%\(line\)', str(line_number), line)
-
         def replace_line_number(match):
             if match.group(1) == '+':
                 return str(line_number + int(match.group(2)))
             if match.group(1) == '-':
                 return str(line_number - int(match.group(2)))
-        line = re.sub(r'%\(line *([\+-]) *(\d+)\)', replace_line_number, line)
-        # Collapse lines with trailing '\\'.
-        if output and output[-1][-1] == '\\':
-            output[-1] = output[-1][:-1] + line
-        else:
+        return re.sub(r'%\(line *([\+-]) *(\d+)\)', replace_line_number, line)
+
+    @classmethod
+    def _handleCommand(cls, line_number, line, output, keyword):
+        """A helper for parsing COMMAND type keywords"""
+        # Substitute line number expressions.
+        line = cls._substituteLineNumbers(line_number, line)
+
+        # Collapse lines with trailing '\\', or add line with line number to
+        # start a new pipeline.
+        if not output or not output[-1].add_continuation(line_number, keyword,
+                                                         line):
             if output is None:
                 output = []
             pdbg = "%dbg({keyword} at line {line_number})".format(
                 keyword=keyword,
                 line_number=line_number)
             assert re.match(kPdbgRegex + "$", pdbg), \
                    "kPdbgRegex expected to match actual %dbg usage"
             line = "{pdbg} {real_command}".format(
                 pdbg=pdbg,
                 real_command=line)
-            output.append(line)
+            output.append(CommandDirective(line_number, line_number, keyword,
+                                           line))
         return output
 
     @staticmethod
     def _handleList(line_number, line, output):
         """A parser for LIST type keywords"""
         if output is None:
             output = []
         output.extend([s.strip() for s in line.split(',')])
def _handleBooleanExpr(line_number, line, output):
         output.extend(parts)
         # Evaluate each expression to verify syntax.
         # We don't want any results, just the raised ValueError.
         for s in output:
             if s != '*' and not s.endswith('\\'):
                 BooleanExpression.evaluate(s, [])
         return output
 
+    @classmethod
+    def _handleSubst(cls, line_number, line, output, keyword, new_subst):
+        """A parser for DEFINE and REDEFINE type keywords"""
+        line = cls._substituteLineNumbers(line_number, line)
+        if output and output[-1].add_continuation(line_number, keyword, line):
+            return output
+        if output is None:
+            output = []
+        output.append(SubstDirective(line_number, line_number, keyword,
+                                     new_subst, line))
+        return output
+
 
 def _parseKeywords(sourcepath, additional_parsers=[],
                    require_script=True):
     """_parseKeywords
 
     Scan an LLVM/Clang style integrated test script and extract all the lines
     pertaining to a special parser. This includes 'RUN', 'XFAIL', 'REQUIRES',
-    'UNSUPPORTED' and 'ALLOW_RETRIES', as well as other specified custom
-    parsers.
+    'UNSUPPORTED', 'ALLOW_RETRIES', 'END', 'DEFINE', 'REDEFINE', as well as
+    other specified custom parsers.
 
     Returns a dictionary mapping each custom parser to its value after
     parsing the test.
     """
     # Install the built-in keyword parsers.
     script = []
     builtin_parsers = [
         IntegratedTestKeywordParser('RUN:', ParserKind.COMMAND, initial_value=script),
         IntegratedTestKeywordParser('XFAIL:', ParserKind.BOOLEAN_EXPR),
         IntegratedTestKeywordParser('REQUIRES:', ParserKind.BOOLEAN_EXPR),
         IntegratedTestKeywordParser('UNSUPPORTED:', ParserKind.BOOLEAN_EXPR),
         IntegratedTestKeywordParser('ALLOW_RETRIES:', ParserKind.INTEGER),
-        IntegratedTestKeywordParser('END.', ParserKind.TAG)
+        IntegratedTestKeywordParser('END.', ParserKind.TAG),
+        IntegratedTestKeywordParser('DEFINE:', ParserKind.DEFINE,
+                                    initial_value=script),
+        IntegratedTestKeywordParser('REDEFINE:', ParserKind.REDEFINE,
+                                    initial_value=script)
     ]
     keyword_parsers = {p.keyword: p for p in builtin_parsers}
 
     # Install user-defined additional parsers.
     for parser in additional_parsers:
         if not isinstance(parser, IntegratedTestKeywordParser):
             raise ValueError('Additional parser must be an instance of '
                              'IntegratedTestKeywordParser')
         if parser.keyword in keyword_parsers:
             raise ValueError("Parser for keyword '%s' already exists"
                              % parser.keyword)
         keyword_parsers[parser.keyword] = parser
 
     # Collect the test lines from the script.
     for line_number, command_type, ln in \
             parseIntegratedTestScriptCommands(sourcepath,
                                               keyword_parsers.keys()):
         parser = keyword_parsers[command_type]
         parser.parseLine(line_number, ln)
         if command_type == 'END.' and parser.getValue() is True:
             break
 
     # Verify the script contains a run line.
-    if require_script and not script:
+    if require_script and not any(isinstance(directive, CommandDirective)
+                                  for directive in script):
         raise ValueError("Test has no 'RUN:' line")
 
-    # Check for unterminated run lines.
-    if script and script[-1][-1] == '\\':
-        raise ValueError("Test has unterminated 'RUN:' lines (with '\\')")
+    # Check for unterminated run or subst lines.
+    #
+    # If, after a line continuation for one kind of directive (e.g., 'RUN:',
+    # 'DEFINE:', 'REDEFINE:') in script, the next directive in script is a
+    # different kind, then the '\\' remains on the former, and we report it
+    # here.
+    for directive in script:
+        if directive.needs_continuation():
+            raise ValueError(f"Test has unterminated '{directive.keyword}' "
+                             f"directive (with '\\') "
+                             f"{directive.get_location()}")
 
     # Check boolean expressions for unterminated lines.
     for key in keyword_parsers:
         kp = keyword_parsers[key]
         if kp.kind != ParserKind.BOOLEAN_EXPR:
             continue
         value = kp.getValue()
         if value and value[-1][-1] == '\\':
def parseIntegratedTestScript(test, additional_parsers=[],
     """
     # Parse the test sources and extract test properties
     try:
         parsed = _parseKeywords(test.getSourcePath(), additional_parsers,
                                 require_script)
     except ValueError as e:
         return lit.Test.Result(Test.UNRESOLVED, str(e))
     script = parsed['RUN:'] or []
+    assert parsed['DEFINE:'] == script
+    assert parsed['REDEFINE:'] == script
     test.xfails += parsed['XFAIL:'] or []
     test.requires += parsed['REQUIRES:'] or []
     test.unsupported += parsed['UNSUPPORTED:'] or []
     if parsed['ALLOW_RETRIES:']:
         test.allowed_retries = parsed['ALLOW_RETRIES:'][0]
 
     # Enforce REQUIRES:
     missing_required_features = test.getMissingRequiredFeatures()

llvm/utils/lit/tests/Inputs/shtest-define/errors/assignment/before-name.txt

+# DEFINE: foo %{name}=value
+# RUN: echo %{name}
+
+#      CHECK: Substitution name 'foo %{name}' is malformed {{.*}}
+# CHECK-NEXT: in {{DEFINE}}: directive on test line [[#@LINE-4]]
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/assignment/between-name-equals.txt

+# ':' is treated like part of the name, so the name is bad.
+
+# DEFINE: %{name}:=value
+# RUN: echo %{name}
+
+#      CHECK: Substitution name '%{name}:' is malformed {{.*}}
+# CHECK-NEXT: in {{DEFINE}}: directive on test line [[#@LINE-4]]
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/assignment/braces-empty.txt

+# DEFINE: %{} = value
+# RUN: echo %{}
+
+#      CHECK: Substitution name '%{}' is malformed {{.*}}
+# CHECK-NEXT: in {{DEFINE}}: directive on test line [[#@LINE-4]]
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/assignment/braces-with-dot.txt

+# In python's re.sub patterns, '.' means any character.  If we permitted that
+# in a substitution name (and escaped it before recording it as a substitution),
+# it would be confusing if someone tried to use the same name in a lit
+# configuration file, where the pattern isn't escaped automatically.
+
+# DEFINE: %{foo.bar} = value
+# RUN: echo %{foo.bar}
+
+#      CHECK: Substitution name '%{foo.bar}' is malformed {{.*}}
+# CHECK-NEXT: in {{DEFINE}}: directive on test line [[#@LINE-4]]
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/assignment/braces-with-equals.txt

+# The first equals counts as the equals.
+
+# DEFINE: %{foo=bar} = value
+# RUN: echo %{foo=bar}
+
+#      CHECK: Substitution name '%{foo' is malformed {{.*}}
+# CHECK-NEXT: in {{DEFINE}}: directive on test line [[#@LINE-4]]
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/assignment/braces-with-newline.txt

+# DEFINE:%{foo\
+# DEFINE:bar} = value
+# RUN: echo %{foo bar}
+
+#      CHECK: Substitution name '%{foo bar}' is malformed {{.*}}
+# CHECK-NEXT: in {{DEFINE}}: directive on test line [[#@LINE-4]]
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/assignment/braces-with-number.txt

+# In python's re.sub patterns, '%{3}' means three '%%%'.  If we permitted that
+# as a substitution name (and escaped it before recording it as a
+# substitution), it would be confusing if someone tried to use the same name in
+# a lit configuration file, where the pattern isn't escaped automatically.
+
+# DEFINE: %{3} = value
+# RUN: echo %{3}
+
+#      CHECK: Substitution name '%{3}' is malformed {{.*}}
+# CHECK-NEXT: in {{DEFINE}}: directive on test line [[#@LINE-4]]
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/assignment/braces-with-ws.txt

+# DEFINE: %{foo bar} = value
+# RUN: echo %{foo bar}
+
+#      CHECK: Substitution name '%{foo bar}' is malformed {{.*}}
+# CHECK-NEXT: in {{DEFINE}}: directive on test line [[#@LINE-4]]
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/assignment/empty.txt

+# DEFINE:
+# RUN: echo Hello World
+
+#      CHECK: Substitution's definition does not contain '='
+# CHECK-NEXT: in {{DEFINE}}: directive on test line [[#@LINE-4]]
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/assignment/no-equals.txt

+# DEFINE: %{name}: value
+# RUN: echo %{name}
+
+#      CHECK: Substitution's definition does not contain '='
+# CHECK-NEXT: in {{DEFINE}}: directive on test line [[#@LINE-4]]
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/assignment/no-name.txt

+# DEFINE:=value
+# RUN: echo Hello World
+
+#      CHECK: Substitution name '' is malformed {{.*}}
+# CHECK-NEXT: in {{DEFINE}}: directive on test line [[#@LINE-4]]
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/assignment/ws-only.txt

+#        v~ whitespace intentional
+# DEFINE:  
+# RUN: echo Hello World
+
+#      CHECK: Substitution's definition does not contain '='
+# CHECK-NEXT: in {{DEFINE}}: directive on test line [[#@LINE-4]]
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/continuation/empty.txt

+# Surely an empty substitution continuation is an accident, so it's an error.
+
+# DEFINE: %{foo} = foo \
+# DEFINE:
+# RUN: echo '%{foo}'
+
+#      CHECK: Substitution's continuation is empty
+# CHECK-NEXT: in DEFINE: directive on test line [[#@LINE-4]]
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/continuation/end-in-double-backslash.txt

+# This seems like a way to end a value with a backslash, but it requires an
+# empty substitution continuation, which isn't permitted.
+
+# DEFINE: %{foo} = \\
+# DEFINE:
+# RUN: echo '%{foo}'
+
+#      CHECK: Substitution's continuation is empty
+# CHECK-NEXT: in {{DEFINE}}: directive on test line [[#@LINE-4]]
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/continuation/unterminated-define-bad-redefine.txt

+# You cannot continue a DEFINE with a REDEFINE.  In this case, the REDEFINE
+# is wrong on its own because it's written as a continuation, so the bad
+# REDEFINE is reported.
+
+# DEFINE: %{foo}=echo \
+# REDEFINE: Hello World
+# DEFINE: too late to continue
+# RUN: %{foo}
+
+#      CHECK: Substitution's definition does not contain '='
+# CHECK-NEXT: in {{REDEFINE}}: directive on test line [[#@LINE-5]]
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/continuation/unterminated-define-continuation.txt

+# Check a define's continuation line that is unterminated.
+
+# RUN: echo "Don't complain about no RUN lines."
+# DEFINE: %{foo} = foo \
+# DEFINE:          bar \
+
+# CHECK: Test has unterminated '{{DEFINE}}:' directive (with '\') from line [[#@LINE-3]] to [[#@LINE-2]]
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/continuation/unterminated-define-redefine.txt

+# You cannot continue a DEFINE with a REDEFINE.  In this case, the REDEFINE
+# looks fine on its own, so the unterminated DEFINE is reported.
+
+# DEFINE: %{name}=x
+# DEFINE: %{value}=3
+# DEFINE: %{deceptive-continue}=echo \
+# REDEFINE: %{name}=%{value}
+# DEFINE: %{too-late-to-continue}=
+# RUN: %{deceptive-continue}
+
+# CHECK: Test has unterminated '{{DEFINE}}:' directive (with '\') at line [[#@LINE-5]]
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/continuation/unterminated-define-run.txt

+# You cannot continue a DEFINE with a RUN.
+
+# DEFINE: %{foo}=echo \
+# RUN: %{foo}
+# DEFINE: %{too-late-to-continue}=
+
+# CHECK: Test has unterminated '{{DEFINE}}:' directive (with '\') at line [[#@LINE-4]]
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/continuation/unterminated-define.txt

+# Simple case of unterminated define as last directive in the script.
+
+# RUN: echo "Don't complain about no RUN lines."
+# DEFINE: %{foo}=foo\
+
+# CHECK: Test has unterminated '{{DEFINE}}:' directive (with '\') at line [[#@LINE-2]]
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/continuation/unterminated-redefine-bad-define.txt

+# You cannot continue a REDEFINE with a DEFINE.  In this case, the DEFINE is
+# wrong on its own because it's written as a continuation, so the bad DEFINE is
+# reported.
+
+# REDEFINE: %{global:what}=echo \
+# DEFINE: Hello World
+# REDEFINE: too late to continue
+# RUN: %{foo}
+
+#      CHECK: Substitution's definition does not contain '='
+# CHECK-NEXT: in {{DEFINE}}: directive on test line [[#@LINE-5]]
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/continuation/unterminated-redefine-continuation.txt

+# Check a redefine's continuation line that is unterminated.
+
+# RUN: echo "Don't complain about no RUN lines."
+# REDEFINE: %{global:what} = foo \
+# REDEFINE:                  bar \
+
+# CHECK: Test has unterminated '{{REDEFINE}}:' directive (with '\') from line [[#@LINE-3]] to [[#@LINE-2]]
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/continuation/unterminated-redefine-define.txt

+# You cannot continue a REDEFINE with a DEFINE.  In this case, the DEFINE
+# looks fine on its own, so the unterminated REDEFINE is reported.
+
+# DEFINE: %{name}=x
+# DEFINE: %{value}=3
+# REDEFINE: %{global:what}=echo \
+# DEFINE: %{name}=%{value}
+# REDEFINE: %{too-late-to-continue}=
+# RUN: %{deceptive-continue}
+
+# CHECK: Test has unterminated '{{REDEFINE}}:' directive (with '\') at line [[#@LINE-5]]
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/continuation/unterminated-redefine-run.txt

+# You cannot continue a REDEFINE with a RUN.
+
+# REDEFINE: %{global:what}=echo \
+# RUN: %{global:what}
+# REDEFINE: %{too-late-to-continue}=
+
+# CHECK: Test has unterminated '{{REDEFINE}}:' directive (with '\') at line [[#@LINE-4]]
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/continuation/unterminated-redefine.txt

+# Simple case of unterminated redefine as last directive in the script.
+
+# RUN: echo "Don't complain about no RUN lines."
+# REDEFINE: %{global:what}=foo\
+
+# CHECK: Test has unterminated '{{REDEFINE}}:' directive (with '\') at line [[#@LINE-2]]
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/continuation/unterminated-run-define.txt

+# You cannot continue a RUN with a DEFINE.
+
+# RUN: echo \
+# DEFINE: %{foo}=bar
+# RUN: too late to continue
+
+# CHECK: Test has unterminated '{{RUN}}:' directive (with '\') at line [[#@LINE-4]]
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/continuation/unterminated-run-redefine.txt

+# You cannot continue a RUN with a REDEFINE.
+
+# RUN: echo Hello \
+# REDEFINE: %{global:what}=bar
+# RUN: too late to continue
+
+# CHECK: Test has unterminated '{{RUN}}:' directive (with '\') at line [[#@LINE-4]]
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/continuation/ws-only.txt

+# Surely an empty substitution continuation is an accident, so it's an error.
+
+# DEFINE: %{foo} = foo \
+# DEFINE:  
+#        ^~ whitespace intentional
+# RUN: echo '%{foo}'
+
+#      CHECK: Substitution's continuation is empty
+# CHECK-NEXT: in {{DEFINE}}: directive on test line [[#@LINE-5]]
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/defined-check/define-already-by-config.txt

+# DEFINE: %{global:greeting}=Hello
+# RUN: %{global:echo}
+
+#      CHECK: ValueError: Substitution whose pattern contains '%{global:greeting}' is already defined before '{{DEFINE}}:' directive at line [[#@LINE-3]]
+# CHECK-NEXT: Existing pattern: %{global:greeting}
+#  CHECK-NOT: Existing pattern:
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/defined-check/define-already-by-test.txt

+# DEFINE: %{foo}=bar
+# RUN: echo %{foo}
+# DEFINE: %{foo}=bar
+# RUN: echo %{foo}
+
+#      CHECK: ValueError: Substitution whose pattern contains '%{foo}' is already defined before '{{DEFINE}}:' directive at line [[#@LINE-3]]
+# CHECK-NEXT: Existing pattern: %{foo}
+#  CHECK-NOT: Existing pattern:
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/defined-check/define-inside-pattern.txt

+# DEFINE: %{global:inside} = @
+# RUN: echo '<%{global:inside}>'
+
+#      CHECK: ValueError: Substitution whose pattern contains '%{global:inside}' is already defined before '{{DEFINE}}:' directive at line [[#@LINE-3]]
+# CHECK-NEXT: Existing pattern: <%{global:inside}>
+#  CHECK-NOT: Existing pattern:
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/defined-check/define-multiple-exact.txt

+# DEFINE: %{global:multiple-exact} = @
+# RUN: echo '%{global:multiple-exact}'
+
+#      CHECK: ValueError: Substitution whose pattern contains '%{global:multiple-exact}' is already defined before '{{DEFINE}}:' directive at line [[#@LINE-3]]
+# CHECK-NEXT: Existing pattern: %{global:multiple-exact}
+# CHECK-NEXT: Existing pattern: %{global:multiple-exact}
+#  CHECK-NOT: Existing pattern:
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/defined-check/define-multiple-once-exact.txt

+# DEFINE: %{global:multiple-once-exact} = @
+# RUN: echo '%{global:multiple-once-exact}'
+
+#      CHECK: ValueError: Substitution whose pattern contains '%{global:multiple-once-exact}' is already defined before '{{DEFINE}}:' directive at line [[#@LINE-3]]
+# CHECK-NEXT: Existing pattern: <%{global:multiple-once-exact}>
+# CHECK-NEXT: Existing pattern: %{global:multiple-once-exact}
+#  CHECK-NOT: Existing pattern:
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/defined-check/define-prefixes-pattern.txt

+# DEFINE: %{global:prefix} = @
+# RUN: echo '%{global:prefix}(foo)'
+
+#      CHECK: ValueError: Substitution whose pattern contains '%{global:prefix}' is already defined before '{{DEFINE}}:' directive at line [[#@LINE-3]]
+# CHECK-NEXT: Existing pattern: %{global:prefix}\((.*)\)
+#  CHECK-NOT: Existing pattern:
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/defined-check/define-suffixes-pattern.txt

+# DEFINE: %{global:suffix} = @
+# RUN: echo '@%{global:suffix}'
+
+#      CHECK: ValueError: Substitution whose pattern contains '%{global:suffix}' is already defined before '{{DEFINE}}:' directive at line [[#@LINE-3]]
+# CHECK-NEXT: Existing pattern: @%{global:suffix}
+#  CHECK-NOT: Existing pattern:
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/defined-check/redefine-inside-pattern.txt

+# REDEFINE: %{global:inside} = @
+# RUN: echo '<%{global:inside}>'
+
+#      CHECK: ValueError: Existing substitution whose pattern contains '%{global:inside}' does not have the pattern specified by '{{REDEFINE}}:' directive at line [[#@LINE-3]]
+# CHECK-NEXT: Expected pattern: %{global:inside}
+# CHECK-NEXT: Existing pattern: <%{global:inside}>
+#  CHECK-NOT: Existing pattern:
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/defined-check/redefine-multiple-exact.txt

+# It's impossible to multiply define a local substitution, but a lit config file
+# substitution can be multiply defined.  The trouble is we then don't know which
+# definition to redefine locally.
+
+# REDEFINE: %{global:multiple-exact}=foo
+# RUN: echo %{global:multiple-exact}
+
+#      CHECK: ValueError: Multiple substitutions whose patterns contain '%{global:multiple-exact}' are defined before '{{REDEFINE}}:' directive at line [[#@LINE-3]]
+# CHECK-NEXT: Existing pattern: %{global:multiple-exact}
+# CHECK-NEXT: Existing pattern: %{global:multiple-exact}
+#  CHECK-NOT: Existing pattern:
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/defined-check/redefine-multiple-once-exact.txt

+# There's only one substitution whose pattern matches exactly, but there's
+# another partial match, and that can lead to confusing expansions.
+
+# REDEFINE: %{global:multiple-once-exact}=foo
+# RUN: echo %{global:multiple-once-exact}
+
+#      CHECK: ValueError: Multiple substitutions whose patterns contain '%{global:multiple-once-exact}' are defined before '{{REDEFINE}}:' directive at line [[#@LINE-3]]
+# CHECK-NEXT: Existing pattern: <%{global:multiple-once-exact}>
+# CHECK-NEXT: Existing pattern: %{global:multiple-once-exact}
+#  CHECK-NOT: Existing pattern:
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/defined-check/redefine-none.txt

+# REDEFINE: %{local}=foo
+# RUN: echo %{local}
+
+#     CHECK: ValueError: No substitution for '%{local}' is defined before '{{REDEFINE}}:' directive at line [[#@LINE-3]]
+# CHECK-NOT: Existing pattern:
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/defined-check/redefine-prefixes-pattern.txt

+# REDEFINE: %{global:prefix} = @
+# RUN: echo '%{global:prefix}(foo)'
+
+#      CHECK: ValueError: Existing substitution whose pattern contains '%{global:prefix}' does not have the pattern specified by '{{REDEFINE}}:' directive at line [[#@LINE-3]]
+# CHECK-NEXT: Expected pattern: %{global:prefix}
+# CHECK-NEXT: Existing pattern: %{global:prefix}\((.*)\)
+#  CHECK-NOT: Existing pattern:
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/defined-check/redefine-suffixes-pattern.txt

+# REDEFINE: %{global:suffix} = @
+# RUN: echo '@%{global:suffix}'
+
+#      CHECK: ValueError: Existing substitution whose pattern contains '%{global:suffix}' does not have the pattern specified by '{{REDEFINE}}:' directive at line [[#@LINE-3]]
+# CHECK-NEXT: Expected pattern: %{global:suffix}
+# CHECK-NEXT: Existing pattern: @%{global:suffix}
+#  CHECK-NOT: Existing pattern:
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/location-range.txt

+# DEFINE: %{foo}=Hello World
+# DEFINE: %{foo}=Hello \
+# DEFINE:   World
+# RUN: echo '%{foo}'
+
+# CHECK: ValueError: Substitution whose pattern contains '%{foo}' is already defined before '{{DEFINE}}:' directive from line [[#@LINE-4]] to [[#@LINE-3]]
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/errors/no-run.txt

+# DEFINE and REDEFINE are not sufficient.  There must be a RUN.
+
+# DEFINE: %{local:echo}=foo
+# REDEFINE: %{global:echo}=bar
+
+# CHECK: Test has no '{{RUN}}:' line
+
+# CHECK: Unresolved: 1

llvm/utils/lit/tests/Inputs/shtest-define/examples/param-subst.txt

+; This example originally appeared in TestingGuide.rst except here we've added
+; echo to the clang/FileCheck command line to be executed.
+
+; DEFINE: %{cflags} =
+; DEFINE: %{fcflags} =
+
+; DEFINE: %{check} =                                                           \
+; DEFINE:   echo '                                                             \
+; DEFINE:   %clang_cc1 -verify -fopenmp -fopenmp-version=51 %{cflags}          \
+; DEFINE:              -emit-llvm -o - %s |                                    \
+; DEFINE:     FileCheck %{fcflags} %s                                          \
+; DEFINE:   '
+
+; REDEFINE: %{cflags} = -triple x86_64-apple-darwin10.6.0 -fopenmp-simd
+; REDEFINE: %{fcflags} = -check-prefix=SIMD
+; RUN: %{check}
+; CHECK: %clang_cc1 -verify -fopenmp -fopenmp-version=51 -triple x86_64-apple-darwin10.6.0 -fopenmp-simd -emit-llvm -o - {{.*}} | FileCheck -check-prefix=SIMD {{.*}}
+
+; REDEFINE: %{cflags} = -triple x86_64-unknown-linux-gnu -fopenmp-simd
+; REDEFINE: %{fcflags} = -check-prefix=SIMD
+; RUN: %{check}
+; CHECK: %clang_cc1 -verify -fopenmp -fopenmp-version=51 -triple x86_64-unknown-linux-gnu -fopenmp-simd -emit-llvm -o - {{.*}} | FileCheck -check-prefix=SIMD {{.*}}
+
+; REDEFINE: %{cflags} = -triple x86_64-apple-darwin10.6.0
+; REDEFINE: %{fcflags} = -check-prefix=NO-SIMD
+; RUN: %{check}
+; CHECK: %clang_cc1 -verify -fopenmp -fopenmp-version=51 -triple x86_64-apple-darwin10.6.0 -emit-llvm -o - {{.*}} | FileCheck -check-prefix=NO-SIMD {{.*}}
+
+; REDEFINE: %{cflags} = -triple x86_64-unknown-linux-gnu
+; REDEFINE: %{fcflags} = -check-prefix=NO-SIMD
+; RUN: %{check}
+; CHECK: %clang_cc1 -verify -fopenmp -fopenmp-version=51 -triple x86_64-unknown-linux-gnu -emit-llvm -o - {{.*}} | FileCheck -check-prefix=NO-SIMD {{.*}}
+
+; CHECK: Passed: 1

llvm/utils/lit/tests/Inputs/shtest-define/expansion-order.txt

+# Redefine the %{global:greeting} parameter of %{global:echo} before using it.
+# The necessary expansion order was established in the test suite config
+# (%{global:echo} before %{global:greeting}), and redefining the parameter
+# doesn't change that expansion order.  That order would be reversed if
+# %{global:greeting} were handled as a new substitution, preventing full
+# expansion.
+#
+# REDEFINE: %{global:greeting}=Hello
+# RUN: %{global:echo}
+# CHECK: GLOBAL: Hello World
+
+# We can redefine the test suite config's substitutions multiple times.  Again,
+# the expansion order remains the same (%{global:echo} before %{global:greeting}
+# before %{global:what}) but would be reversed if these were handled as new
+# substitutions, preventing full expansion.
+#
+# REDEFINE: %{global:greeting}=Goodbye %{global:what}
+# REDEFINE: %{global:what}=Sleep
+# RUN: %{global:echo}
+# CHECK: GLOBAL: Goodbye Sleep Sleep
+
+# A new local substitution is prepended to the substitution list so that it can
+# depend on all substitutions that were defined previously, including those from
+# the test suite config.
+#
+# DEFINE: %{local:greeting}=Hey %{global:what}
+# DEFINE: %{local:echo}=echo "LOCAL: %{local:greeting} %{global:what}"
+# RUN: %{local:echo}
+# CHECK: LOCAL: Hey Sleep Sleep
+
+# As for substitutions from the test suite config, redefining local
+# substitutions should not change the expansion order.  Again, the expansion
+# order would be reversed if these were instead handled as new substitutions,
+# preventing full expansion.
+#
+# REDEFINE: %{local:greeting}=So Long %{global:what}
+# REDEFINE: %{global:what}=World
+# RUN: %{local:echo}
+# CHECK: LOCAL: So Long World World
+
+# CHECK: Passed: 1

llvm/utils/lit/tests/Inputs/shtest-define/line-number-substitutions.txt

+# Does it work as expected directly in RUN lines?
+# RUN: echo %(line), %(line-1), %(line+2)
+# CHECK: [[#@LINE-1]], [[#@LINE-2]], [[#@LINE+1]]
+
+# %(line) substitutions refer to the original DEFINE/REDEFINE line not the RUN
+# line they eventually appear within.
+#
+# DEFINE: %{lines} = %(line)
+# RUN: echo '%{lines}'
+# CHECK: [[#@LINE-2]]
+#
+# REDEFINE: %{lines} = %(line),                                                \
+# REDEFINE:            %(line),                                                \
+# REDEFINE:            %(line)
+# RUN: echo '%{lines}'
+# CHECK: [[#@LINE-4]], [[#@LINE-3]], [[#@LINE-2]]
+
+# %(line+N) and %{line-N) should work too.
+#
+# DEFINE: %{lines-rel} = %(line+1),                                            \
+# DEFINE:                %(line),                                              \
+# DEFINE:                %(line-1)
+# RUN: echo '%{lines-rel}'
+# CHECK: [[#@LINE-3]], [[#@LINE-3]], [[#@LINE-3]]
+#
+# REDEFINE: %{lines-rel} = %(line+5),                                          \
+# REDEFINE:                %(line+0),                                          \
+# REDEFINE:                %(line-10)
+# RUN: echo '%{lines-rel}'
+# CHECK: [[#@LINE+1]], [[#@LINE-3]], [[#@LINE-12]]
+
+# CHECK: Passed: 1

llvm/utils/lit/tests/Inputs/shtest-define/lit.cfg

+import lit.formats
+config.name = 'shtest-define'
+config.suffixes = ['.txt']
+config.test_format = lit.formats.ShTest()
+config.test_source_root = None
+config.test_exec_root = None
+
+# When config.recursiveExpansionLimit is not specified, it's important to
+# prepend substitutions before substitutions they might now or later (upon a
+# redefinition) depend upon.  For example, %{global:greeting} and %{global:what}
+# act as parameters for %{global:echo}, so we make sure the latter expands
+# before the former.  Moreover, some tests redefine %{global:greeting} in terms
+# of %{global:what}, so we make sure the former expands before the latter.
+# If we always insert at the beginning of the substitution list (as DEFINE
+# does), then the rule is simple: define a substitution before you refer to it.
+config.substitutions.insert(0, ('%{global:what}', 'World'))
+config.substitutions.insert(0, ('%{global:greeting}', ''))
+config.substitutions.insert(0,
+    ('%{global:echo}', "echo GLOBAL: %{global:greeting} %{global:what}"))
+
+# The following substitution definitions are confusing and should be avoided.
+# We define them here so we can test that 'DEFINE:' and 'REDEFINE:' directives
+# guard against the confusion they cause.
+
+# Even though each of '%{global:inside}', '%{global:prefix}', and
+# '%{global:suffix}' is not already the exact pattern of a substitution,
+# 'DEFINE:' and 'REDEFINE:' will refuse to (re)define a substitution with that
+# pattern because it is a substring of one of the following substitution's
+# patterns.
+config.substitutions.insert(0, ('<%{global:inside}>', '<@>'))
+config.substitutions.insert(0, (r'%{global:prefix}\((.*)\)', r'@(\g<1>)'))
+config.substitutions.insert(0, ('@%{global:suffix}', '@@'))
+
+# These cannot be redefined by 'REDEFINE:', which doesn't know which one to
+# redefine.
+config.substitutions.insert(0, ('%{global:multiple-exact}', 'first'))
+config.substitutions.insert(0, ('%{global:multiple-exact}', 'second'))
+
+# Even though '%{global:multiple-once-exact}' is the exact pattern of only one
+# existing substitution, 'REDEFINE:' will refuse to redefine that substitution
+# because that string is a substring of another substitution's pattern.
+config.substitutions.insert(0, ('%{global:multiple-once-exact}', '@'))
+config.substitutions.insert(0, ('<%{global:multiple-once-exact}>', '<@>'))
+
+recur = lit_config.params.get('recur', None)
+if recur:
+  config.recursiveExpansionLimit = int(recur)

llvm/utils/lit/tests/Inputs/shtest-define/name-chars.txt

+# DEFINE: %{abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789} = ok
+# RUN: echo '%{abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789}'
+# CHECK: ok
+
+# DEFINE: %{FooBar} = ok at %(line)
+# RUN: echo '%{FooBar}'
+# CHECK: ok at [[#@LINE - 2]]
+
+# DEFINE: %{fooBar} = ok at %(line)
+# RUN: echo '%{fooBar}'
+# CHECK: ok at [[#@LINE - 2]]
+
+# DEFINE: %{foo-bar-} = ok at %(line)
+# RUN: echo '%{foo-bar-}'
+# CHECK: ok at [[#@LINE - 2]]
+
+# DEFINE: %{foo:bar:} = ok at %(line)
+# RUN: echo '%{foo:bar:}'
+# CHECK: ok at [[#@LINE - 2]]
+
+# DEFINE: %{_foo_bar_} = ok at %(line)
+# RUN: echo '%{_foo_bar_}'
+# CHECK: ok at [[#@LINE - 2]]
+
+# CHECK: Passed: 1

llvm/utils/lit/tests/Inputs/shtest-define/recursiveExpansionLimit.txt

+# These are defined in the wrong order for non-recursive expansion: %{inner} is
+# defined after it's referenced.
+
+# DEFINE: %{outer}=%{inner}
+# DEFINE: %{inner}=expanded
+
+# RUN: echo '%{outer}'
+
+# CHECK-NON-RECUR:%{inner}
+# CHECK-RECUR:expanded
+
+# CHECK: Passed: 1

llvm/utils/lit/tests/Inputs/shtest-define/shared-substs-0.txt

+# RUN: echo shared-substs-0.txt
+
+# Make sure we don't see modifications that other shared-substs-*.txt have made
+# to the test suite config's substitutions.
+#
+# RUN: %{global:echo}
+
+# Next, modify substs that would affect the above.  Verify they are set locally.
+#
+# REDEFINE: %{global:what}=LOCAL0:World
+# REDEFINE: %{global:greeting}=LOCAL0:Hello
+# REDEFINE: %{global:echo}=echo LOCAL0: %{global:greeting} %{global:what}
+# RUN: %{global:echo}
+
+# Finally, set a local that other shared-substs-*.txt also set to be sure
+# there's no redefinition complaint because they left it behind.  Verify it is
+# set locally.
+#
+# DEFINE: %{local:echo}=echo LOCAL0: subst
+# RUN: %{local:echo}
+
+# CHECK: Passed: 1

llvm/utils/lit/tests/Inputs/shtest-define/shared-substs-1.txt

+# RUN: echo shared-substs-1.txt
+
+# Make sure we don't see modifications that other shared-substs-*.txt have made
+# to the test suite config's substitutions.
+#
+# RUN: %{global:echo}
+
+# Next, modify substs that would affect the above.  Verify they are set locally.
+#
+# REDEFINE: %{global:what}=LOCAL1:World
+# REDEFINE: %{global:greeting}=LOCAL1:Hello
+# REDEFINE: %{global:echo}=echo LOCAL1: %{global:greeting} %{global:what}
+# RUN: %{global:echo}
+
+# Finally, set a local that other shared-substs-*.txt also set to be sure
+# there's no redefinition complaint because they left it behind.  Verify it is
+# set locally.
+#
+# DEFINE: %{local:echo}=echo LOCAL1: subst
+# RUN: %{local:echo}
+
+# CHECK: Passed: 1

llvm/utils/lit/tests/Inputs/shtest-define/value-equals.txt

+# After the initial '=', the value can contain '=' with no special meaning.
+
+# DEFINE: %{equals} = FileCheck -check-prefixes=FOO,BAR
+# RUN: echo '%{equals}'
+# CHECK: FileCheck -check-prefixes=FOO,BAR
+#
+# REDEFINE: %{equals} == == =
+# RUN: echo '%{equals}'
+# CHECK: = == =
+
+# DEFINE: %{continue-equals} = FileCheck -strict-whitespace -match-full-lines \
+# DEFINE:                      -check-prefixes=FOO,BAR
+# RUN: echo '%{continue-equals}'
+# CHECK: FileCheck -strict-whitespace -match-full-lines -check-prefixes=FOO,BAR
+#
+# REDEFINE: %{continue-equals} = FileCheck -input-file=test.txt                \
+# REDEFINE:                      -implicit-check-not=foobar                    \
+# REDEFINE:                      -check-prefixes=FOO,BAR
+# RUN: echo '%{continue-equals}'
+# CHECK: FileCheck -input-file=test.txt -implicit-check-not=foobar -check-prefixes=FOO,BAR
+
+# CHECK: Passed: 1

llvm/utils/lit/tests/Inputs/shtest-define/value-escaped.txt

+# Escape sequences that can appear in python re.sub replacement strings have no
+# special meaning in the value.
+
+# DEFINE: %{escape} = \g<0>\n
+# RUN: echo '%{escape}'
+# CHECK: \g<0>\n
+
+# REDEFINE: %{escape} = \n                                                     \
+# REDEFINE:             \g<param>
+# RUN: echo '%{escape}'
+# CHECK: \n \g<param>
+
+# CHECK: Passed: 1

llvm/utils/lit/tests/Inputs/shtest-define/ws-and-continuations.txt

+# Empty values are permitted and reasonable, especially when just establishing
+# expansion order.
+#
+# DEFINE: %{empty}=
+# RUN: echo "'%{empty}'"
+# CHECK:''
+#
+# REDEFINE: %{empty}=
+# RUN: echo "'%{empty}'"
+# CHECK:''
+
+# A value consisting only of whitespace is trimmed to the empty string.
+#
+#               v~~ intentional whitespace
+# DEFINE: %{ws}=   
+# RUN: echo "'%{ws}'"
+# CHECK:''
+#
+#                 v intentional whitespace
+# REDEFINE: %{ws}= 
+# RUN: echo "'%{ws}'"
+# CHECK:''
+
+# Whitespace is not required around the name or value.
+#
+# DEFINE:%{no-whitespace}=abc
+# RUN: echo "'%{no-whitespace}'"
+# CHECK:'abc'
+#
+# REDEFINE:%{no-whitespace}=HelloWorld
+# RUN: echo "'%{no-whitespace}'"
+# CHECK:'HelloWorld'
+
+# Whitespace is not required between substitutions in a value.
+#
+# DEFINE: %{adjacent0} = foo
+# DEFINE: %{adjacent1} = bar
+# DEFINE: %{has-adjacent-substs} = %{adjacent0}%{adjacent1}
+# RUN: echo "'%{has-adjacent-substs}'"
+# CHECK:'foobar'
+#
+# REDEFINE: %{has-adjacent-substs} = %{adjacent0}%{adjacent1}%{adjacent0}
+# RUN: echo "'%{has-adjacent-substs}'"
+# CHECK:'foobarfoo'
+
+# Exact whitespace is preserved within the value, but whitespace enclosing the
+# name or value is discarded.  ('%{' and '}' are part of the name, and
+# whitespace in between isn't permitted.)
+#
+#                                       v~~ intentional whitespace
+# DEFINE:   %{whitespace}  =  abc    def   
+# RUN: echo "'%{whitespace}'"
+# CHECK:'abc    def'
+#                                      v intentional whitespace
+# REDEFINE: %{whitespace} = Hello World 
+# RUN: echo "'%{whitespace}'"
+# CHECK:'Hello World'
+
+# Line continuations in the value are permitted and collapse whitespace.
+#
+# DEFINE: %{continue} = abc\
+# DEFINE:def \
+# DEFINE:ghi\
+# DEFINE: jkl \
+# DEFINE: mno  \
+# DEFINE:  pqr 
+#             ^ intentional whitespace
+# RUN: echo "'%{continue}'"
+# CHECK:'abc def ghi jkl mno pqr'
+#
+# REDEFINE: %{continue} =  abc  \
+# REDEFINE: def
+# RUN: echo "'%{continue}'"
+# CHECK:'abc def'
+
+# Whitespace at the end of the line after a '\' is ignored, and it's treated as
+# a line continuation.  Otherwise, the behavior would be hard to understand
+# because it looks like a line continuation.
+#
+#                                   v~~~~~~~~~~~ intentional whitespace
+# DEFINE: %{ws-after-continue}=foo \            
+# DEFINE: bar                      \ 
+#                                   ^ intentional whitespace
+# DEFINE: baz
+# RUN: echo "'%{ws-after-continue}'"
+# CHECK:'foo bar baz'
+#
+#                                     v intentional whitespace
+# REDEFINE: %{ws-after-continue}=foo \ 
+# REDEFINE: bar                      \            
+#                                     ^~~~~~~~~~~~ intentional whitespace
+# REDEFINE: baz
+# RUN: echo "'%{ws-after-continue}'"
+# CHECK:'foo bar baz'
+
+# A line continuation is recognized anywhere.  It should be used only where
+# whitespace is permitted because it reduces to a single space.
+#
+# Directives with at least one non-whitespace character (could be '\') are
+# permitted even if they contribute nothing to the value.  There might be no
+# practical use, but check that it behaves as expected.
+#
+# DEFINE:\
+# DEFINE:%{blank-lines}\
+# DEFINE:\
+# DEFINE:=\
+# DEFINE:\
+# DEFINE:a
+# RUN: echo "'%{blank-lines}'"
+# CHECK:'a'
+#
+# REDEFINE:                  \
+# REDEFINE: %{blank-lines}   \
+# REDEFINE:                  \
+# REDEFINE:                = \
+# REDEFINE:                  \
+# REDEFINE:      a           \
+# REDEFINE:                  \
+# REDEFINE:      b           \
+# REDEFINE:                  \
+# REDEFINE:      c
+# RUN: echo "'%{blank-lines}'"
+# CHECK:'a b c'
+
+# The fourth DEFINE line is deceptive because it looks like a new substitution,
+# but it's actually a continuation of the previous value.
+#
+# DEFINE: %{name}=x
+# DEFINE: %{value}=3
+# DEFINE: %{deceptive-continue}=echo \
+# DEFINE: %{name}=%{value}
+# RUN: %{deceptive-continue}
+# CHECK:x=3
+
+# CHECK:{{ *}}Passed: 1

llvm/utils/lit/tests/Inputs/shtest-shell/continuations.txt

+RUN: echo 'foo' \
+RUN:      'bar' >> %t.out
+CHECK: foo bar
+
+RUN: echo 'foo' \
+RUN:      'bar' \
+RUN:      'baz' >> %t.out
+CHECK: foo bar baz
+
+#                 v~~ intentional whitespace
+RUN: echo 'foo'  \   
+RUN:      'bar'  \ 
+#                 ^ intentional whitespace
+RUN:      'baz' >> %t.out
+CHECK: foo bar baz
+
+RUN: FileCheck -match-full-lines -input-file=%t.out %s

llvm/utils/lit/tests/Inputs/testrunner-custom-parsers/test.txt

 // MY_BOOL: b)
 // MY_BOOL: d
 //
 // MY_INT: 4
 // MY_INT: 6
 //
 // MY_BOOL_UNTERMINATED: a \
 //
+// MY_DEFINE: %{name} = value one
+//
+// MY_REDEFINE: %{name} = value \
+// MY_REDEFINE:           two
+//
 // END.
 // MY_LIST: five

llvm/utils/lit/tests/shtest-define.py

+# We're using DEFINE/REDEFINE to help us write tests for DEFINE/REDEFINE.
+
+# RUN: echo '-- Available Tests --' > %t.tests.actual.txt
+
+# DEFINE: %{my-inputs} = %{inputs}/shtest-define
+
+# DEFINE: %{test} =
+# DEFINE: %{lit-pre} =
+# DEFINE: %{lit-args} =
+# DEFINE: %{fc-args} =
+# DEFINE: %{run-test} =                                                        \
+# DEFINE:   %{lit-pre} %{lit} -va  %{lit-args} %{my-inputs}/%{test} 2>&1 |     \
+# DEFINE:     FileCheck -match-full-lines %{fc-args} %{my-inputs}/%{test}
+# DEFINE: %{record-test} =                                                     \
+# DEFINE:   echo '  shtest-define :: %{test}' >> %t.tests.actual.txt
+# DEFINE: %{run-and-record-test} = %{run-test} && %{record-test}
+
+# REDEFINE: %{lit-pre} = not
+#
+# REDEFINE: %{test} = errors/assignment/before-name.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/assignment/between-name-equals.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/assignment/braces-empty.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/assignment/braces-with-dot.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/assignment/braces-with-equals.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/assignment/braces-with-newline.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/assignment/braces-with-number.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/assignment/braces-with-ws.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/assignment/empty.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/assignment/no-equals.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/assignment/no-name.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/assignment/ws-only.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/continuation/empty.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/continuation/end-in-double-backslash.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/continuation/unterminated-define-bad-redefine.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/continuation/unterminated-define-continuation.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/continuation/unterminated-define-redefine.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/continuation/unterminated-define-run.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/continuation/unterminated-define.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/continuation/unterminated-redefine-bad-define.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/continuation/unterminated-redefine-continuation.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/continuation/unterminated-redefine-define.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/continuation/unterminated-redefine-run.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/continuation/unterminated-redefine.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/continuation/unterminated-run-define.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/continuation/unterminated-run-redefine.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/continuation/ws-only.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/defined-check/define-already-by-config.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/defined-check/define-already-by-test.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/defined-check/define-inside-pattern.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/defined-check/define-multiple-exact.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/defined-check/define-multiple-once-exact.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/defined-check/define-prefixes-pattern.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/defined-check/define-suffixes-pattern.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/defined-check/redefine-inside-pattern.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/defined-check/redefine-multiple-exact.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/defined-check/redefine-multiple-once-exact.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/defined-check/redefine-none.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/defined-check/redefine-prefixes-pattern.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/defined-check/redefine-suffixes-pattern.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/location-range.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{test} = errors/no-run.txt
+# RUN: %{run-and-record-test}
+#
+# REDEFINE: %{lit-pre} =
+
+# REDEFINE: %{test} = examples/param-subst.txt
+# RUN: %{run-and-record-test}
+
+# REDEFINE: %{test} = expansion-order.txt
+# RUN: %{run-and-record-test}
+
+# REDEFINE: %{test} = line-number-substitutions.txt
+# RUN: %{run-and-record-test}
+
+# REDEFINE: %{test} = name-chars.txt
+# RUN: %{run-and-record-test}
+
+# REDEFINE: %{test} = recursiveExpansionLimit.txt
+#
+# REDEFINE: %{fc-args} = -check-prefix=CHECK-NON-RECUR
+# RUN: %{run-test}
+#
+# REDEFINE: %{lit-args} = -Drecur=2
+# REDEFINE: %{fc-args} = -check-prefix=CHECK-RECUR
+# RUN: %{run-test}
+#
+# RUN: %{record-test}
+# REDEFINE: %{lit-args} =
+# REDEFINE: %{fc-args} =
+
+# Check that per-test changes to substitutions don't affect other tests in the
+# same LIT invocation.
+#
+# RUN: %{lit} -va %{my-inputs}/shared-substs-*.txt 2>&1 |                      \
+# RUN:   FileCheck -check-prefix=SHARED-SUBSTS -match-full-lines %s
+#
+# SHARED-SUBSTS: shared-substs-0.txt
+# SHARED-SUBSTS: GLOBAL: World
+# SHARED-SUBSTS: LOCAL0: LOCAL0:Hello LOCAL0:World
+# SHARED-SUBSTS: LOCAL0: subst
+#
+# SHARED-SUBSTS: shared-substs-1.txt
+# SHARED-SUBSTS: GLOBAL: World
+# SHARED-SUBSTS: LOCAL1: LOCAL1:Hello LOCAL1:World
+# SHARED-SUBSTS: LOCAL1: subst
+#
+# REDEFINE: %{test} = shared-substs-0.txt
+# RUN: %{record-test}
+# REDEFINE: %{test} = shared-substs-1.txt
+# RUN: %{record-test}
+
+# REDEFINE: %{test} = value-equals.txt
+# RUN: %{run-and-record-test}
+
+# REDEFINE: %{test} = value-escaped.txt
+# RUN: %{run-and-record-test}
+
+# REDEFINE: %{fc-args} = -strict-whitespace
+# REDEFINE: %{test} = ws-and-continuations.txt
+# RUN: %{run-and-record-test}
+# REDEFINE: %{fc-args} =
+
+# Make sure we didn't forget to run something.
+#
+# RUN: %{lit} --show-tests %{my-inputs} > %t.tests.expected.txt
+# RUN: diff -u %t.tests.expected.txt %t.tests.actual.txt

llvm/utils/lit/tests/shtest-keyword-parse-errors.py

 # RUN: not %{lit} -vv %{inputs}/shtest-keyword-parse-errors > %t.out
 # RUN: FileCheck -input-file %t.out %s
 #
 # END.
 
 # CHECK: Testing: 3 tests
 
 # CHECK-LABEL: UNRESOLVED: shtest-keyword-parse-errors :: empty.txt
 # CHECK:       {{^}}Test has no 'RUN:' line{{$}}
 
 # CHECK-LABEL: UNRESOLVED: shtest-keyword-parse-errors :: multiple-allow-retries.txt
 # CHECK:       {{^}}Test has more than one ALLOW_RETRIES lines{{$}}
 
 # CHECK-LABEL: UNRESOLVED: shtest-keyword-parse-errors :: unterminated-run.txt
-# CHECK:       {{^}}Test has unterminated 'RUN:' lines (with '\'){{$}}
+# CHECK:       {{^}}Test has unterminated 'RUN:' directive (with '\') at line 1{{$}}

llvm/utils/lit/tests/shtest-shell.py

 # CHECK: FAIL: shtest-shell :: colon-error.txt
 # CHECK: *** TEST 'shtest-shell :: colon-error.txt' FAILED ***
 # CHECK: $ ":"
 # CHECK: # command stderr:
 # CHECK: Unsupported: ':' cannot be part of a pipeline
 # CHECK: error: command failed with exit status: 127
 # CHECK: ***
 
+# CHECK: PASS: shtest-shell :: continuations.txt
+
 # CHECK: PASS: shtest-shell :: dev-null.txt
 
 # CHECK: FAIL: shtest-shell :: diff-b.txt
 # CHECK: *** TEST 'shtest-shell :: diff-b.txt' FAILED ***
 # CHECK: $ "diff" "-b" "{{[^"]*}}.0" "{{[^"]*}}.1"
 # CHECK: # command output:
 # CHECK: 1,2
 # CHECK-NEXT: {{^  }}f o o

llvm/utils/lit/tests/unit/TestRunner.py

def make_parsers():
             IntegratedTestKeywordParser("MY_TAG.", ParserKind.TAG),
             IntegratedTestKeywordParser("MY_DNE_TAG.", ParserKind.TAG),
             IntegratedTestKeywordParser("MY_LIST:", ParserKind.LIST),
             IntegratedTestKeywordParser("MY_BOOL:", ParserKind.BOOLEAN_EXPR),
             IntegratedTestKeywordParser("MY_INT:", ParserKind.INTEGER),
             IntegratedTestKeywordParser("MY_RUN:", ParserKind.COMMAND),
             IntegratedTestKeywordParser("MY_CUSTOM:", ParserKind.CUSTOM,
                                         custom_parse),
-
+            IntegratedTestKeywordParser("MY_DEFINE:", ParserKind.DEFINE),
+            IntegratedTestKeywordParser("MY_REDEFINE:", ParserKind.REDEFINE),
         ]
 
     @staticmethod
     def get_parser(parser_list, keyword):
         for p in parser_list:
             if p.keyword == keyword:
                 return p
         assert False and "parser not found"
def test_lists(self):
                               ['one', 'two', 'three', 'four'])
 
     def test_commands(self):
         parsers = self.make_parsers()
         self.parse_test(parsers)
         cmd_parser = self.get_parser(parsers, 'MY_RUN:')
         value = cmd_parser.getValue()
         self.assertEqual(len(value), 2)  # there are only two run lines
-        self.assertEqual(value[0].strip(), "%dbg(MY_RUN: at line 4)  baz")
-        self.assertEqual(value[1].strip(), "%dbg(MY_RUN: at line 7)  foo  bar")
+        self.assertEqual(value[0].command.strip(),
+                         "%dbg(MY_RUN: at line 4)  baz")
+        self.assertEqual(value[1].command.strip(),
+                         "%dbg(MY_RUN: at line 7)  foo  bar")
 
     def test_boolean(self):
         parsers = self.make_parsers()
         self.parse_test(parsers)
         bool_parser = self.get_parser(parsers, 'MY_BOOL:')
         value = bool_parser.getValue()
         self.assertEqual(len(value), 2)  # there are only two run lines
         self.assertEqual(value[0].strip(), "a && (b)")
class TestIntegratedTestKeywordParser(unittest.TestCase):
 
     def test_custom(self):
         parsers = self.make_parsers()
         self.parse_test(parsers)
         custom_parser = self.get_parser(parsers, 'MY_CUSTOM:')
         value = custom_parser.getValue()
         self.assertEqual(value, ['a', 'b', 'c'])
 
+    def test_defines(self):
+        parsers = self.make_parsers()
+        self.parse_test(parsers)
+        cmd_parser = self.get_parser(parsers, 'MY_DEFINE:')
+        value = cmd_parser.getValue()
+        self.assertEqual(len(value), 1) # there's only one MY_DEFINE directive
+        self.assertEqual(value[0].new_subst, True)
+        self.assertEqual(value[0].name, '%{name}')
+        self.assertEqual(value[0].value, 'value one')
+
+    def test_redefines(self):
+        parsers = self.make_parsers()
+        self.parse_test(parsers)
+        cmd_parser = self.get_parser(parsers, 'MY_REDEFINE:')
+        value = cmd_parser.getValue()
+        self.assertEqual(len(value), 1) # there's only one MY_REDEFINE directive
+        self.assertEqual(value[0].new_subst, False)
+        self.assertEqual(value[0].name, '%{name}')
+        self.assertEqual(value[0].value, 'value two')
+
     def test_bad_keywords(self):
         def custom_parse(line_number, line, output):
             return output
         
         try:
             IntegratedTestKeywordParser("TAG_NO_SUFFIX", ParserKind.TAG),
             self.fail("TAG_NO_SUFFIX failed to raise an exception")
         except ValueError as e: