[lit] Support %if ... %else syntax for RUN lines

[asavonic]

Summary

This syntax allows to modify RUN lines based on features
available. For example:

RUN: ... | FileCheck %s --check-prefix=%if windows %{W%} %else %{NON-W%}
CHECK-W: ...
CHECK-NON-W: ...

The whole command can be put under %if ... %else:

RUN: %if tool_available %{ %tool %} %else %{ true %}

or:

RUN: %if tool_available %{ %tool %}

If tool_available feature is missing, we'll have an empty command in this RUN line.
LIT used to emit an error for empty commands, but now it treats such commands as nop
in all cases.

Multi-line expressions are also supported:

RUN: %if tool_available %{ \
RUN:   %tool              \
RUN: %} %else %{             \
RUN:   true               \
RUN: %}

Background and motivation:
D121727 [NVPTX] Integrate ptxas to LIT tests
https://reviews.llvm.org/D121727

[tra]

llvm/docs/TestingGuide.rst

615

Would any of the following be expected to work?

• %if feature {do_something} | FileCheck %s -- this will probably fail as empty substitution would produce no output to check
• %if feature {do_something | FileCheck %s} -- this would probably work.
• %if featureA { %if featureB {do_AB} %else {do_A_notB}} %else {do_notA} -- I suspect we'll fail to parse it correctly.

I'd document that we currently can't nest those %if/%else and, maybe add the %if feature {do_something | FileCheck %s} as a canonical example of how to use it for conditional output-checking tests.

llvm/utils/lit/tests/shtest-if-else.py

2–3

Why not just pipe the output straight to FileCheck and avoid the temp file.

[delcypher]

Is

FileCheck %s --check-prefix={%if windows {W} else {NON-W}}

the main motivating example? You can already solve that by introducing a %check_prefix substitution in your lit.cfg.py file and expanding it to the correct thing depending on your platform. Would that suffice?

[asavonic]

llvm/docs/TestingGuide.rst

615

Would any of the following be expected to work?

• %if feature {do_something} | FileCheck %s -- this will probably fail as empty substitution would produce no output to check

Right. If the feature is not available then we'll get | FileCheck %s.

• %if feature {do_something | FileCheck %s} -- this would probably work.

This does not work for verbose output: LIT formats a command into echo "RUN at line #"; ${command}; and if ${command} is an empty string we'll get echo "RUN at line #"; ;. Bash cannot parse the double semicolon at the end.

We can probably handle this as a special case.

• %if featureA { %if featureB {do_AB} %else {do_A_notB}} %else {do_notA} -- I suspect we'll fail to parse it correctly.

Recursive substitution actually works if config.recursiveExpansionLimit is set in lit.cfg. I'll this case to the test.

I'd document that we currently can't nest those %if/%else and, maybe add the %if feature {do_something | FileCheck %s} as a canonical example of how to use it for conditional output-checking tests.

As I mentioned above, a canonical example would be:
%if feature {do_something | FileCheck %s} %else {true}

[delcypher]

Have you considered the different approach of having conditional RUN lines instead? E.g.

RUN: echo this always runs
RUN(feature): echo this only runs when feature is true
RUN(!feature): echo this only runs when feature is false

RUN(feature || feature2): echo this only runs where feature or feature2 is true.
RUN(!(feature || feature2)): echo this only runs where feature and feature2 are both false

I think this would make RUN lines easier to read. It doesn't support the "else" case quite as cleanly (you have to duplicate the condition and then negate it on separate line) but I think that's pretty minor because the condition can easily be made a "feature" that is computed from other features (in a test suite's lit.cfg.py)

llvm/docs/TestingGuide.rst

615

I am not a fan of this syntax.

1. Being prefixed with % suggests it is substitution, but %if and %else are not normal substitutions. It might be worth introducing different syntax for these.
2. Lit has this annoying problem with substitutions are prefixes of other substitutions (e.g. %s and %source) where the order that substitutions added matter. There's a possibility that by introducing %if and %else that this will break existing substitutions (e.g. %if_something and %else_something). You could avoid this by using something like %if% and %else%.

llvm/utils/lit/tests/Inputs/shtest-if-else/test.txt

10

Does parsing more complicated expressions work? I would expect it to, so we should probably add tests for it.

[tra]

llvm/docs/TestingGuide.rst

615

LIT formats a command into echo "RUN at line #"; ${command}; and if ${command} is an empty string we'll get echo "RUN at line #"; ;. Bash cannot parse the double semicolon at the end.

Perhaps we should expand the empty branch to : which *is* the standard null command in shell https://pubs.opengroup.org/onlinepubs/009695399/utilities/colon.html

[asavonic]

In D122569#3412197, @delcypher wrote:

Is

FileCheck %s --check-prefix={%if windows {W} else {NON-W}}

the main motivating example? You can already solve that by introducing a %check_prefix substitution in your lit.cfg.py file and expanding it to the correct thing depending on your platform. Would that suffice?

The main motivation is integration of external tools (ptxas, see D121727), where (1) it is important to make it clear that a command may not execute if the corresponding tool is not available, and (2) we need to support output redirection, different options for different tests, etc. Doing this in lit.cfg is either not clear, or not flexible (see https://reviews.llvm.org/D121727?id=416283, where we have ptxas-verify implemented in lit.cfg in comparison with if-like syntax).
Having this feature implemented as a substitution allows us to use it for other things, like platform dependent prefixes.

[jhenderson]

llvm/docs/TestingGuide.rst

615

Perhaps we should expand the empty branch to : which *is* the standard null command in shell https://pubs.opengroup.org/onlinepubs/009695399/utilities/colon.html

Don't forget Windows (where colons are not null actions).

I am not a fan of this syntax.

1. Being prefixed with % suggests it is substitution, but %if and %else are not normal substitutions. It might be worth introducing different syntax for these.
2. Lit has this annoying problem with substitutions are prefixes of other substitutions (e.g. %s and %source) where the order that substitutions added matter. There's a possibility that by introducing %if and %else that this will break existing substitutions (e.g. %if_something and %else_something). You could avoid this by using something like %if% and %else%.

I'm not convinced about adding more substitutions, especially %if% style since that's the canonical way of specifying environment variables in a Windows command prompt. %if/%else is pretty intuitive, I reckon. I personally don't think 2 is a real issue, unless there is already existing problems within the LLVM code-base like this. Yes, there might be downstream users, but a) we don't bend over backwards to avoid breaking them, and b) there's a clear path forward (rename the downstream substitution).

[jhenderson]

llvm/docs/TestingGuide.rst

615

I am not a fan of this syntax.

1. Being prefixed with % suggests it is substitution, but %if and %else are not normal substitutions. It might be worth introducing different syntax for these.
2. Lit has this annoying problem with substitutions are prefixes of other substitutions (e.g. %s and %source) where the order that substitutions added matter. There's a possibility that by introducing %if and %else that this will break existing substitutions (e.g. %if_something and %else_something). You could avoid this by using something like %if% and %else%.

llvm/utils/lit/tests/shtest-if-else.py

20

Nit: too many blank lines at EOF.

[probinson]

I see that for %if feature {A} %else {B} the space before the %else is not preserved in the result.
If there is no %else what happens? e.g., XX %if feature {YY} ZZ Asking because I see no test case for this.

I take it the %if expressions take only lit feature keywords, like REQUIRES? No triple substrings, as allowed by UNSUPPORTED/XFAIL?
(I have a back-burner task to make all three of those be consistent, this would be a fourth situation.)

I actually agree with @delcypher that RUN(feature): seems cleaner and certainly handles the motivating case very well. But it is obviously less powerful, and I won't object to the %if solution.

[asavonic]

• Added support for binary expressions.
• Added handling for empty commands in RUN lines.
• Added new tests.

[asavonic]

In D122569#3412230, @delcypher wrote:

Have you considered the different approach of having conditional RUN lines instead? E.g.

RUN: echo this always runs
RUN(feature): echo this only runs when feature is true
RUN(!feature): echo this only runs when feature is false

RUN(feature || feature2): echo this only runs where feature or feature2 is true.
RUN(!(feature || feature2)): echo this only runs where feature and feature2 are both false

I think this would make RUN lines easier to read. It doesn't support the "else" case quite as cleanly (you have to duplicate the condition and then negate it on separate line) but I think that's pretty minor because the condition can easily be made a "feature" that is computed from other features (in a test suite's lit.cfg.py)

Both options should work fine for ptxas or any other external tools.
%if .. %else syntax is more generic and it is actually easier to implement in LIT: it is just a bit more complicated substitution, whereas RUN(cond) is completely new syntax that (may) require more work to enable (at least I couldn't figure out how to do this quickly).

In D122569#3414112, @probinson wrote:

I see that for %if feature {A} %else {B} the space before the %else is not preserved in the result.
If there is no %else what happens? e.g., XX %if feature {YY} ZZ Asking because I see no test case for this.

Spaces before %else are ignored only if %else is present. Added this test:

# RUN: echo XX %if feature {YY} ZZ
# RUN: echo AA %if feature {BB} %else {CC} DD
# RUN: echo AA %if nofeature {BB} %else {CC} DD

echo XX YY ZZ
echo AA BB DD
echo AA CC DD

I take it the %if expressions take only lit feature keywords, like REQUIRES? No triple substrings, as allowed by UNSUPPORTED/XFAIL?
(I have a back-burner task to make all three of those be consistent, this would be a fourth situation.)

Yes, identifiers are taken from config.available_features, just like for REQUIRES. The last patch adds binary expression evaluation, e.g. feature1 && !feature2 or regex match {{fea.+}}.

I actually agree with @delcypher that RUN(feature): seems cleaner and certainly handles the motivating case very well. But it is obviously less powerful, and I won't object to the %if solution.

Agree.

llvm/docs/TestingGuide.rst

615

• %if feature {do_something | FileCheck %s} -- this would probably work.

This does not work for verbose output: LIT formats a command into echo "RUN at line #"; ${command}; and if ${command} is an empty string we'll get echo "RUN at line #"; ;. Bash cannot parse the double semicolon at the end.

We can probably handle this as a special case.

Fixed this by changing how this debug print is implemented. Now we don't print an extra semicolon if the command is empty.

llvm/utils/lit/tests/Inputs/shtest-if-else/test.txt

10

Does parsing more complicated expressions work? I would expect it to, so we should probably add tests for it.

This is now implemented in the latest revision of the patch.

llvm/utils/lit/tests/shtest-if-else.py

2–3

Done.

20

Fixed.

[tra]

llvm/utils/lit/tests/shtest-if-else.py

2

I'd use the full name %{inputs}/shtest-if-else/test.txt.
This way it's obvious how the input file is used and it avoids accidentally picking up some other file we may eventually have in that directory.

Perhaps it would make sense to incorporate the test.txt into this file. E.g add another RUN line to generate the test input file and then run lit on it.
This would allow interleaving the RUN/check directives in the same source. Right now it's a bit of a pain to follow what we're exacuting and which output it's supposed to generate.

E.g. we could do something like this:

RUN: grep LITRUN %s | sed -e 's/LITRUN/RUN/' > %t/test.txt
RUN: grep LITCFG %s | sed -e 's/# LITCFG://' > %t/lit.cfg
RUN: %{lit} -v --show-all %t.txt | FileCheck %s

# LITCFG: lit config lines go here.

# LITRUN: echo XX %if feature {YY} ZZ
# CHECK-NEXT: 'RUN: at line 32'; echo XX YY ZZ
# LITRUN: ...
# CHECK-NEXT: ...

[asavonic]

• Removed extra line breaks at EOF.

[asavonic]

llvm/utils/lit/tests/shtest-if-else.py

2

RUN: grep LITRUN %s | sed -e 's/LITRUN/RUN/' > %t/test.txt
RUN: grep LITCFG %s | sed -e 's/# LITCFG://' > %t/lit.cfg

I wonder if grep and sed are available (and required) for windows builds. Probably not.
In any case, I don't like this separation between inputs and checks either, so I'll try to figure out the way to combine them in one file.

[probinson]

llvm/utils/lit/tests/shtest-if-else.py

2

You probably want split-file which is an LLVM-built tool.

[tra]

llvm/utils/lit/tests/shtest-if-else.py

2

We can also use %python and run this test to write out the files we need.

[asavonic]

llvm/utils/lit/tests/shtest-if-else.py

2

We can easily combine CHECK and RUN lines if we "escape" RUN: in the patterns by using a regex. I decided to keep the lit.cfgfile separate to keep things simple. Otherwise we'll have to deal with test discovery and split-file.

[asavonic]

• Combined RUN and CHECK lines into a single file.

[tra]

llvm/utils/lit/tests/shtest-if-else.py

2

Nice. It's so much easier to follow.

[tra]

LGTM as a potential end user of this feature.
Please wait for OK from one of the LIT owners.

[jdenny]

@asavonic: Thanks for working on this. I've pointed out a few issues that need to be discussed before this lands. I also need to make another pass before I'm ready.

llvm/utils/lit/lit/TestRunner.py

54

It appears this change and related changes below make it possible to have blank RUN: lines in general, even when not using %if. That's probably worth mentioning in the review summary.

Also, the summary misspells %else as else a few times. Please fix.

1308

This processes %if and %else after other substitutions. That means a test suite's config could (probably unwittingly) interfere with lit's normal %if and %else processing by defining its own %if and %else substitutions.

I think lit shouldn't permit that. One way to prevent that is to process %if and %else before other substitutions. You'd also need to move the escape(ln) call before that.

1320

Is requiring a space before each { and after } necessary? If omitted, the whole if-else won't be unrecognized. I think that would be confusing.

Eventually, it would be nice to parse this properly: match %if, then search for the condition and complain if it's not there, then search for the { and complain if it's not there, etc.

llvm/utils/lit/tests/Inputs/shtest-if-else/test.txt

8

Please use [[#@LINE-1]] instead of 6 (likewise below) so that we can remove and add checks without renumbering throughout the rest of the file.

52

This case happens to work, but nested expressions do not work in general. It needs a proper parser with a stack to handle that.

The regex for the outer %if matches until the inner } because of the minimal munch .*? in the regex. That leaves behind the outer } for the inner %if. Lucky.

But what happens in the case of an outer nofeature? The left-behind } results in echo }. Also consider cases with inner and outer %else.

Non-nested %if cases involving braces don't work either:

%if have_sed { sed 's/<a\{2,3\}>/<FOUND>/g' %s | FileCheck %s }

By the way, it might be worthwhile to add {{$}} to the end of the FileCheck patterns in this file to catch any portion of the RUN unexpectedly left behind. FileCheck's --match-full-lines plus {{^.*}} at the start of FileCheck patterns might be even better.

[asavonic]

• Reimplemented %if..%else parser to support nested expressions.
• Added --match-full-lines to the LIT test; added more test cases.

llvm/utils/lit/lit/TestRunner.py

54

Thanks! Fixed the review summary.

1308

Thank you, that makes sense. Moved %if..%else processing before other substitutions.

1320

Right, spaces are no longer required in the last revision of the patch.

llvm/utils/lit/tests/Inputs/shtest-if-else/test.txt

8

Done.

52

Right, I was hoping that we can live with a regex parser for now, but with all these issues it is unacceptable. I've re-implemented the parser and now it should support all these cases well enough. We also no longer need recursiveExpansionLimit flag to be set for nested %if..%else expressions.

[MaskRay]

llvm/utils/lit/lit/TestRunner.py

1302

It's conventional to use tuples instead of lists to represent immutable objects (stack elements here).

[asavonic]

llvm/utils/lit/lit/TestRunner.py

1302

Expression lists are mutable. When we pop the stack, all items are evaluated and the resulting value is appended to the last expression on the previous stack level. For example:

foo-%if feature { bar }
                      ^~~ cursor
stack: [ ['foo-', [token_if, 'feature']],
         [' bar '] ]

When we reach the closing brace, 'bar' is poped out and appended to the 'if' expression:

stack: [ ['foo-', [token_if, 'feature', ' bar ']] ]

[lebedev.ri]

Please don't forget to also update the llvm/utils/update_*_checks.py.

[asavonic]

In D122569#3441717, @lebedev.ri wrote:

Please don't forget to also update the llvm/utils/update_*_checks.py.

It seems that update_*_checks.py scripts are completely independent of lit.py, so there is no way to easily reuse code between them. Parsing of RUN lines and standard substitutions is duplicated there. We have to either integrate these scripts, or copy paste the code.

[jdenny]

@asavonic: Thanks for continuing to work on this feature!

In D122569#3442909, @asavonic wrote:

In D122569#3441717, @lebedev.ri wrote:

Please don't forget to also update the llvm/utils/update_*_checks.py.

It seems that update_*_checks.py scripts are completely independent of lit.py, so there is no way to easily reuse code between them. Parsing of RUN lines and standard substitutions is duplicated there. We have to either integrate these scripts, or copy paste the code.

Has anyone said they need %if in those scripts? Upon seeing %if in a RUN line, those scripts could probably be adjusted to simply complain that %if is not yet supported there.

llvm/utils/lit/lit/TestRunner.py

1213–1214

Can we make the name more specific now that there's also an escapeBraces? Maybe escapePercents? That might be better even if escapeBraces goes away while addressing other issues.

1275

1307

Instead of a loop with a custom stack data structure, a recursive descent parser might be easier to understand and extend, especially if we add more constructs similar to %if later. That is, the python call stack would be used as the parser stack, and stack entries would be simple local variables, function parameters, and function returns.

This is just a suggestion. I'm not insisting. However, I do invite other reviewers to offer their opinion.

1318

Why does %% need to be escaped again here?

1347

llvm/utils/lit/tests/Inputs/shtest-if-else/test.txt

76

If we're going to have these %{ and %} escapes, they need to be documented somewhere. However, I think they're going to be confusing:

• First, %{foobar} is already a common syntax for lit substitutions. The if-then-else syntax's %{ and } collide with it. For example, how should I write %{pathsep} in a then or else block? I have to write it as %%{pathsep%}, right?
• More generally, whether { and } need to be escaped and whether %{ and %} are defined depend on the context. That seems error-prone.

What if we go the other way? That is, we could change the { and } around a then or else block to %{ and %}. That way, the entire if-then-else syntax is clearly at the preprocessor/substitution level: %if, %else, %{, and %}. On the other hand, { and } behave as usual in the shell code, and the only character in shell code that ever needs to be escaped continues to be %.

But does that syntax have collisions?

• I just tried git grep 'RUN.*%}' and found no matches in llvm's repo. It seems %} is unambiguous. If a use arises, it can be escaped as %%}. Again, only % ever has to be escaped.
• The %{ is really just for symmetry with %}. The %{ doesn't actually need to be recognized in arbitrary shell code anywhere. That is, the if-then-else parser would only need to recognize it after an %if cond or %else. There doesn't appear to be a way for it to collide with, for example, %{pathsep}.

Did I miss anything?

[arichardson]

In D122569#3442909, @asavonic wrote:

In D122569#3441717, @lebedev.ri wrote:

Please don't forget to also update the llvm/utils/update_*_checks.py.

It seems that update_*_checks.py scripts are completely independent of lit.py, so there is no way to easily reuse code between them. Parsing of RUN lines and standard substitutions is duplicated there. We have to either integrate these scripts, or copy paste the code.

I had some code in the downstream CHERI LLVM to expand substitutions using lit. The downside is that the update scripts need to know the path to the build dir to get all the lit config files, but if there is interest I'd be happy to upstream that change.

[asavonic]

• Reimplemented the parser as a recursive function
• Changed syntax: %if cond { branch_if } %else { branch_else } to %if cond %{ branch_if %} %else %{ branch_else %}

[asavonic]

In D122569#3446867, @arichardson wrote:

In D122569#3442909, @asavonic wrote:

In D122569#3441717, @lebedev.ri wrote:

Please don't forget to also update the llvm/utils/update_*_checks.py.

It seems that update_*_checks.py scripts are completely independent of lit.py, so there is no way to easily reuse code between them. Parsing of RUN lines and standard substitutions is duplicated there. We have to either integrate these scripts, or copy paste the code.

I had some code in the downstream CHERI LLVM to expand substitutions using lit. The downside is that the update scripts need to know the path to the build dir to get all the lit config files, but if there is interest I'd be happy to upstream that change.

@arichardson, I think it makes sense as an optional feature. Right now there is no way to handle substitutions defined in lit.cfg or share the code with lit.py.

llvm/utils/lit/lit/TestRunner.py

1213–1214

Done. escape is now renamed to escapePercents.

1307

Done. It does look better as a recursive function.

1318

This is because of recursive substitution:

substitutions = [
  ("%rec1, "%%s"),
  ("%s", "/path")
]

If we expand %rec1 to %%s in the first iteration of the loop, we have to escape %% before the next iteration. Otherwise instead of of %s we'll get %/path. This is tested in shtest-recursive-substitution/escaping test.

llvm/utils/lit/tests/Inputs/shtest-if-else/test.txt

76

Done. The patch now uses %{ and %} as begin/end tokens. It also helps to distinguish them from boolean expressions with a regex-match ({{regex}}).

[arichardson]

While this is not a huge amount of code to be added to lit, I do wonder if we really need it. Wouldn't something like %feature{foo} that expands to 0 or 1 be sufficient? You could then use shell conditionals/test to avoid running lines.
Looking at the examples from D121727, you could do the following:

RUN: if [ %feature{ptxas} -ne 0 ]; then ptxas -c %t-nvptx.ptx -o /dev/null; fi
RUN: test %feature{ptxas} -ne 0 && ptxas -c %t-nvptx.ptx -o /dev/null

Or alternatively if %feature expands to an empty string ("") or "1":

RUN: if [ -n %feature{ptxas} ]; then ptxas -c %t-nvptx.ptx -o /dev/null; fi
RUN: test -n %feature{ptxas} && ptxas -c %t-nvptx.ptx -o /dev/null

Depending on how you set up the ptxas substitution it might even be possible without a %feature{} lit substitution.

[asavonic]

In D122569#3458476, @arichardson wrote:

While this is not a huge amount of code to be added to lit, I do wonder if we really need it. Wouldn't something like %feature{foo} that expands to 0 or 1 be sufficient? You could then use shell conditionals/test to avoid running lines.
Looking at the examples from D121727, you could do the following:

RUN: if [ %feature{ptxas} -ne 0 ]; then ptxas -c %t-nvptx.ptx -o /dev/null; fi
RUN: test %feature{ptxas} -ne 0 && ptxas -c %t-nvptx.ptx -o /dev/null

The variant with if is probably not going to work on windows.
As for the test - I'm not sure how it works in this case. If test returns a non-zero exit code, then LIT should report an error for the whole RUN line, right?
It is also not clear if we can use more complicated binary expression like test 1 -eq 1 && ( test 2 -eq 2 || test 3 -eq 3 ).
For ptxas we also need to support nested expressions:

; The -arch option is only needed for ptxas < 11.0
; RUN: %if ptxas %{ %ptxas-verify %if !ptxas-11.0 %{-arch=sm_30%} %t.ptx %}

[jdenny]

In D122569#3458804, @asavonic wrote:

In D122569#3458476, @arichardson wrote:

While this is not a huge amount of code to be added to lit, I do wonder if we really need it. Wouldn't something like %feature{foo} that expands to 0 or 1 be sufficient? You could then use shell conditionals/test to avoid running lines.
Looking at the examples from D121727, you could do the following:

RUN: if [ %feature{ptxas} -ne 0 ]; then ptxas -c %t-nvptx.ptx -o /dev/null; fi
RUN: test %feature{ptxas} -ne 0 && ptxas -c %t-nvptx.ptx -o /dev/null

The variant with if is probably not going to work on windows.

Agreed. It won't work in lit's internal shell currently either. %if is a portable if. Maybe it would be good to point this out in the documentation?

As for the test - I'm not sure how it works in this case.

My hunch is that test is also not portable.

llvm/utils/lit/lit/TestRunner.py

1235

It would be nice to extend the test suite to check that these syntax errors are actually detected.

1307

Nice. Thanks.

1318

Make sense. Thanks for explaining.

llvm/utils/lit/tests/Inputs/shtest-if-else/test.txt

22

Maybe test the preservation of space in %else as well.

76

Thanks. Please add a test that a substitution like %{foo} works in the then and else blocks. You'll probably have to define your own as I'm not sure it's safe to rely on the way the built-in substitutions expand.

[jdenny]

I've made a few requests for minor test and doc extensions, but otherwise this LGTM.

Given the recent syntax changes, please give other reviewers a couple more days to comment before landing this.

Thanks for all your work.

[asavonic]

• Added negative tests for parsing errors.
• Added tests for spaces in an %else block, tests with %{sub} substitution.

[asavonic]

In D122569#3460461, @jdenny wrote:

I've made a few requests for minor test and doc extensions, but otherwise this LGTM.

Given the recent syntax changes, please give other reviewers a couple more days to comment before landing this.

Thanks for all your work.

Thanks a lot! We can wait at least until the next week. D121727 is not ready anyway.

llvm/docs/TestingGuide.rst

based on the host platform.
 
    The following error codes are currently supported:
    ENOENT, EISDIR, EINVAL, EACCES.
 
    Example: ``Linux %errc_ENOENT: No such file or directory``
 
    Example: ``Windows %errc_ENOENT: no such file or directory``
 
+``%if feature %{<if branch>%} %else %{<else branch>%}``
+
+ Conditional substitution: if ``feature`` is available it expands to
+ ``<if branch>``, otherwise it expands to ``<else branch>``.
+ ``%else %{<else branch>%}`` is optional and treated like ``%else %{%}``
+ if not present.
+
 **LLVM-specific substitutions:**
 
 ``%shlibext``
    The suffix for the host platforms shared library files. This includes the
    period as the first character.
 
    Example: ``.so`` (Linux), ``.dylib`` (macOS), ``.dll`` (Windows)
 

llvm/utils/lit/lit/TestRunner.py

 # run command pipeline such that ARG specifies the pipeline's source line
 # number.  lit later expands each %dbg(ARG) to a command that behaves as a null
 # command in the target shell so that the line number is seen in lit's verbose
 # mode.
 #
 # This regex captures ARG.  ARG must not contain a right parenthesis, which
 # terminates %dbg.  ARG must not contain quotes, in which ARG might be enclosed
 # during expansion.
-kPdbgRegex = '%dbg\\(([^)\'"]*)\\)'
+#
+# COMMAND that follows %dbg(ARG) is also captured. COMMAND can be
+# empty as a result of conditinal substitution.
+kPdbgRegex = '%dbg\\(([^)\'"]*)\\)(.*)'
 
 class ShellEnvironment(object):
 
     """Mutable shell environment containing things like CWD and env vars.
 
     Environment variables are not implemented, but cwd tracking is.
     """
 
def _executeShCmd(cmd, shenv, results, timeoutHelper):
     if cmd.negate:
         exitCode = not exitCode
 
     return exitCode
 
 def executeScriptInternal(test, litConfig, tmpBase, commands, cwd):
     cmds = []
     for i, ln in enumerate(commands):
-        ln = commands[i] = re.sub(kPdbgRegex, ": '\\1'; ", ln)
+        match = re.match(kPdbgRegex, ln)
+        if match:
+            command = match.group(2)
+            ln = commands[i] = \
+                match.expand(": '\\1'; \\2" if command else ": '\\1'")
         try:
             cmds.append(ShUtil.ShParser(ln, litConfig.isWindows,
                                         test.config.pipefail).parse())
         except:
             return lit.Test.Result(Test.FAIL, "shell parser error on: %r" % ln)
 
     cmd = cmds[0]
     for c in cmds[1:]:
def executeScript(test, litConfig, tmpBase, commands, cwd):
     open_kwargs = {}
     if litConfig.isWindows and not isWin32CMDEXE:
         mode += 'b'  # Avoid CRLFs when writing bash scripts.
     elif sys.version_info > (3,0):
         open_kwargs['encoding'] = 'utf-8'
     f = open(script, mode, **open_kwargs)
     if isWin32CMDEXE:
         for i, ln in enumerate(commands):
-            commands[i] = re.sub(kPdbgRegex, "echo '\\1' > nul && ", ln)
+            match = re.match(kPdbgRegex, ln)
+            if match:
+                command = match.group(2)
+                commands[i] = \
+                    match.expand("echo '\\1' > nul && " if command
+                                 else "echo '\\1' > nul")
         if litConfig.echo_all_commands:
             f.write('@echo on\n')
         else:
             f.write('@echo off\n')
         f.write('\n@if %ERRORLEVEL% NEQ 0 EXIT\n'.join(commands))
     else:
         for i, ln in enumerate(commands):
-            commands[i] = re.sub(kPdbgRegex, ": '\\1'; ", ln)
+            match = re.match(kPdbgRegex, ln)
+            if match:
+                command = match.group(2)
+                commands[i] = match.expand(": '\\1'; \\2" if command
+                                           else ": '\\1'")
         if test.config.pipefail:
             f.write(b'set -o pipefail;' if mode == 'wb' else 'set -o pipefail;')
         if litConfig.echo_all_commands:
             f.write(b'set -x;' if mode == 'wb' else 'set -x;')
         if sys.version_info > (3,0) and mode == 'wb':
             f.write(bytes('{ ' + '; } &&\n{ '.join(commands) + '; }', 'utf-8'))
         else:
             f.write('{ ' + '; } &&\n{ '.join(commands) + '; }')
def memoized(x):
             cache[x] = f(x)
         return cache[x]
     return memoized
 
 @_memoize
 def _caching_re_compile(r):
     return re.compile(r)
 
-def applySubstitutions(script, substitutions, recursion_limit=None):
+def applySubstitutions(script, substitutions, conditions={},
+                       recursion_limit=None):
     """
     Apply substitutions to the script.  Allow full regular expression syntax.
     Replace each matching occurrence of regular expression pattern a with
     substitution b in line ln.
 
     If a substitution expands into another substitution, it is expanded
     recursively until the line has no more expandable substitutions. If
     the line can still can be substituted after being substituted
     `recursion_limit` times, it is an error. If the `recursion_limit` is
     `None` (the default), no recursive substitution is performed at all.
     """
 
     # We use #_MARKER_# to hide %% while we do the other substitutions.
-    def escape(ln):
+    def escapePercents(ln):
         return _caching_re_compile('%%').sub('#_MARKER_#', ln)
 
-    def unescape(ln):
+    def unescapePercents(ln):
         return _caching_re_compile('#_MARKER_#').sub('%', ln)
 
+    def substituteIfElse(ln):
+        # early exit to avoid wasting time on lines without
+        # conditional substitutions
+        if ln.find('%if ') == -1:
+            return ln
+
+        def tryParseIfCond(ln):
+            # space is important to not conflict with other (possible)
+            # substitutions
+            if not ln.startswith('%if '):
+                return None, ln
+            ln = ln[4:]
+
+            # stop at '%{'
+            match = _caching_re_compile('%{').search(ln)
+            if not match:
+                raise ValueError("'%{' is missing for %if substitution")
+            cond = ln[:match.start()]
+
+            # eat '%{' as well
+            ln = ln[match.end():]
+            return cond, ln
+
+        def tryParseElse(ln):
+            match = _caching_re_compile('^\s*%else\s*%{').search(ln)
+            if not match:
+                return False, ln
+            return True, ln[match.end():]
+
+        def tryParseEnd(ln):
+            if ln.startswith('%}'):
+                return True, ln[2:]
+            return False, ln
+
+        def parseText(ln, isNested):
+            # parse everything until %if, or %} if we're parsing a
+            # nested expression.
+            match = _caching_re_compile(
+                '(.*?)(?:%if|%})' if isNested else '(.*?)(?:%if)').search(ln)
+            if not match:
+                # there is no terminating pattern, so treat the whole
+                # line as text
+                return ln, ''
+            text_end = match.end(1)
+            return ln[:text_end], ln[text_end:]
+
+        def parseRecursive(ln, isNested):
+            result = ''
+            while len(ln):
+                if isNested:
+                    found_end, _ = tryParseEnd(ln)
+                    if found_end:
+                        break
+
+                # %if cond %{ branch_if %} %else %{ branch_else %}
+                cond, ln = tryParseIfCond(ln)
+                if cond:
+                    branch_if, ln = parseRecursive(ln, isNested=True)
+                    found_end, ln = tryParseEnd(ln)
+                    if not found_end:
+                        raise ValueError("'%}' is missing for %if branch")
+
+                    branch_else = ''
+                    found_else, ln = tryParseElse(ln)
+                    if found_else:
+                        branch_else, ln = parseRecursive(ln, isNested=True)
+                        found_end, ln = tryParseEnd(ln)
+                        if not found_end:
+                            raise ValueError("'%}' is missing for %else branch")
+
+                    if BooleanExpression.evaluate(cond, conditions):
+                        result += branch_if
+                    else:
+                        result += branch_else
+                    continue
+
+                # The rest is handled as plain text.
+                text, ln = parseText(ln, isNested)
+                result += text
+
+            return result, ln
+
+        result, ln = parseRecursive(ln, isNested=False)
+        assert len(ln) == 0
+        return result
+
     def processLine(ln):
         # Apply substitutions
+        ln = substituteIfElse(escapePercents(ln))
         for a,b in substitutions:
             if kIsWindows:
                 b = b.replace("\\","\\\\")
             # re.compile() has a built-in LRU cache with 512 entries. In some
             # test suites lit ends up thrashing that cache, which made e.g.
             # check-llvm run 50% slower.  Use an explicit, unbounded cache
             # to prevent that from happening.  Since lit is fairly
             # short-lived, since the set of substitutions is fairly small, and
             # since thrashing has such bad consequences, not bounding the cache
             # seems reasonable.
-            ln = _caching_re_compile(a).sub(str(b), escape(ln))
+            ln = _caching_re_compile(a).sub(str(b), escapePercents(ln))
 
         # Strip the trailing newline and any extra whitespace.
         return ln.strip()
 
     def processLineToFixedPoint(ln):
         assert isinstance(recursion_limit, int) and recursion_limit >= 0
         origLine = ln
         steps = 0
         processed = processLine(ln)
         while processed != ln and steps < recursion_limit:
             ln = processed
             processed = processLine(ln)
             steps += 1
 
         if processed != ln:
             raise ValueError("Recursive substitution of '%s' did not complete "
                              "in the provided recursion limit (%s)" % \
                              (origLine, recursion_limit))
 
         return processed
 
     process = processLine if recursion_limit is None else processLineToFixedPoint
     
-    return [unescape(process(ln)) for ln in script]
+    return [unescapePercents(process(ln)) for ln in script]
 
 
 class ParserKind(object):
     """
     An enumeration representing the style of an integrated test keyword or
     command.
 
     TAG: A keyword taking no value. Ex 'END.'
     COMMAND: A keyword taking a list of shell commands. Ex 'RUN:'
     LIST: A keyword taking a comma-separated list of values.
     BOOLEAN_EXPR: A keyword taking a comma-separated list of
         boolean expressions. Ex 'XFAIL:'
     INTEGER: A keyword taking a single integer. Ex 'ALLOW_RETRIES:'
def executeShTest(test, litConfig, useExternalSh,
 
     if litConfig.noExecute:
         return lit.Test.Result(Test.PASS)
 
     tmpDir, tmpBase = getTempPaths(test)
     substitutions = list(extra_substitutions)
     substitutions += getDefaultSubstitutions(test, tmpDir, tmpBase,
                                              normalize_slashes=useExternalSh)
-    script = applySubstitutions(script, substitutions,
+    conditions = { feature: True for feature in test.config.available_features }
+    script = applySubstitutions(script, substitutions, conditions,
                                 recursion_limit=test.config.recursiveExpansionLimit)
 
     return _runShTest(test, litConfig, useExternalSh, script, tmpBase)

llvm/utils/lit/tests/Inputs/shtest-if-else/lit.cfg

+import lit.formats
+config.name = 'shtest-if-else'
+config.test_format = lit.formats.ShTest()
+config.test_source_root = None
+config.test_exec_root = None
+config.suffixes = ['.txt']
+config.available_features.add("feature")

llvm/utils/lit/tests/Inputs/shtest-if-else/test.txt

+# CHECK: -- Testing:{{.*}}
+# CHECK-NEXT: PASS: shtest-if-else :: test.txt (1 of 1)
+# CHECK-NEXT: Script:
+# CHECK-NEXT: --
+
+# RUN: %if feature %{ echo "test-1" %}
+# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-1]]'; echo "test-1"
+
+# If %else is not present it is treated like %else %{%}. Empty commands
+# are ignored.
+#
+# RUN: %if nofeature %{ echo "fail" %}
+# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-1]]'
+# CHECK-NOT: fail
+
+# RUN: %if nofeature %{ echo "fail" %} %else %{ echo "test-2" %}
+# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-1]]'; echo "test-2"
+
+# Spaces inside curly braces are not ignored
+#
+# RUN: echo test-%if feature %{ 3 %} %else %{ fail %}-test
+# RUN: echo test-%if feature %{ 4 4 %} %else %{ fail %}-test
+# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-2]]'; echo test- 3 -test
+# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-2]]'; echo test- 4 4 -test
+
+# Escape line breaks for multi-line expressions
+#
+# RUN: %if feature  \
+# RUN:   %{ echo     \
+# RUN:     "test-5" \
+# RUN:   %}
+# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-4]]'; echo "test-5"
+
+# RUN: %if nofeature       \
+# RUN:   %{ echo "fail" %}   \
+# RUN: %else               \
+# RUN:   %{ echo "test-6" %}
+# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-4]]'; echo "test-6"
+
+# RUN: echo "test%if feature %{%} %else %{%}-7"
+# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-1]]'; echo "test-7"
+
+# Escape %if. Without %if..%else context '%{' and '%}' are treated
+# literally.
+#
+# RUN: echo %%if feature %{ echo "test-8" %}
+# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-1]]'; echo %if feature %{ echo "test-8" %}
+
+# Nested expressions are supported:
+#
+# RUN: echo %if feature %{ %if feature %{ %if nofeature %{"fail"%} %else %{"test-9"%} %} %}
+# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-1]]'; echo "test-9"
+
+# Binary expression evaluation and regex match can be used as
+# conditions.
+#
+# RUN: echo %if feature && !nofeature %{ "test-10" %}
+# RUN: echo %if feature && nofeature %{ "fail" %} %else %{ "test-11" %}
+# RUN: echo %if {{fea.+}} %{ "test-12" %} %else %{ "fail" %}
+# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-3]]'; echo "test-10"
+# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-3]]'; echo "test-11"
+# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-3]]'; echo "test-12"
+
+# Spaces between %if and %else are ignored. If there is no %else -
+# space after %if %{...%} is not ignored.
+#
+# RUN: echo XX %if feature %{YY%} ZZ
+# RUN: echo AA %if feature %{BB%} %else %{CC%} DD
+# RUN: echo AA %if nofeature %{BB%} %else %{CC%} DD
+# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-3]]'; echo XX YY ZZ
+# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-3]]'; echo AA BB DD
+# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-3]]'; echo AA CC DD
+
+# '{' and '}' can be used without escaping
+#
+# RUN: %if feature %{echo {}%}
+# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-1]]'; echo {}
+
+# Spaces are not required
+#
+# RUN: echo %if feature%{"ok"%}%else%{"fail"%}
+# CHECK-NEXT: {{^.*'RUN}}: at line [[#@LINE-1]]'; echo "ok"
+
+# CHECK-NEXT: --
+# CHECK-NEXT: Exit Code: 0

llvm/utils/lit/tests/shtest-if-else.py

+# RUN: %{lit} -v --show-all %{inputs}/shtest-if-else/test.txt | FileCheck %{inputs}/shtest-if-else/test.txt --match-full-lines