[lit] Add --xfail and --filter-out (inverse of --filter)

[davezarzycki]

Summary

In semi-automated environments, XFAILing or filtering out known regressions without actually committing changes or temporarily modifying the test suite can be quite useful.

[davezarzycki]

Add docs not included in the first diff.

[jhenderson]

This definitely seems like something we could use. I haven't looked at the implementation yet, but just thinking about the feature - would it be better to have a flag that marks the test as XFAIL instead? This would mean that once the issue is fixed, the continued use of the switch would cause an XPASS, indicating the switch could be dropped. What do you think?

[thopre]

In D96662#2563019, @jhenderson wrote:

This definitely seems like something we could use. I haven't looked at the implementation yet, but just thinking about the feature - would it be better to have a flag that marks the test as XFAIL instead? This would mean that once the issue is fixed, the continued use of the switch would cause an XPASS, indicating the switch could be dropped. What do you think?

On the other hand, someone might use LIT_SKIP to skip tests that take too long time to run only on slow machine (others would have no LIT_SKIP). Marking the tests as XFAIL would prevent to skip passing tests.

[jhenderson]

In D96662#2563106, @thopre wrote:

In D96662#2563019, @jhenderson wrote:

This definitely seems like something we could use. I haven't looked at the implementation yet, but just thinking about the feature - would it be better to have a flag that marks the test as XFAIL instead? This would mean that once the issue is fixed, the continued use of the switch would cause an XPASS, indicating the switch could be dropped. What do you think?

On the other hand, someone might use LIT_SKIP to skip tests that take too long time to run only on slow machine (others would have no LIT_SKIP). Marking the tests as XFAIL would prevent to skip passing tests.

The specified use-case in the description is about regressions, which I assume mean test failures, not slower tests.

That being said, if there's a requested use-case for actually skipping tests, due to resource limitations of the machine, maybe a complement to my suggestion would be to have an "UNSUPPORTED" equivalent.

[davezarzycki]

I agree that both --skip (a.k.a. LIT_SKIP) and --xfail (a.k.a. LIT_XFAIL) are independently useful so I've updated the patch.

[jdenny]

llvm/docs/CommandGuide/lit.rst

220

Why do we need more environment variables? Isn't LIT_OPTS sufficient? (LIT_FILTER, etc. predate LIT_OPTS.)

[davezarzycki]

llvm/docs/CommandGuide/lit.rst

220

I'm generally biased against designs that require thinking about shell escaping (or at least more than necessary).

[davezarzycki]

If I might critique myself, I'm not sure that regular expression semantics are best for --xfail. One really wants and should use a precise list of some kind. Any thoughts on what that might look like? I'd lean toward a semicolon separated list.

[davezarzycki]

After some thought, I think a list is better than a regex for --xfail. In contrast, having --skip take a regex makes sense because it is the opposite of --filter (like grep -v and grep respectively).

[jdenny]

llvm/docs/CommandGuide/lit.rst

220

I've been using --filter within LIT_OPTS for a while because I use LIT_OPTS for other purposes anyway. I don't find the escaping to be challenging. However, maybe quoting is harder in some shells, or maybe I've not imagined some use cases. OK, fair enough.

[yln]

I can get behind --skip (patch LGTM), but would a little help to understand when --xfail=path/to/test1;path/to/test2 would be useful (which workflow?). Thanks!

[jhenderson]

In D96662#2566903, @yln wrote:

I can get behind --skip (patch LGTM), but would a little help to understand when --xfail=path/to/test1;path/to/test2 would be useful (which workflow?). Thanks!

One example where it would be useful - in our downstream merge process, we may identify a test that has started failing due to an upstream bug. We could modify the test explicitly by adding XFAIL to the start of it, or we could add it to the build script that is used to build and test the changes before they are merged in. The --xfail option allows the latter to happen in such a way that when the issue is fixed by a later upstream change, we'll notice (due to the test starting to pass) and remove the option from the script, whereas with --skip, we won't notice (and thus lose the testing downstream the test gave us). I could imagine other cases along these lines too, where --xfail is superior to --skip.

In D96662#2565257, @davezarzycki wrote:

If I might critique myself, I'm not sure that regular expression semantics are best for --xfail. One really wants and should use a precise list of some kind. Any thoughts on what that might look like? I'd lean toward a semicolon separated list.

A precise list would make sense. I'd use commas rather than semicolons, because semicolons are more likely to require shell escaping, I think (commas are less likely to).

[thopre]

In D96662#2567744, @jhenderson wrote:

In D96662#2566903, @yln wrote:

I can get behind --skip (patch LGTM), but would a little help to understand when --xfail=path/to/test1;path/to/test2 would be useful (which workflow?). Thanks!

One example where it would be useful - in our downstream merge process, we may identify a test that has started failing due to an upstream bug. We could modify the test explicitly by adding XFAIL to the start of it, or we could add it to the build script that is used to build and test the changes before they are merged in. The --xfail option allows the latter to happen in such a way that when the issue is fixed by a later upstream change, we'll notice (due to the test starting to pass) and remove the option from the script, whereas with --skip, we won't notice (and thus lose the testing downstream the test gave us). I could imagine other cases along these lines too, where --xfail is superior to --skip.

I was thinking this could be useful to us for the same reason. Some tests fail because they make assumptions about default target or host. We send patch upstream for those but while they are in review and until we merge again, we tend to add xfail in the test. Sometimes the failure is due to the target being different in which case we just mark it unsupported. --skip and --xfail would be quite useful.

[davezarzycki]

In D96662#2567744, @jhenderson wrote:

In D96662#2566903, @yln wrote:

I can get behind --skip (patch LGTM), but would a little help to understand when --xfail=path/to/test1;path/to/test2 would be useful (which workflow?). Thanks!

One example where it would be useful - in our downstream merge process, we may identify a test that has started failing due to an upstream bug. We could modify the test explicitly by adding XFAIL to the start of it, or we could add it to the build script that is used to build and test the changes before they are merged in. The --xfail option allows the latter to happen in such a way that when the issue is fixed by a later upstream change, we'll notice (due to the test starting to pass) and remove the option from the script, whereas with --skip, we won't notice (and thus lose the testing downstream the test gave us). I could imagine other cases along these lines too, where --xfail is superior to --skip.

Indeed, this is the problem I'm trying to solve.

In D96662#2565257, @davezarzycki wrote:

If I might critique myself, I'm not sure that regular expression semantics are best for --xfail. One really wants and should use a precise list of some kind. Any thoughts on what that might look like? I'd lean toward a semicolon separated list.

A precise list would make sense. I'd use commas rather than semicolons, because semicolons are more likely to require shell escaping, I think (commas are less likely to).

In my experience, quotes solve most "escaping" problems, so what is used as a separator mostly doesn't matter as long as the character isn't used for variable expansion. Using the shell's statement separator is a good choice because shell users/programmers naturally avoid files with that character embedded in the name.

[jhenderson]

In D96662#2565257, @davezarzycki wrote:

If I might critique myself, I'm not sure that regular expression semantics are best for --xfail. One really wants and should use a precise list of some kind. Any thoughts on what that might look like? I'd lean toward a semicolon separated list.

A precise list would make sense. I'd use commas rather than semicolons, because semicolons are more likely to require shell escaping, I think (commas are less likely to).

In my experience, quotes solve most "escaping" problems, so what is used as a separator mostly doesn't matter as long as the character isn't used for variable expansion. Using the shell's statement separator is a good choice because shell users/programmers naturally avoid files with that character embedded in the name.

Actually, maybe we should just use multiple --xfail options, rather than a single one that takes a list. That way, the individual file names people use isn't relevant.

[davezarzycki]

In D96662#2568128, @jhenderson wrote:

In D96662#2565257, @davezarzycki wrote:

If I might critique myself, I'm not sure that regular expression semantics are best for --xfail. One really wants and should use a precise list of some kind. Any thoughts on what that might look like? I'd lean toward a semicolon separated list.

A precise list would make sense. I'd use commas rather than semicolons, because semicolons are more likely to require shell escaping, I think (commas are less likely to).

In my experience, quotes solve most "escaping" problems, so what is used as a separator mostly doesn't matter as long as the character isn't used for variable expansion. Using the shell's statement separator is a good choice because shell users/programmers naturally avoid files with that character embedded in the name.

Actually, maybe we should just use multiple --xfail options, rather than a single one that takes a list. That way, the individual file names people use isn't relevant.

We can do that, but that won't work for LIT_XFAIL, so one would be forced to use LIT_OPTS and now we've come full circle back to the shell escaping problem.

[davezarzycki]

In D96662#2568191, @davezarzycki wrote:

In D96662#2568128, @jhenderson wrote:

In D96662#2565257, @davezarzycki wrote:

If I might critique myself, I'm not sure that regular expression semantics are best for --xfail. One really wants and should use a precise list of some kind. Any thoughts on what that might look like? I'd lean toward a semicolon separated list.

A precise list would make sense. I'd use commas rather than semicolons, because semicolons are more likely to require shell escaping, I think (commas are less likely to).

In my experience, quotes solve most "escaping" problems, so what is used as a separator mostly doesn't matter as long as the character isn't used for variable expansion. Using the shell's statement separator is a good choice because shell users/programmers naturally avoid files with that character embedded in the name.

Actually, maybe we should just use multiple --xfail options, rather than a single one that takes a list. That way, the individual file names people use isn't relevant.

We can do that, but that won't work for LIT_XFAIL, so one would be forced to use LIT_OPTS and now we've come full circle back to the shell escaping problem.

Of course, after I sent this, I thought of a better way to articulate this. The problem with shell escapes is nesting and getting expansions to happen at the right level. This is why I don't care for LIT_OPTS because that's already one layer of escaping, and if a lit command line option needs further escaping, that's really annoying (for people that know better) and a common source of errors (for people that don't know better, which is to say, most people). This typically manifests when one tries to reason about white space in file names, but other escaping problems can cause nesting challenges.

[davezarzycki]

Ping. Is anybody willing to sign off on this? There is clearly interest by those that maintain downstream repos to have something like --xfail and/or LIT_XFAIL. (I'm partial to the latter due to how my CI works.)

[yln]

--skip: is it okay if we rename this to --filter-not? It's an uglier name but

• more "self explanatory", i..e, "opposite of --filter". Anyone who knows --filter will know what it does just by looking the arg name.
• Discoverable: in the usage and help output, it will appear next to --filter.
• In the summary, we have a category of Skipped tests: tests that were not run because the user hit [Ctrl+C] or because of overall timeout. --skipped tests would appear in the Excluded category (e.g., --filter and sharding feature). Not using the word skip may help prevent some confusion.

Can we also add the summary counts in the test, so that the test shows that --filter-not excluded tests are counted in the Excluded category? Thanks!
Otherwise, LGTM, will approve today if you split this part out.

--xfail: Thanks everyone for explaining. As someone who also has to deal with downstream test failures, you convinced me that this is useful to have!
Summarizing my understanding: this is about the desire to (temporarily) mark tests as XFAIL in a centralized location (instead of editing individual test files), especially useful for a downstream integration workflows.
@davezarzycki @jhenderson @thopre Is this accurate?

How about --xfail-file which names a file for which each line means one XFAILed test. I imagine that:

• Working with a file seems better (echo "test/path" >> xfail.txt, sed '/test/d' ./file), than fiddling with the lit invocation itself. Count how many XFAILed tests we have in upstream without even opening the file, etc.
• The file would "make sense" in version control: line-wise addition/removals. "Didn't this test fail before? What was the issue previously? git log -- xfail.txt and see commit for fix and line removal.

What do you think?

[jdenny]

In D96662#2572321, @yln wrote:

--skip: is it okay if we rename this to --filter-not? It's an uglier name but

I like this suggestion for all the reasons @yln gave. However, I'd prefer --filter-out. It has the same advantages @yln gave, such as being obvious if you know what --filter does. A small disadvantage is lack of grammatical symmetry. My issue is that --filter is not a clear name whereas --filter-in would've been clear. Without that context, --filter-not is worse, in my opinion. It sounds like "don't filter these tests", but isn't that the meaning of --filter? --filter-out is clear in any context, in my opinion.

[thopre]

In D96662#2574149, @jdenny wrote:

In D96662#2572321, @yln wrote:

--skip: is it okay if we rename this to --filter-not? It's an uglier name but

I like this suggestion for all the reasons @yln gave. However, I'd prefer --filter-out. It has the same advantages @yln gave, such as being obvious if you know what --filter does. A small disadvantage is lack of grammatical symmetry. My issue is that --filter is not a clear name whereas --filter-in would've been clear. Without that context, --filter-not is worse, in my opinion. It sounds like "don't filter these tests", but isn't that the meaning of --filter? --filter-out is clear in any context, in my opinion.

I like --filter-out. Could we maybe introduce --filter-in and make --filter an alias for --filter-in?

[thopre]

In D96662#2572321, @yln wrote:

--skip: is it okay if we rename this to --filter-not? It's an uglier name but

• more "self explanatory", i..e, "opposite of --filter". Anyone who knows --filter will know what it does just by looking the arg name.
• Discoverable: in the usage and help output, it will appear next to --filter.
• In the summary, we have a category of Skipped tests: tests that were not run because the user hit [Ctrl+C] or because of overall timeout. --skipped tests would appear in the Excluded category (e.g., --filter and sharding feature). Not using the word skip may help prevent some confusion.

Can we also add the summary counts in the test, so that the test shows that --filter-not excluded tests are counted in the Excluded category? Thanks!
Otherwise, LGTM, will approve today if you split this part out.

--xfail: Thanks everyone for explaining. As someone who also has to deal with downstream test failures, you convinced me that this is useful to have!
Summarizing my understanding: this is about the desire to (temporarily) mark tests as XFAIL in a centralized location (instead of editing individual test files), especially useful for a downstream integration workflows.
@davezarzycki @jhenderson @thopre Is this accurate?

Yes. Besides the centralisation, it also avoids changing upstream source (i.e. it allows to separate modifications to test as opposed to disabled test).

How about --xfail-file which names a file for which each line means one XFAILed test. I imagine that:

• Working with a file seems better (echo "test/path" >> xfail.txt, sed '/test/d' ./file), than fiddling with the lit invocation itself. Count how many XFAILed tests we have in upstream without even opening the file, etc.
• The file would "make sense" in version control: line-wise addition/removals. "Didn't this test fail before? What was the issue previously? git log -- xfail.txt and see commit for fix and line removal.

What do you think?

I think a file can be more convenient when lots of tests need to be disabled. I don't think that's ever been our case and I would personally be wary of disabling too many cases but it could be useful. By the way, would the syntax allow to skip entire directories?

[davezarzycki]

I've renamed --skip to --filter-out. I prefer "out" to "not" because that matches the documentation for the command.

I'd like to be done with this patch proposal.

If people want to add --xfail-file later (or rename --filter), go for it. I agree that --xfail-file will be useful but I don't need it yet and I think designing that feature is slightly complicated. Personally, if one is going to have a file, why limit it to just XFAIL? Why not support other lit features like REQUIRES or UNSUPPORTED or ALLOW_RETRIES? All of these might be useful to a downstream maintainer that needs to temporarily tweak a test until something is fixed.

[jdenny]

In D96662#2574156, @thopre wrote:

I like --filter-out. Could we maybe introduce --filter-in and make --filter an alias for --filter-in?

Possibly. We'd have to decide whether there should be a LIT_FILTER_IN and how it would override or otherwise combine with LIT_FILTER. The aliasing probably should be a separate patch.

In D96662#2574214, @davezarzycki wrote:

I've renamed --skip to --filter-out. I prefer "out" to "not" because that matches the documentation for the command.

Thanks.

I agree that --xfail-file will be useful but I don't need it yet and I think designing that feature is slightly complicated. Personally, if one is going to have a file, why limit it to just XFAIL? Why not support other lit features like REQUIRES or UNSUPPORTED or ALLOW_RETRIES? All of these might be useful to a downstream maintainer that needs to temporarily tweak a test until something is fixed.

I do see some appeal to having a general config file (for test suite users as opposed to test suite authors) instead of an xfail-specific file, but I've formed no strong opinions on the way --xfail is specified.

Other than some inline comments I just added, I'm fine with this patch as is, but I'm going to leave acceptance to reviewers who described their use cases.

llvm/utils/lit/tests/Inputs/xfail-cl/lit.cfg

21

Most of this config appears irrelevant to this test.

llvm/utils/lit/tests/xfail-cl.py

8

Shouldn't these say XFAIL?

[yln]

I am also happy with --filter-out.

In D96662#2574214, @davezarzycki wrote:

I agree that --xfail-file will be useful but I don't need it yet and I think designing that feature is slightly complicated. Personally, if one is going to have a file, why limit it to just XFAIL? Why not support other lit features like REQUIRES or UNSUPPORTED or ALLOW_RETRIES? All of these might be useful to a downstream maintainer that needs to temporarily tweak a test until something is fixed.

Point taken. Thanks for your patience and for this whole effort! :)

LGTM, with Joel's nits plus one small one from myself.

llvm/utils/lit/lit/main.py

77

Can we do

selected_tests = [t for t in discovered_tests if opts.filter.search(t.getFullName()) and not opts.filter_out.search(t.getFullName()]

and have a function mark_xfailing() just above mark_excluded() below?

Thanks!

99–101

Here

[davezarzycki]

I believe I have addressed all of the feedback to date.

[jdenny]

llvm/utils/lit/tests/xfail-cl.py

6

It's END. not END:. Then you shouldn't need those {{ and }} around the XFAIL.

[davezarzycki]

Thanks. Fixed: 4550fdff2b2eec15143fac536e41ce967e522a3a

[jdenny]

If a lit invocation runs multiple test suites that contain identically named test files, is there any way to make an --xfail entry apply to only one of them?

[davezarzycki]

In D96662#2593042, @jdenny wrote:

If a lit invocation runs multiple test suites that contain identically named test files, is there any way to make an --xfail entry apply to only one of them?

As the code exists today, no. That'd need to be a followup patch.

llvm/docs/CommandGuide/lit.rst

 
 .. option:: --filter=REGEXP
 
   Run only those tests whose name matches the regular expression specified in
   ``REGEXP``. The environment variable ``LIT_FILTER`` can be also used in place
   of this option, which is especially useful in environments where the call
   to ``lit`` is issued indirectly.
 
+.. option:: --filter-out=REGEXP
+
+  Filter out those tests whose name matches the regular expression specified in
+  ``REGEXP``. The environment variable ``LIT_FILTER_OUT`` can be also used in
+  place of this option, which is especially useful in environments where the
+  call to ``lit`` is issued indirectly.
+
+.. option:: --xfail=LIST
+
+  Treat those tests whose name is in the semicolon separated list ``LIST`` as
+  ``XFAIL``. This can be helpful when one does not want to modify the test
+  suite. The environment variable ``LIT_XFAIL`` can be also used in place of
+  this option, which is especially useful in environments where the call to
+  ``lit`` is issued indirectly.
+
 ADDITIONAL OPTIONS
 ------------------
 
 .. option:: --debug
 
  Run :program:`lit` in debug mode, for debugging configuration issues and
  :program:`lit` itself.
 

llvm/utils/lit/lit/cl_arguments.py

def parse_args():
     selection_group.add_argument("-i", "--incremental",
             help="Run modified and failing tests first (updates mtimes)",
             action="store_true")
     selection_group.add_argument("--filter",
             metavar="REGEX",
             type=_case_insensitive_regex,
             help="Only run tests with paths matching the given regular expression",
             default=os.environ.get("LIT_FILTER", ".*"))
+    selection_group.add_argument("--filter-out",
+            metavar="REGEX",
+            type=_case_insensitive_regex,
+            help="Filter out tests with paths matching the given regular expression",
+            default=os.environ.get("LIT_FILTER_OUT", "^$"))
+    selection_group.add_argument("--xfail",
+            metavar="LIST",
+            type=_semicolon_list,
+            help="XFAIL tests with paths in the semicolon separated list",
+            default=os.environ.get("LIT_XFAIL", ""))
     selection_group.add_argument("--num-shards",
             dest="numShards",
             metavar="M",
             help="Split testsuite into M pieces and only run one",
             type=_positive_int,
             default=os.environ.get("LIT_NUM_SHARDS"))
     selection_group.add_argument("--run-shard",
             dest="runShard",

 def _case_insensitive_regex(arg):
     import re
     try:
         return re.compile(arg, re.IGNORECASE)
     except re.error as reason:
         raise _error("invalid regular expression: '{}', {}", arg, reason)
 
 
+def _semicolon_list(arg):
+    return arg.split(';')
+
+
 def _error(desc, *args):
     msg = desc.format(*args)
     return argparse.ArgumentTypeError(msg)

llvm/utils/lit/lit/main.py

if opts.maxIndividualTestTime is not None:  # `not None` is important (default: 0)
                 ' seconds')
                 .format(lit_config.maxIndividualTestTime,
                         opts.maxIndividualTestTime))
             lit_config.maxIndividualTestTime = opts.maxIndividualTestTime
 
     determine_order(discovered_tests, opts.order)
 
     selected_tests = [t for t in discovered_tests if
-                      opts.filter.search(t.getFullName())]
+        opts.filter.search(t.getFullName()) and not
+        opts.filter_out.search(t.getFullName())]
+
     if not selected_tests:
         sys.stderr.write('error: filter did not match any tests '
                          '(of %d discovered).  ' % len(discovered_tests))
         if opts.allow_empty_runs:
             sys.stderr.write("Suppressing error because '--allow-empty-runs' "
                              'was specified.\n')
             sys.exit(0)
         else:
             sys.stderr.write("Use '--allow-empty-runs' to suppress this "
                              'error.\n')
             sys.exit(2)
 
     # When running multiple shards, don't include skipped tests in the xunit
     # output since merging the files will result in duplicates.
     tests_for_report = discovered_tests
     if opts.shard:
         (run, shards) = opts.shard
         selected_tests = filter_by_shard(selected_tests, run, shards, lit_config)
         tests_for_report = selected_tests
         if not selected_tests:
             sys.stderr.write('warning: shard does not contain any tests.  '
                              'Consider decreasing the number of shards.\n')
             sys.exit(0)
 
     selected_tests = selected_tests[:opts.max_tests]
 
+    mark_xfail(discovered_tests, opts)
+
     mark_excluded(discovered_tests, selected_tests)
 
     start = time.time()
     run_tests(selected_tests, lit_config, opts, len(discovered_tests))
     elapsed = time.time() - start
 
     if opts.time_tests:
         print_histogram(discovered_tests)
if len(test_ixs) > preview_len:
         preview += ', ...'
     msg = f'Selecting shard {run}/{shards} = ' \
           f'size {len(selected_tests)}/{len(tests)} = ' \
           f'tests #({shards}*k)+{run} = [{preview}]'
     lit_config.note(msg)
     return selected_tests
 
 
+def mark_xfail(selected_tests, opts):
+    for t in selected_tests:
+        if os.sep.join(t.path_in_suite) in opts.xfail:
+            t.xfails += '*'
+
 def mark_excluded(discovered_tests, selected_tests):
     excluded_tests = set(discovered_tests) - set(selected_tests)
     result = lit.Test.Result(lit.Test.EXCLUDED)
     for t in excluded_tests:
         t.setResult(result)
 
 
 def run_tests(tests, lit_config, opts, discovered_tests):

llvm/utils/lit/tests/Inputs/xfail-cl/false.txt

+# RUN: false

llvm/utils/lit/tests/Inputs/xfail-cl/false2.txt

+# RUN: false

llvm/utils/lit/tests/Inputs/xfail-cl/lit.cfg

+import lit.formats
+config.name = 'top-level-suite'
+config.suffixes = ['.txt']
+config.test_format = lit.formats.ShTest()

llvm/utils/lit/tests/Inputs/xfail-cl/true.txt

+# RUN: true

llvm/utils/lit/tests/selecting.py

 # RUN: %{lit} %{inputs}/discovery | FileCheck --check-prefix=CHECK-BASIC %s
 # CHECK-BASIC: Testing: 5 tests
 
 
 # Check that we exit with an error if we do not discover any tests, even with --allow-empty-runs.
 #
 # RUN: not %{lit} %{inputs}/nonexistent                    2>&1 | FileCheck --check-prefix=CHECK-BAD-PATH %s
 # RUN: not %{lit} %{inputs}/nonexistent --allow-empty-runs 2>&1 | FileCheck --check-prefix=CHECK-BAD-PATH %s
 # CHECK-BAD-PATH: error: did not discover any tests for provided path(s)
 
 # Check that we exit with an error if we filter out all tests, but allow it with --allow-empty-runs.
+# Check that we exit with an error if we skip all tests, but allow it with --allow-empty-runs.
 #
 # RUN: not %{lit} --filter 'nonexistent'                    %{inputs}/discovery 2>&1 | FileCheck --check-prefixes=CHECK-BAD-FILTER,CHECK-BAD-FILTER-ERROR %s
 # RUN:     %{lit} --filter 'nonexistent' --allow-empty-runs %{inputs}/discovery 2>&1 | FileCheck --check-prefixes=CHECK-BAD-FILTER,CHECK-BAD-FILTER-ALLOW %s
+# RUN: not %{lit} --filter-out '.*'                    %{inputs}/discovery 2>&1 | FileCheck --check-prefixes=CHECK-BAD-FILTER,CHECK-BAD-FILTER-ERROR %s
+# RUN:     %{lit} --filter-out '.*' --allow-empty-runs %{inputs}/discovery 2>&1 | FileCheck --check-prefixes=CHECK-BAD-FILTER,CHECK-BAD-FILTER-ALLOW %s
 # CHECK-BAD-FILTER: error: filter did not match any tests (of 5 discovered).
 # CHECK-BAD-FILTER-ERROR: Use '--allow-empty-runs' to suppress this error.
 # CHECK-BAD-FILTER-ALLOW: Suppressing error because '--allow-empty-runs' was specified.
 
 # Check that regex-filtering works, is case-insensitive, and can be configured via env var.
 #
 # RUN: %{lit} --filter 'o[a-z]e' %{inputs}/discovery | FileCheck --check-prefix=CHECK-FILTER %s
 # RUN: %{lit} --filter 'O[A-Z]E' %{inputs}/discovery | FileCheck --check-prefix=CHECK-FILTER %s
 # RUN: env LIT_FILTER='o[a-z]e' %{lit} %{inputs}/discovery | FileCheck --check-prefix=CHECK-FILTER %s
+# RUN: %{lit} --filter-out 'test-t[a-z]' %{inputs}/discovery | FileCheck --check-prefix=CHECK-FILTER %s
+# RUN: %{lit} --filter-out 'test-t[A-Z]' %{inputs}/discovery | FileCheck --check-prefix=CHECK-FILTER %s
+# RUN: env LIT_FILTER_OUT='test-t[a-z]' %{lit} %{inputs}/discovery | FileCheck --check-prefix=CHECK-FILTER %s
 # CHECK-FILTER: Testing: 2 of 5 tests
 # CHECK-FILTER: Excluded: 3
 
 
 # Check that maximum counts work
 #
 # RUN: %{lit} --max-tests 3 %{inputs}/discovery | FileCheck --check-prefix=CHECK-MAX %s
 # CHECK-MAX: Testing: 3 of 5 tests

llvm/utils/lit/tests/xfail-cl.py

+# Check that regex-XFAILing works and can be configured via env var.
+#
+# RUN: %{lit} --xfail 'false.txt;false2.txt' %{inputs}/xfail-cl | FileCheck --check-prefix=CHECK-FILTER %s
+# RUN: env LIT_XFAIL='false.txt;false2.txt' %{lit} %{inputs}/xfail-cl | FileCheck --check-prefix=CHECK-FILTER %s
+# END:
+# CHECK-FILTER: Testing: 3 tests, 3 workers
+# CHECK-FILTER-DAG: {{XFAIL}}: top-level-suite :: false.txt
+# CHECK-FILTER-DAG: {{XFAIL}}: top-level-suite :: false2.txt
+# CHECK-FILTER-DAG: PASS: top-level-suite :: true.txt