[lit] Have REQUIRES support the target triple

[probinson]

Summary

Currently the UNSUPPORTED and XFAIL clauses support specifying
substrings of the target triple; but REQUIRES does not, which can trip
people up or lead to hacking config files to insert substitute feature
names. Consistency across all three lit clauses seems preferable.

[probinson]

I'll likely follow up with removing some unused or now-redundant feature keywords.
This does seem like the same direction as D104572 although I haven't actually tested pattern-matching on triples in REQUIRES; hard to see how it would differ from the matching for UNSUPPORTED and XFAIL.

[probinson]

FTR, implementing this discovered the never-run test that I fixed in e6620a35

[yln]

Currently the UNSUPPORTED and XFAIL clauses support specifying
substrings of the target triple; but REQUIRES does not

My belief was that this was intentional but I never questioned that belief: disable tests broadly, but mark them running narrowly. Consistency is definitely better.

[delcypher]

@probinson I'm not a huge fan of this change. If I've understood this change correctly then this could have unintended consequences in existing test suites. If an existing test suite uses "features" that are substrings of the target triple then this behaviour change means they'll be a match now where there wasn't before. That doesn't seem desirable at all. I also think this "matching a substring of the target triple" should be removed completely. I believe @ldionne recently added regex support in REQUIRES conditions and that seems like a better solution here.

[probinson]

What we've run into, repeatedly, is that people will naively do something like REQUIRES: linux expecting it to match against the triple; except it doesn't, because REQUIRES doesn't match triples, and you end up with a test that inadvertently doesn't run at all (as in the one I fixed on Friday).

It's true that you can substitute UNSUPPORTED: !linux to get the desired effect, but this is less than obvious, besides needing a double negative.

Eliminating triple substring matching (in favor of explicit regexes) would affect a lot more existing tests than the hypothetical feature-that-looks-like-a-substring case, IMO. I don't claim that my check-all run found everything, but summary of test counts at the end differed by only that one test mentioned previously.

[jdenny]

@probinson: Can you confirm my understanding, as follows?

• Without this patch, REQUIRES doesn't match the target triple at all (unless it happens to have been added as a feature to a particular test suite). This patch enables matching the entire target triple or a substring of it.
• In any hypothetical test suite, this patch can only expand the number of tests that run because it enables REQUIRES to be more easily satisfied (by the current target triple). It is impossible for this patch to reduce the number of tests that run.
• In practice so far, only one test is affected.

A few comments, assuming my above understanding is correct:

• Hiding test failures accidentally is bad. Exposing test failures accidentally is not so bad.
• From that perspective, it seems exactly backward that UNSUPPORTED and XFAIL currently match more broadly than REQUIRES. Why was that decision made?
• The existence of a test case and documentation pointing out the existing behavior means this behavior was intentional. That makes me concerned I've misunderstood what's happening.
• I do agree that eventually it would be nice to get rid of substring matching in favor of regexes for XFAIL and UNSUPPORTED, where matching more broadly than intended is dangerous. I'm not sure I buy that argument for REQUIRES.
• If this patch will be landed, it should update the docs: https://llvm.org/docs/TestingGuide.html#constraining-test-execution

llvm/utils/lit/tests/Inputs/xunit-output/lit.cfg

10

Why does this need to change?

[yln]

In D107162#2926617, @jdenny wrote:

• Hiding test failures accidentally is bad. Exposing test failures accidentally is not so bad.
• From that perspective, it seems exactly backward that UNSUPPORTED and XFAIL currently match more broadly than REQUIRES.
• I do agree that eventually it would be nice to get rid of substring matching in favor of regexes for XFAIL and UNSUPPORTED, where matching more broadly than intended is dangerous.

I completely agree with these points made by Joel.

[probinson]

In D107162#2926617, @jdenny wrote:

@probinson: Can you confirm my understanding, as follows?

• Without this patch, REQUIRES doesn't match the target triple at all (unless it happens to have been added as a feature to a particular test suite). This patch enables matching the entire target triple or a substring of it.

Correct.

I'll note that there are a bunch of triple-based (and sometimes triple-like) feature keywords added by lit, based on the target triple (see the block starting if target_triple: at circa line 106 in llvm/utils/lit/lit/llvm/config.py), which of course could be replaced by regexes in UNSUPPORTED/XFAIL already, and also in REQUIRES if this patch goes in.

• In any hypothetical test suite, this patch can only expand the number of tests that run because it enables REQUIRES to be more easily satisfied (by the current target triple). It is impossible for this patch to reduce the number of tests that run.

Correct.

• In practice so far, only one test is affected.

Out of the projects I generally build: llvm;clang;clang-tools-extra;lld;lldb. I can try doing "all" projects and see what happens, although my track record on getting clean builds and tests is imperfect.

A few comments, assuming my above understanding is correct:

• Hiding test failures accidentally is bad. Exposing test failures accidentally is not so bad.
• From that perspective, it seems exactly backward that UNSUPPORTED and XFAIL currently match more broadly than REQUIRES. Why was that decision made?

"It was like that when I found it." Others who were more involved in lit in the past might be better able to explain. Roaming through the history, UNSUPPORTED was added by Eric Fiselier in 2014; it did not support triple-checking at first either, that was added by Peter Collingbourne about a year later. I didn't try to research the XFAIL history.

• The existence of a test case and documentation pointing out the existing behavior means this behavior was intentional. That makes me concerned I've misunderstood what's happening.

I accept that REQUIRES intentionally has not supported target-triple matching; I still think we're better off with it in place, unless someone can come up with a really nice statement of why it's beneficial enough that we should live with the inconsistency and the need for miscellaneous ad-hoc triple substitutes as mentioned above.

• I do agree that eventually it would be nice to get rid of substring matching in favor of regexes for XFAIL and UNSUPPORTED, where matching more broadly than intended is dangerous. I'm not sure I buy that argument for REQUIRES.
• If this patch will be landed, it should update the docs: https://llvm.org/docs/TestingGuide.html#constraining-test-execution

Ah, thanks for pointing to the doc; I will update it. I see that REQUIRES was not documented at all until 2016 but that's no excuse for not keeping it up-to-date now.

llvm/utils/lit/tests/Inputs/xunit-output/lit.cfg

10

Because the test execution goes through the REQUIRES path, and the target_triple check gets a Python error if it's None (says something like "not iterable").

[jdenny]

In D107162#2926821, @probinson wrote:

Correct.

Thanks for confirming.

I'll note that there are a bunch of triple-based (and sometimes triple-like) feature keywords added by lit, based on the target triple (see the block starting if target_triple: at circa line 106 in llvm/utils/lit/lit/llvm/config.py),

Ah, thanks.

which of course could be replaced by regexes in UNSUPPORTED/XFAIL already, and also in REQUIRES if this patch goes in.

Maybe I missed a patch since D104572, but it did not add support for regexes matching the target triple. It added support for regexes matching features, as described in the documentation I pointed out. Now that I think about it more, I'm not sure I know what @ldionne ultimately wanted to see happen with target triples. It might be good to hear him comment before moving forward.

I can try doing "all" projects and see what happens, although my track record on getting clean builds and tests is imperfect.

Same here. Sometimes we just have to let the bots find things, unfortunately.

• Hiding test failures accidentally is bad. Exposing test failures accidentally is not so bad.
• From that perspective, it seems exactly backward that UNSUPPORTED and XFAIL currently match more broadly than REQUIRES. Why was that decision made?

"It was like that when I found it."

:-)

[delcypher]

@jdenny

In any hypothetical test suite, this patch can only expand the number of tests that run because it enables REQUIRES to be more easily satisfied (by the current target triple). It is impossible for this patch to reduce the number of tests that run.

That is a good point and sometime I missed the first time I looked over this patch. Overly broad matching would mostly be a problem for the UNSUPPORTED declaration which we are already doing. So I withdraw my objections to this patch.

I also took a quick look at the regex support in lit. Looks like the current implementation doesn't actually support regexes for target triples based on the tests we have. So it can't be used as alternative right now.

# When matching against the triple, a regex is treated as an identifier and checked
# for a literal match. This preserves existing behavior before regexes were introduced.
self.assertFalse(BooleanExpression.evaluate('arch-{{vendor}}-os', {}, triple))
self.assertTrue(BooleanExpression.evaluate('arch-{{vendor}}-os', {}, 'arch-{{vendor}}-os'))

I'd still like to see support for substring matching on target triples completely removed but I don't think we should do that until we have a viable replacement for it.

[delcypher]

llvm/utils/lit/tests/Inputs/xunit-output/lit.cfg

10

To avoid breaking any existing test suite that sets config.target_triple to None we could do this.

def getMissingRequiredFeaturesFromList(self, features):
     triple = getattr(self.suite.config, 'target_triple', "")
     if triple is None:
       triple = ''
     try:

However getUnsupportedFeatures() doesn't do this currently so that would be a bit inconsistent.

[jdenny]

Reviewing comments from @yln and @delcypher, it seems we're all mostly in agreement now, and I haven't found a good reason to object.

LGTM with a few remaining caveats:

1. It needs the documentation we discussed.
2. I'd like to give @ldionne a few more days to comment given that he's already been thinking about the handling of target triples.
3. We might have to revert the patch if it undesirably enables tests in test suites that haven't been tried yet.

Thanks for working on this, @probinson.

[probinson]

In order to find other affected tests, I enabled as many projects as possible; this turns out to be all of them except libc and cross-project-tests. (libc won't build with gcc 7.5, apparently, and cross-project-tests appears to have an error that keeps it from building at all.)

The test summary counts show there are two Unsupported tests that change to Pass, and one Fail that changes to Pass (which really doesn't make sense, maybe it's just a flaky test). I'm rerunning with and without the patch, collecting the complete lists of Unsupported tests in order to identify which ones have started passing. Most likely they are in the same category as the one I fixed previously, i.e. some inadvertent mistake made when writing the REQUIRES clause, except they don't happen to fail when actually run.

I'll get the docs fixed up so they can be reviewed before I push this, and also report back on which tests were affected and why.

Thanks all, especially for the correction about which cases allow regexes.

llvm/utils/lit/tests/Inputs/xunit-output/lit.cfg

10

In order for an existing test suite to have config.target_triple = None and not get this error today, all tests controlled by that lit.cfg file would have to avoid using UNSUPPORTED and XFAIL. That seems unlikely.
It comes up here only because we're looking at lit's own tests, which each tend to have their own config file.
I think the risk is miniscule, but I'm willing to make the change (for all 3 clauses) if you think it's appropriate.

[probinson]

Update doc

[probinson]

The two tests that went from Unsupported to Pass are

LLVM :: tools/llvm-readobj/COFF/call-graph-profile-err.s
LLVM :: tools/llvm-readobj/COFF/call-graph-profile.s

both of which have REQUIRES: x86 which matches the default triple on my system (x86_64-unknown-linux) but isn't the right way to constrain the test in a pre-patch world; that would be target-x86 or target-x86_64 or both.

Reviewers have expressed some need for caution in applying this patch, and there might well be tests out there that mistakenly have things like REQUIRES: arm or REQUIRES: powerpc and don't happily pass once they are suddenly enabled for the first time. So, I guess that means I won't be pushing this upstream at quitting time on Friday after all. :-)

[probinson]

FTR, I found an old note to myself that D83759 fixed a test that originally expected REQUIRES to support the triple. (The patch did other things too, but that was one of them.) In case anyone wanted more evidence that people somewhat naturally expect all three clauses to behave the same way.

[jdenny]

llvm/docs/TestingGuide.rst

472–473

I'm not sure this comment about precedence is meaningful. It seems to say that, if you have an UNSUPPORTED directive, any REQUIRES directive is ignored. Actually, in the implementation, REQUIRES is checked first.

I think there's no sense of precedence. The test runs if REQUIRES is matched and UNSUPPORTED isn't matched. I think they could be evaluated in either order.

[probinson]

llvm/docs/TestingGuide.rst

472–473

They can be evaluated in either order; but if they are both true, which one wins? The one that has precedence.

[jdenny]

llvm/docs/TestingGuide.rst

472–473

You could just as easily ask: If they are both false, which one wins?

[probinson]

llvm/docs/TestingGuide.rst

472–473

Now that I think about it, REQUIRES isn't required :) so by default tests are enabled. That means REQUIRES doesn't actually enable a test, So really we should say:

REQUIRES disables the test if any expression is false.
UNSUPPORTED disables the test if any expression is true.

Then there's no debate about precedence; both clauses act to disable the test. Which is how it actually works.

[jdenny]

llvm/docs/TestingGuide.rst

472–473

Yes, I like that better too.

And now I get what led to the suggestion of precedence. :-)

[probinson]

Revised description of REQUIRES.

[jdenny]

Documentation changes LGTM. Thanks, Paul.

[thakis]

Looks like this breaks check-lld on macOS: http://45.33.8.238/macm1/15677/step_10.txt

Please take a look.

[probinson]

In D107162#2931422, @thakis wrote:

Looks like this breaks check-lld on macOS: http://45.33.8.238/macm1/15677/step_10.txt

Please take a look.

I don't have a mac to try it on, but based on the FileCheck output it looks like there's one extra CHECK line, so I tried removing it in f88ad8d00f970ea937e8d3fcb71f4d4019f2821a
Let me know whether that works.

[probinson]

In D107162#2931422, @thakis wrote:

Looks like this breaks check-lld on macOS: http://45.33.8.238/macm1/15677/step_10.txt

Please take a look.

Seems to be better now: http://45.33.8.238/macm1/15689/summary.html

[thakis]

In D107162#2931752, @probinson wrote:

In D107162#2931422, @thakis wrote:

Looks like this breaks check-lld on macOS: http://45.33.8.238/macm1/15677/step_10.txt

Please take a look.

Seems to be better now: http://45.33.8.238/macm1/15689/summary.html

Looks like adding the not helped. That's a bit unexpected since lld isn't supposed to exit non-0 there. Do you know how the test worked before? Did it just never run?

@int3 too.

[probinson]

In D107162#2932060, @thakis wrote:

Looks like adding the not helped. That's a bit unexpected since lld isn't supposed to exit non-0 there. Do you know how the test worked before? Did it just never run?

@int3 too.

Correct, the REQUIRES: x86 would never have matched anything so it just didn't run.

[ldionne]

Sorry this went under my Radar, I think I was on vacation at the time when this discussion happened and it went straight to the bottom of my Differential email pile.

Anyway, what we do in libc++/libc++abi/libunwind is that we explicitly add a feature that has the name of the target triple, so that we're able to regex-match on it. I've never liked that the target triple is treated specially by Lit (IMO there shouldn't even be a notion of a target triple in Lit, since Lit is more general than just compiler testing). I think it used to be necessary because the triple was handled specially, but it doesn't need to anymore since one can regex-match on arbitrary features.

So one migration path could be:

1. Manually add a feature consisting of the target triple in all existing test suites (or have Lit add the triple as a feature automatically)
2. Migrate tests that use substring matching of triples to using regex-matching
3. Remove the special handling of target triples in Lit (since nobody should be using it)

To avoid confusion between the target triple as defined by Lit and the target triple as a feature, we've named the feature target=<xxxxxx> in libc++ and it works quite well. So for example, I moved a bunch of

XFAIL: apple-macosx10.9
XFAIL: apple-macosx10.10
XFAIL: apple-macosx10.11

to instead become:

XFAIL: target={{.+}}-apple-macosx10.{{9|10|11}}

Having a feature named target=<xxx> is a bit unusual, but it has been working really well and I feel like it clearly expresses the nature of the Lit feature.

[probinson]

As this patch has had to be reverted due to a completely baffling failure on one bot, we can reopen the debate about how to solve the original problem, which is: XFAIL and UNSUPPORTED allow triples (with substring handling), but REQUIRES doesn't know about triples at all.

I think adding things into lit itself is a means to avoid redundantly adding the same feature to many subprojects, which carries the attendant risk of adding *almost* the same feature to many subprojects. I understand that lit can be adapted to other kinds of testing, but the name after all is an acronym for "LLVM Integrated Tester" so having llvm-project-specific stuff in "lit itself" doesn't seem beyond the pale to me.

I'm in the middle of something else but getting back to this is very near the top of my list. I'd want to scope the existing use of triples in XFAIL and UNSUPPORTED clauses before I'd sign up to convert all of them, rather than do what this patch tried to do (make REQUIRES behave exactly the same as XFAIL/UNSUPPORTED already do).

Also, if we're leaning toward doing a bulk syntactic change like you propose, that should be brought up on the dev lists first to get broader exposure and buy-in.

[ldionne]

In D107162#3017984, @probinson wrote:

As this patch has had to be reverted due to a completely baffling failure on one bot, we can reopen the debate about how to solve the original problem, which is: XFAIL and UNSUPPORTED allow triples (with substring handling), but REQUIRES doesn't know about triples at all.

I think adding things into lit itself is a means to avoid redundantly adding the same feature to many subprojects, which carries the attendant risk of adding *almost* the same feature to many subprojects.

Lit could add the feature itself. The main point is that instead of treating the triple specially during boolean evaluation, we'd add a feature consisting of the triple and then handle all features the same way.

Also, if we're leaning toward doing a bulk syntactic change like you propose, that should be brought up on the dev lists first to get broader exposure and buy-in.

I was just talking about libc++'s experience -- we don't really have to rename everything if people think it causes too much churn. But using target=<xxx> instead of just <xxx> has the benefit that

1. it's clear that we're talking about a property of the target, not a property of the host, and
2. one can't confuse the triple-as-specially-handled-by-lit and the triple-as-a-normal-lit-feature during the transitional period where both exist

[probinson]

FTR, posted an RFC here: https://discourse.llvm.org/t/rfc-lits-requires-and-triples/66041
Please let's have additional discussion there. I'm hoping to more-or-less resurrect this but a wider discussion felt appropriate.
@yln I would have tagged you there but I'm not finding a corresponding Discourse handle.

llvm/docs/TestingGuide.rst

 Constraining test execution
 ---------------------------
 
 Some tests can be run only in specific configurations, such as
 with debug builds or on particular platforms. Use ``REQUIRES``
 and ``UNSUPPORTED`` to control when the test is enabled.
 
 Some tests are expected to fail. For example, there may be a known bug
-that the test detect. Use ``XFAIL`` to mark a test as an expected failure.
+that a test detects. Use ``XFAIL`` to mark a test as an expected failure.
 An ``XFAIL`` test will be successful if its execution fails, and
 will be a failure if its execution succeeds.
 
 .. code-block:: llvm
 
     ; This test will be only enabled in the build with asserts.
     ; REQUIRES: asserts
     ; This test is disabled on Linux.
     ; UNSUPPORTED: -linux-
     ; This test is expected to fail on PowerPC.
     ; XFAIL: powerpc
 
 ``REQUIRES`` and ``UNSUPPORTED`` and ``XFAIL`` all accept a comma-separated
 list of boolean expressions. The values in each expression may be:
 
 - Features added to ``config.available_features`` by configuration files such as ``lit.cfg``.
   String comparison of features is case-sensitive. Furthermore, a boolean expression can
   contain any Python regular expression enclosed in ``{{ }}``, in which case the boolean
   expression is satisfied if any feature matches the regular expression. Regular
   expressions can appear inside an identifier, so for example ``he{{l+}}o`` would match
   ``helo``, ``hello``, ``helllo``, and so on.
-- Substrings of the target triple (``UNSUPPORTED`` and ``XFAIL`` only).
+- Substrings of the target triple. These must be exact substrings, not regular
+  expressions.
 
 | ``REQUIRES`` enables the test if all expressions are true.
-| ``UNSUPPORTED`` disables the test if any expression is true.
+| ``UNSUPPORTED`` disables the test if any expression is true, and takes
+precedence over ``REQUIRES``.
 | ``XFAIL`` expects the test to fail if any expression is true.
 
 As a special case, ``XFAIL: *`` is expected to fail everywhere.
 
 .. code-block:: llvm
 
     ; This test is disabled on Windows,
     ; and is disabled on Linux, except for Android Linux.

llvm/utils/lit/lit/Test.py

def isWithinFeatureLimits(self):
         featuresMinusLimits = [f for f in self.config.available_features
                                if not f in self.config.limit_to_features]
         if not self.getMissingRequiredFeaturesFromList(featuresMinusLimits):
             return False
 
         return True
 
     def getMissingRequiredFeaturesFromList(self, features):
+        triple = getattr(self.suite.config, 'target_triple', "")
         try:
             return [item for item in self.requires
-                    if not BooleanExpression.evaluate(item, features)]
+                    if not BooleanExpression.evaluate(item, features, triple)]
         except ValueError as e:
             raise ValueError('Error in REQUIRES list:\n%s' % str(e))
 
     def getMissingRequiredFeatures(self):
         """
         getMissingRequiredFeatures() -> list of strings
 
         Returns a list of features from REQUIRES that are not satisfied."

llvm/utils/lit/tests/Inputs/shtest-format/requires-triple.txt

-# REQUIRES line that uses target triple, which doesn't work. Test should not run
+# REQUIRES line that uses target triple.
 REQUIRES: x86_64
-RUN: false
+RUN: true

llvm/utils/lit/tests/Inputs/xunit-output/lit.cfg

 import site
 site.addsitedir(os.path.dirname(__file__))
 import dummy_format
 
 config.name = 'test-data'
 config.suffixes = ['.ini']
 config.test_format = dummy_format.DummyFormat()
 config.test_source_root = None
 config.test_exec_root = None
-config.target_triple = None
+config.target_triple = ''

llvm/utils/lit/tests/shtest-format.py

 # CHECK-NEXT: line 1: failed test output on stdout
 # CHECK-NEXT: line 2: failed test output on stdout
 
 # CHECK: UNRESOLVED: shtest-format :: no-test-line.txt
 # CHECK: PASS: shtest-format :: pass.txt
 # CHECK: UNSUPPORTED: shtest-format :: requires-missing.txt
 # CHECK: PASS: shtest-format :: requires-present.txt
 # CHECK: UNRESOLVED: shtest-format :: requires-star.txt
-# CHECK: UNSUPPORTED: shtest-format :: requires-triple.txt
+# CHECK: PASS: shtest-format :: requires-triple.txt
 # CHECK: PASS: shtest-format :: unsupported-expr-false.txt
 # CHECK: UNSUPPORTED: shtest-format :: unsupported-expr-true.txt
 # CHECK: UNRESOLVED: shtest-format :: unsupported-star.txt
 # CHECK: UNSUPPORTED: shtest-format :: unsupported_dir/some-test.txt
 # CHECK: PASS: shtest-format :: xfail-expr-false.txt
 # CHECK: XFAIL: shtest-format :: xfail-expr-true.txt
 # CHECK: XFAIL: shtest-format :: xfail-feature.txt
 # CHECK: XFAIL: shtest-format :: xfail-target.txt

 # CHECK: shtest-format :: external_shell/fail_with_bad_encoding.txt
 # CHECK: shtest-format :: external_shell/fail_with_control_chars.txt
 # CHECK: shtest-format :: fail.txt
 
 # CHECK: Unexpectedly Passed Tests (1)
 # CHECK: shtest-format :: xpass.txt
 
 # CHECK: Testing Time:
-# CHECK: Unsupported        : 4
-# CHECK: Passed             : 6
+# CHECK: Unsupported        : 3
+# CHECK: Passed             : 7
 # CHECK: Expectedly Failed  : 4
 # CHECK: Unresolved         : 3
 # CHECK: Failed             : 4
 # CHECK: Unexpectedly Passed: 1
 
 
 # XUNIT: <?xml version="1.0" encoding="UTF-8"?>
 # XUNIT-NEXT: <testsuites time="{{[0-9.]+}}">
-# XUNIT-NEXT: <testsuite name="shtest-format" tests="22" failures="8" skipped="4">
+# XUNIT-NEXT: <testsuite name="shtest-format" tests="22" failures="8" skipped="3">
 
 # XUNIT: <testcase classname="shtest-format.external_shell" name="fail.txt" time="{{[0-9]+\.[0-9]+}}">
 # XUNIT-NEXT: <failure{{[ ]*}}>
 # XUNIT: </failure>
 # XUNIT-NEXT: </testcase>
 
 
 # XUNIT: <testcase classname="shtest-format.external_shell" name="fail_with_bad_encoding.txt" time="{{[0-9]+\.[0-9]+}}">

 # XUNIT: <testcase classname="shtest-format.shtest-format" name="requires-present.txt" time="{{[0-9]+\.[0-9]+}}"/>
 
 # XUNIT: <testcase classname="shtest-format.shtest-format" name="requires-star.txt" time="{{[0-9]+\.[0-9]+}}">
 # XUNIT-NEXT: <failure{{[ ]*}}>
 # XUNIT: </failure>
 # XUNIT-NEXT: </testcase>
 
 
-# XUNIT: <testcase classname="shtest-format.shtest-format" name="requires-triple.txt" time="{{[0-9]+\.[0-9]+}}">
-# XUNIT-NEXT:<skipped message="Missing required feature(s): x86_64"/>
+# XUNIT: <testcase classname="shtest-format.shtest-format" name="requires-triple.txt" time="{{[0-9]+\.[0-9]+}}"/>
 
 # XUNIT: <testcase classname="shtest-format.shtest-format" name="unsupported-expr-false.txt" time="{{[0-9]+\.[0-9]+}}"/>
 
 # XUNIT: <testcase classname="shtest-format.shtest-format" name="unsupported-expr-true.txt" time="{{[0-9]+\.[0-9]+}}">
 # XUNIT-NEXT:<skipped message="Unsupported configuration"/>
 
 # XUNIT: <testcase classname="shtest-format.shtest-format" name="unsupported-star.txt" time="{{[0-9]+\.[0-9]+}}">
 # XUNIT-NEXT: <failure{{[ ]*}}>