[llvm-lit][test] Fix regex to capture scientific notation

[gbreynoo]

Summary

We were seeing an intermittent local test failure of utils\lit\tests\test-output.py in which the elapsed time output was being given in scientific notation. Python automatically represents small floating-point values in scientific notation so I have altered these tests regex to capture output in that format.

[gbreynoo]

Intermittent error below:

[09:50:05] :	 [Step 3/3] next:10'0                     X error: no match found
[09:50:05] :	 [Step 3/3]           11:  "elapsed": 7.295608520507812e-05,

[jdenny]

llvm/utils/lit/tests/test-output-micro.py

42

If the goal is to be strict, then . needs to be escaped to make it literal.

However, is it important to be so strict here? What about something simple like {{[-+0-9.eE]+}}?

[gbreynoo]

Updated after jdenny's suggested regex change.

[gbreynoo]

llvm/utils/lit/tests/test-output-micro.py

42

Thanks @jdenny, that's much better.

[jdenny]

LGTM. Thanks for the fix.

If there's a publicly viewable build log showing the original error, it would be nice to link to that from the review.

[gbreynoo]

Apologies @jdenny, the failure was occurring on a private build job and has no public access. The test failure can be found below:

08:56:44   ******************** TEST 'lit :: test-output.py' FAILED ********************
08:56:44   Script:
08:56:44   --
08:56:44   : 'RUN: at line 1';   env -u FILECHECK_OPTS "C:\Program Files\Python310\python.exe" D:\BuildAgent\work\e280ceab0c58d514\llvm\utils\lit\lit.py -j1 --order=lexical -v Inputs/test-data --output D:\BuildAgent\work\e280ceab0c58d514\build\utils\lit\tests\Output\test-output.py.tmp.results.out > D:\BuildAgent\work\e280ceab0c58d514\build\utils\lit\tests\Output\test-output.py.tmp.out
08:56:44   : 'RUN: at line 2';   FileCheck < D:\BuildAgent\work\e280ceab0c58d514\build\utils\lit\tests\Output\test-output.py.tmp.results.out D:\BuildAgent\work\e280ceab0c58d514\build\utils\lit\tests\test-output.py
08:56:44   --
08:56:44   Exit Code: 1
08:56:44   
08:56:44   Command Output (stdout):
08:56:44   --
08:56:44   $ ":" "RUN: at line 1"
08:56:44   $ "env" "-u" "FILECHECK_OPTS" "C:\Program Files\Python310\python.exe" "D:\BuildAgent\work\e280ceab0c58d514\llvm\utils\lit\lit.py" "-j1" "--order=lexical" "-v" "Inputs/test-data" "--output" "D:\BuildAgent\work\e280ceab0c58d514\build\utils\lit\tests\Output\test-output.py.tmp.results.out"
08:56:44   $ ":" "RUN: at line 2"
08:56:44   $ "FileCheck" "D:\BuildAgent\work\e280ceab0c58d514\build\utils\lit\tests\test-output.py"
08:56:44   # command stderr:
08:56:44   D:\BuildAgent\work\e280ceab0c58d514\build\utils\lit\tests\test-output.py:10:15: error: CHECK-NEXT: expected string not found in input
08:56:44   # CHECK-NEXT: "elapsed": {{[0-9.]+}},
08:56:44                 ^
08:56:44   <stdin>:10:17: note: scanning from here
08:56:44    "code": "PASS",
08:56:44                   ^
08:56:44   <stdin>:11:2: note: possible intended match here
08:56:44    "elapsed": 7.510185241699219e-05,
08:56:44    ^
08:56:44   
08:56:44   Input file: <stdin>
08:56:44   Check file: D:\BuildAgent\work\e280ceab0c58d514\build\utils\lit\tests\test-output.py
08:56:44   
08:56:44   -dump-input=help explains the following input dump.
08:56:44   
08:56:44   Input was:
08:56:44   <<<<<<
08:56:44              .
08:56:44              .
08:56:44              .
08:56:44              5:  0
08:56:44              6:  ],
08:56:44              7:  "elapsed": 0.46836328506469727,
08:56:44              8:  "tests": [
08:56:44              9:  {
08:56:44             10:  "code": "PASS",
08:56:44   next:10'0                     X error: no match found
08:56:44             11:  "elapsed": 7.510185241699219e-05,
08:56:44   next:10'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
08:56:44   next:10'1      ?                                  possible intended match
08:56:44             12:  "metrics": {
08:56:44   next:10'0     ~~~~~~~~~~~~~~
08:56:44             13:  "value0": 1,
08:56:44   next:10'0     ~~~~~~~~~~~~~~
08:56:44             14:  "value1": 2.3456
08:56:44   next:10'0     ~~~~~~~~~~~~~~~~~~
08:56:44             15:  },
08:56:44   next:10'0     ~~~~
08:56:44             16:  "name": "test-data :: metrics.ini",
08:56:44   next:10'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
08:56:44              .
08:56:44              .
08:56:44              .
08:56:44   >>>>>>
08:56:44   
08:56:44   error: command failed with exit status: 1

llvm/utils/lit/tests/test-output-micro.py

 # CHECK-NEXT:       "micro_value0": 4,
 # CHECK-NEXT:       "micro_value1": 1.3
 # CHECK-NEXT:     },
 # CHECK-NEXT:     "name": "test-data-micro :: micro-tests.ini:test{{[0-2]}}",
 # CHECK-NEXT:     "output": ""
 # CHECK-NEXT:   },
 # CHECK-NEXT:   {
 # CHECK-NEXT:     "code": "PASS",
-# CHECK-NEXT:     "elapsed": {{[0-9.]+}},
+# CHECK-NEXT:     "elapsed": {{[-+0-9.eE]+}},
 # CHECK-NEXT:     "metrics": {
 # CHECK-NEXT:       "value0": 1,
 # CHECK-NEXT:       "value1": 2.3456
 # CHECK-NEXT:     },
 # CHECK-NEXT:     "name": "test-data-micro :: micro-tests.ini",
 # CHECK-NEXT:     "output": "Test passed."
 # CHECK-NEXT:   }
 # CHECK-NEXT: ]
 # CHECK-NEXT: }

llvm/utils/lit/tests/test-output.py

 # RUN: %{lit} -v %{inputs}/test-data --output %t.results.out > %t.out
 # RUN: FileCheck < %t.results.out %s
 
 # CHECK: {
 # CHECK: "__version__"
 # CHECK: "elapsed"
 # CHECK-NEXT: "tests": [
 # CHECK-NEXT:   {
 # CHECK-NEXT:     "code": "PASS",
-# CHECK-NEXT:     "elapsed": {{[0-9.]+}},
+# CHECK-NEXT:     "elapsed": {{[-+0-9.eE]+}},
 # CHECK-NEXT:     "metrics": {
 # CHECK-NEXT:       "value0": 1,
 # CHECK-NEXT:       "value1": 2.3456
 # CHECK-NEXT:     }
 # CHECK-NEXT:     "name": "test-data :: metrics.ini",
 # CHECK-NEXT:     "output": "Test passed."
 # CHECK-NEXT:   }
 # CHECK-NEXT: ]
 # CHECK-NEXT: }