One problem that makes adding new tools to the test-suite complicated is the fact that timeit appends the exit code to the produced output file. So when using alternative tools we cannot simply redirect output to a file as that will miss the exit code.
The feature to expect non-null exit code is only used by a handful of benchmarks/tests and is not critical. This gets rid of the feature and the complexity and simply always expects an exit code of 0 in the future.
- Changes some benchmark code to return a 0 exit code in the code paths used by our testing:
- MultiSource/Applications/minisat/Main.cpp
- MultiSource/Applications/spiff/spiff.c
- MultiSource/Benchmarks/MiBench/office-ispell/ispell.c
- MultiSource/Benchmarks/Prolangs-C/archie-client/archie.c
- MultiSource/Benchmarks/Prolangs-C/football/stats.c
- SingleSource/UnitTests/testcase-Expr-1.c
- SingleSource/UnitTests/testcase-ExprConstant-1.c
- Change test inputs to not end up in error state:
- MultiSource/Benchmarks/Prolangs-C/gnugo/stdin
- Adds a DO_NOT_RUN option and uses it for SingleSource/UnitTests/Matrix/AMXINT8 (instead of expecting an "illegal opcode" exit status)
- Allows us to get rid of the "traditional_output" setting that was used to enable/disable the behavior of timeit to add the exit code.