With this patch all the tests specifying an FP_TOLERANCE or FP_ABSTOLERANCE will be run in two stages:
- first step is with with extra flags "-ffp-contract=off -fno-fast-math", and recording the full output that will be checked against in the second step
- second step is with the default flags specified by the user, and checks against the output recorded in the first step with the FP_TOLERANCE specified by the test.
There are at least two main problems with this approach:
- some of the current golden files in the test-suite require testing with FP_TOLERANCE even for stage1
- the build directory goes from 300M to 1.2G due to the extra reference outputs recorded by stage1, and for at least 2 tests (polybench/bicg, polybench/mvt disabled in the current patch) fpcmp takes more than 10 minutes to completely parse the outputs of stage1 and 2. When running test-suite over small devices will cost 1G more transfer over the network.
The current implementation has another smaller technical issue:
lit does not allow to specify dependences between tests.
When running "lit -j8" one can end up running stage 2 before stage 1.
The solution I was considering is to run step 1 as a PREPARE: step,
though that would fail to add the exit code in the output,
and that will end up mismatching the hash.