[test-suite] NIST Fortran Compiler Validation Suite.

[naromero77]

Summary

Update README to reflect modifications to test-suite.

Support input from standard-in in llvm_singlesource function.

Modify Fortran IO to allow tests to run in parallel.

Include source directory in path to reference input.

[Meinersbur]

Could you delete unused driver_parse and driver_run?

cmake/modules/SingleMultiSource.cmake

87

Consider

list(APPEND RUN_OPTIONS < "${name}.reference_input")

[naromero77]

In D104150#2814417, @Meinersbur wrote:

Could you delete unused driver_parse and driver_run?

Neither are used by the test-suite.

However, if anyone ever needed to update the reference results, the driver_run script would come in handy.

We could get rid of driver_parse, since it is not needed at all and was just part of the test-suite.

[naromero77]

• More robust way to get rid of fort.* files.
• More compact code that achieves the same result.
• Remove driver_parse script.

[rovka]

Is this expected to pass with current upstream flang?
I gave this a quick spin on an AArch64 machine similar to our buildbots (same hardware, not necessarily same ubuntu version, system compiler etc) and I'm seeing one failure in FM509:

/path/to/build/bin/flang -DNDEBUG  -w -Werror=date-time -c /path/to/llvm-test-suite/Fortran/UnitTests/fcvs21_f95/FM509.f -o CMakeFiles/FM509.dir/FM509.f.o
f18-c234.f90:237:12:

  237 |  CALL sn512(c1n001(5)(2:9), cvcomp)
      |            1
Error: Actual argument contains too few elements for dummy argument ‘c1d001’ (19/48) at (1)
execvp(gfortran) failed: 
flang: in /path/to/test-suite-sandbox/build/Fortran/UnitTests/fcvs21_f95, f18 failed with exit status 0: /path/to/build/bin/f18 -module-suffix .f18.mod -DNDEBUG -w -Werror=date-time -c /path/to/llvm-test-suite/Fortran/UnitTests/fcvs21_f95/FM509.f -o CMakeFiles/FM509.dir/FM509.f.o

I'm attaching the produced f18-c234.f90 in case anyone wants to have a look.

f18-c234.f909 KBDownload

[naromero77]

In D104150#2818673, @rovka wrote:

Is this expected to pass with current upstream flang?
I gave this a quick spin on an AArch64 machine similar to our buildbots (same hardware, not necessarily same ubuntu version, system compiler etc) and I'm seeing one failure in FM509:

/path/to/build/bin/flang -DNDEBUG  -w -Werror=date-time -c /path/to/llvm-test-suite/Fortran/UnitTests/fcvs21_f95/FM509.f -o CMakeFiles/FM509.dir/FM509.f.o
f18-c234.f90:237:12:

  237 |  CALL sn512(c1n001(5)(2:9), cvcomp)
      |            1
Error: Actual argument contains too few elements for dummy argument ‘c1d001’ (19/48) at (1)
execvp(gfortran) failed: 
flang: in /path/to/test-suite-sandbox/build/Fortran/UnitTests/fcvs21_f95, f18 failed with exit status 0: /path/to/build/bin/f18 -module-suffix .f18.mod -DNDEBUG -w -Werror=date-time -c /path/to/llvm-test-suite/Fortran/UnitTests/fcvs21_f95/FM509.f -o CMakeFiles/FM509.dir/FM509.f.o

I'm attaching the produced f18-c234.f90 in case anyone wants to have a look.

f18-c234.f909 KBDownload

Sorry for the slow reply. I am on vacation this week and will return to a regular schedule next week. I had no failures with an GCC 7.5, but did have one failure with GCC 10.2 with this same test. Are you using the Flang throw away driver? If so, which GCC are you using.

It would be pretty easy to remove this test if we decided that is the best thing to do.

[Meinersbur]

In D104150#2814958, @naromero77 wrote:

In D104150#2814417, @Meinersbur wrote:

Could you delete unused driver_parse and driver_run?

However, if anyone ever needed to update the reference results, the driver_run script would come in handy.

Does driver_run come from the NIST Validation suite? Generally, we would not to support multiple means to run tests (llvm-lit and driver_run).

If its only purpose is to update reference_output files, it should make that clear in comments.
There is the possibility to extend llvm-lit to update all reference-output (instead of comparing to it) without each benchmark having its own script.

[naromero77]

In D104150#2825117, @Meinersbur wrote:

In D104150#2814958, @naromero77 wrote:

In D104150#2814417, @Meinersbur wrote:

Could you delete unused driver_parse and driver_run?

However, if anyone ever needed to update the reference results, the driver_run script would come in handy.

Does driver_run come from the NIST Validation suite? Generally, we would not to support multiple means to run tests (llvm-lit and driver_run).

If its only purpose is to update reference_output files, it should make that clear in comments.
There is the possibility to extend llvm-lit to update all reference-output (instead of comparing to it) without each benchmark having its own script.

Yes, driver_run is only for updating reference_output files. I do mention this both in the README and CMakeLists.txt. Let me know if its not explicit enough.

[naromero77]

• -std=legacy flag needed by newer versions of GCC.

[rovka]

In D104150#2824793, @naromero77 wrote:

In D104150#2818673, @rovka wrote:

Is this expected to pass with current upstream flang?
I gave this a quick spin on an AArch64 machine similar to our buildbots (same hardware, not necessarily same ubuntu version, system compiler etc) and I'm seeing one failure in FM509:

/path/to/build/bin/flang -DNDEBUG  -w -Werror=date-time -c /path/to/llvm-test-suite/Fortran/UnitTests/fcvs21_f95/FM509.f -o CMakeFiles/FM509.dir/FM509.f.o
f18-c234.f90:237:12:

  237 |  CALL sn512(c1n001(5)(2:9), cvcomp)
      |            1
Error: Actual argument contains too few elements for dummy argument ‘c1d001’ (19/48) at (1)
execvp(gfortran) failed: 
flang: in /path/to/test-suite-sandbox/build/Fortran/UnitTests/fcvs21_f95, f18 failed with exit status 0: /path/to/build/bin/f18 -module-suffix .f18.mod -DNDEBUG -w -Werror=date-time -c /path/to/llvm-test-suite/Fortran/UnitTests/fcvs21_f95/FM509.f -o CMakeFiles/FM509.dir/FM509.f.o

I'm attaching the produced f18-c234.f90 in case anyone wants to have a look.

f18-c234.f909 KBDownload

Sorry for the slow reply. I am on vacation this week and will return to a regular schedule next week. I had no failures with an GCC 7.5, but did have one failure with GCC 10.2 with this same test. Are you using the Flang throw away driver? If so, which GCC are you using.

No worries :) I'm using GCC 9.3.0.

It would be pretty easy to remove this test if we decided that is the best thing to do.

For now it would be best to investigate :)

If the test looks fishy (e.g. isn't conformant to the standard or uses obsolete features that we're not interested in), then of course we can remove it.
If it looks sensible and the issue is in gfortran, then we should commit the test and mark it as XFAIL for certain versions of GCC (looks like probably >= 9.3.0). I don't know exactly how that's done but we can probably look for other examples in the test-suite. If the issue is in flang itself, then it's best to open a bug report and commit the test as XFAIL until the bug is fixed.

TL;DR: We shouldn't block this patch just because that test is failing, but we don't necessarily have to remove it either.

[naromero77]

In D104150#2826484, @rovka wrote:

In D104150#2824793, @naromero77 wrote:

In D104150#2818673, @rovka wrote:

Is this expected to pass with current upstream flang?
I gave this a quick spin on an AArch64 machine similar to our buildbots (same hardware, not necessarily same ubuntu version, system compiler etc) and I'm seeing one failure in FM509:

/path/to/build/bin/flang -DNDEBUG  -w -Werror=date-time -c /path/to/llvm-test-suite/Fortran/UnitTests/fcvs21_f95/FM509.f -o CMakeFiles/FM509.dir/FM509.f.o
f18-c234.f90:237:12:

  237 |  CALL sn512(c1n001(5)(2:9), cvcomp)
      |            1
Error: Actual argument contains too few elements for dummy argument ‘c1d001’ (19/48) at (1)
execvp(gfortran) failed: 
flang: in /path/to/test-suite-sandbox/build/Fortran/UnitTests/fcvs21_f95, f18 failed with exit status 0: /path/to/build/bin/f18 -module-suffix .f18.mod -DNDEBUG -w -Werror=date-time -c /path/to/llvm-test-suite/Fortran/UnitTests/fcvs21_f95/FM509.f -o CMakeFiles/FM509.dir/FM509.f.o

I'm attaching the produced f18-c234.f90 in case anyone wants to have a look.

f18-c234.f909 KBDownload

Sorry for the slow reply. I am on vacation this week and will return to a regular schedule next week. I had no failures with an GCC 7.5, but did have one failure with GCC 10.2 with this same test. Are you using the Flang throw away driver? If so, which GCC are you using.

No worries :) I'm using GCC 9.3.0.

Can you try running it again with the latest version of the patch? It should compile and pass now. This test needs the -std=legacy flag.

It would be pretty easy to remove this test if we decided that is the best thing to do.

For now it would be best to investigate :)

If the test looks fishy (e.g. isn't conformant to the standard or uses obsolete features that we're not interested in), then of course we can remove it.

It kind of looks like this type of thing:
https://gcc.gnu.org/gcc-10/porting_to.html

Except that the

-fallow-argument-mismatch

does not actually help, you have to use

-std=legacy

It looks like it might be non-conformant, but I am not 100% sure.

If it looks sensible and the issue is in gfortran, then we should commit the test and mark it as XFAIL for certain versions of GCC (looks like probably >= 9.3.0). I don't know exactly how that's done but we can probably look for other examples in the test-suite. If the issue is in flang itself, then it's best to open a bug report and commit the test as XFAIL until the bug is fixed.

TL;DR: We shouldn't block this patch just because that test is failing, but we don't necessarily have to remove it either.

[rovka]

This passes now, thanks :)

[Meinersbur]

In D104150#2826222, @naromero77 wrote:

Yes, driver_run is only for updating reference_output files. I do mention this both in the README and CMakeLists.txt. Let me know if its not explicit enough.

Please also mention that in driver_run itself, especially that it is not meant to run the test-suite (unlike the file name implies)

[naromero77]

• More explicit comment about the purpose and usage of driver_run script.

[Meinersbur]

Fortran/UnitTests/fcvs21_f95/driver_run

4–6

[naromero77]

• Comments about the use of the driver_run script moved from CMakeLists.txt to driver_run.

[Meinersbur]

LGTM

[DavidSpickett]

Fortran/UnitTests/fcvs21_f95/driver_run

49

This will not stop the script if it fails, is that intentional?

Some tests in this suite are now failing on our bot:
https://lab.llvm.org/buildbot/#/builders/184/builds/109

And I'm trying to work out why, there's a lot in the stdio like:

cd /home/tcwg-buildslave/worker/clang-aarch64-sve-vls/test/sandbox/build/Fortran/UnitTests/hello && /usr/local/bin/cmake -E cmake_link_script CMakeFiles/hello.dir/link.txt --verbose=1
f951: Error: Can't open file ‘flang_unparsed_source_file_0.f90’
<built-in>: Fatal Error: can't open input file: flang_unparsed_source_file_0.f90
compilation terminated.
gfortran: error: flang_unparsed_source_file_0.f90: No such file or directory
gfortran: fatal error: no input files
compilation terminated.
flang: in /home/tcwg-buildslave/worker/clang-aarch64-sve-vls/test/sandbox/build/Fortran/UnitTests/fcvs21_f95, gfortran failed with exit status 0: gfortran -O3 -O3 -w -Werror=date-time -c -DNDEBUG -O3 -DNDEBUG -O3 -w -Werror=date-time -c /home/tcwg-buildslave/worker/clang-aarch64-sve-vls/test/test-suite/Fortran/UnitTests/fcvs21_f95/FM816.f -o CMakeFiles/FM816.dir/FM816.f.o
f951: Error: Can't open file ‘flang_unparsed_source_file_0.f90’
Dependee "/home/tcwg-buildslave/worker/clang-aarch64-sve-vls/test/sandbox/build/Fortran/UnitTests/fcvs21_f95/CMakeFiles/FM026.dir/DependInfo.cmake" is newer than depender "/home/tcwg-buildslave/worker/clang-aarch64-sve-vls/test/sandbox/build/Fortran/UnitTests/fcvs21_f95/CMakeFiles/FM026.dir/depend.internal".
Dependee "/home/tcwg-buildslave/worker/clang-aarch64-sve-vls/test/sandbox/build/Fortran/UnitTests/fcvs21_f95/CMakeFiles/CMakeDirectoryInformation.cmake" is newer than depender "/home/tcwg-buildslave/worker/clang-aarch64-sve-vls/test/sandbox/build/Fortran/UnitTests/fcvs21_f95/CMakeFiles/FM026.dir/depend.internal".
flang: in /home/tcwg-buildslave/worker/clang-aarch64-sve-vls/test/sandbox/build/Fortran/UnitTests/fcvs21_f95, gfortran failed with exit status 0: gfortran -O3 -O3 -w -Werror=date-time -c -DNDEBUG -O3 -DNDEBUG -O3 -w -Werror=date-time -c /home/tcwg-buildslave/worker/clang-aarch64-sve-vls/test/test-suite/Fortran/UnitTests/fcvs21_f95/FM808.f -o CMakeFiles/FM808.dir/FM808.f.o
<built-in>: Fatal Error: can't open input file: flang_unparsed_source_file_0.f90
compilation terminated.

(also "failed with exit status 0" feels like someone is forgetting to read the status code but that's off topic)

[DavidSpickett]

Fortran/UnitTests/fcvs21_f95/driver_run

49

Doh, I should read more carefully!

# `driver_run` script is NOT intended to used for running the LLVM
# test-suite, nor is it invoked by any of the LLVM test-suite

Which means it's nothing to do with what I'm seeing.

Fortran/UnitTests/fcvs21_f95/driver_run

+#! /bin/sh -u
+
+# `driver_run` script is NOT intended to used for running the LLVM
+# test-suite, nor is it invoked by any of the LLVM test-suite
+# infrastructure. It is provided for convenience for the specific
+# purpose of updating the reference_output files should the need to do
+# so arise.
+
+# Fortran compiler name
+FC='gfortran'
+# option to produce executable name
+EXECOPT='-o'
+# run test 257 (that uses PAUSE)
+EXE257=1
+# do not run tests that include deleted features in Fortran 95
+AVOID_DELETED_F95=0
+
+#
+# main loop
+#
+for file in `ls FM???.f`
+do
+  process_it=1
+  #
+  # If requested, do not run tests that contain features
+  # deleted in Fortran 95
+  #
+  if [ x"${AVOID_DELETED_F95}" != x0 ]; then
+    if [ `grep -c '^[^C*].....[ ]*\bASSIGN\b' ${file}` -ne 0 ]; then
+      process_it=0
+    elif [ `grep -c '^[^C*].....[ ]*\bPAUSE\b' ${file}` -ne 0 ]; then
+      process_it=0
+    elif [ x"${file}" = x'FM719.f' ]; then
+      process_it=0
+    fi
+  fi
+
+  basename=`basename ${file} .f`
+
+  # If requested, run the test
+  if [ ${process_it} -eq 0 ]; then
+    echo "Skip ${basename}"
+  else
+    result="${basename}".reference_output
+    data="${basename}".reference_input
+
+    echo "Process ${basename}"
+
+    eval "${FC}" "${file}" "${EXECOPT}" a.out
+
+    # FM257 tests PAUSE
+    if [ x"${basename}" = xFM257 ]; then
+      if [ x"${EXE257}" = x0 ]; then
+        echo "Skip ${basename}"
+      else
+        {
+         cat <<__EOT__
+go
+
+
+
+__EOT__
+        } < /dev/null | \
+        ./a.out  > "${result}" &
+        pn=`ps | grep a.out | grep -v grep | sed 's/ *\([0-9]*\).*/\1/'`
+        kill -15 "${pn}"
+        kill -15 "${pn}"
+        kill -15 "${pn}"
+        kill -15 "${pn}"
+        kill -15 "${pn}"
+      fi
+    elif [ -f "${data}" ]; then
+      ./a.out < "${data}" > "${result}"
+    else
+      ./a.out > "${result}"
+    fi
+
+    # hard-coded to exit 0
+    echo "exit 0" >> "${result}"
+    
+    rm -f a.out
+  fi
+done
+
+rm -f fort.*
+echo "Finished."
+
+#
+# Possible error extraction
+#
+
+#grep -n 'FAIL' *.res | grep -v '0 TESTS FAILED' | grep -v 'TEST[ ]*PASS/FAIL'

cmake/modules/SingleMultiSource.cmake

 # HASH_PROGRAM_OUTPUT, etc.
 # Create llvm_test_run() and llvm_test_verify() invocation for that.
 function(llvm_test_traditional target)
   get_property(name TARGET ${target} PROPERTY TEST_NAME)
   if(NOT name)
     set(name ${target})
   endif()
 
+  # Find the reference input
+  if(EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/${name}.reference_input)
+    list(APPEND RUN_OPTIONS < ${CMAKE_CURRENT_SOURCE_DIR}/${name}.reference_input)
+  endif()
+
   # Always run in the same directory as the executable
   list(INSERT RUN_OPTIONS 0 WORKDIR ${CMAKE_CURRENT_BINARY_DIR})
   llvm_test_run(${RUN_OPTIONS})
 
   # Hash if we've been asked to.
   if(HASH_PROGRAM_OUTPUT)
     llvm_test_verify_hash_program_output(%o)
   endif()