[test-suite] Disable Fortran FM509 test when -std=legacy is not supported

[DavidSpickett]

Summary

This test requires -std=legacy which is not supported by flang.
Currently we check if the flag is supported and add if so,
but run the test anyway if not supported.

This has been failing on our bots for a long time but was not
noticed due to some issues reporting NOEXE failures.

https://lab.llvm.org/buildbot/#/builders/179/builds/3132

Executable Missing Tests (1):

test-suite :: Fortran/UnitTests/fcvs21_f95/FM509.test

This fixes llvm_singlesource to allow you to pass a list
of source files (which used to work when it was a macro).

(this was done for multisource in cecca649c310cfa78076ef96dadea315ac176efa)

If the flag is not supported we remove the test from the list
of source files. This means it's disabled for flang and
enabled for gfortran.

[awarzynski]

@DavidSpickett , many thanks for looking into this!

I think that it would be worth clarifying (perhaps in CMake) that this test compiles fine with gfortran -std=legacy FM509.f and that (as far as I know) there are no intentions to add such flag in LLVM Flang. Also - how about running the test suite with gfortran? If that's to be supported then perhaps the test should be only XFAIL'ed for Flang?

AFAIK, @naromero77 no longer works in this area. I included @Meinersbur as a reviewer as. I know that Michael helped Nick when adding these tests, so might have some suggestions too.

[DavidSpickett]

Make it more clear that the test is valid if you have -std=legacy
but that's not planned to be possible with flang.

[DavidSpickett]

Also - how about running the test suite with gfortran?

I assume that somewhere in gnu land this suite is run with gfortran but yes I don't see why you couldn't take this exact copy of the tests and run them with it. Ideally we'd preserve that.

The GCC torture tests do have skip lists I could adapt that to this: https://reviews.llvm.org/D115994

[DavidSpickett]

Instead of removing the test file just don't add it if the flag
is not supported.

[awarzynski]

Haha, great minds @DavidSpickett ! Just when I was to comment that this would be my preferred approach :) Thanks for doing this!

This makes a lot of sense and IMO can be merged as is. My comments are nits.

Fortran/UnitTests/fcvs21_f95/CMakeLists.txt

36–39

cmake/modules/SingleMultiSource.cmake

17–19

18

29–30

35

It took me a while to grasp what's going on here, so I've suggested expanding the comment :) WDYT?

[DavidSpickett]

cmake/modules/SingleMultiSource.cmake

35

I feel like I evolved to a higher plane once I understood cmake functions :)

So yes I'll expand this.

[DavidSpickett]

Update comments.

[DavidSpickett]

Turns out it the "if (DEFINED Source" did work I just managed to miss the other users of it and none of them did anything on AArch64. Caused at least one failure on armv7 so I've reverted and will fix tomorrow.

[DavidSpickett]

Reland worked, test is now disabled for flang builds.

Fortran/UnitTests/fcvs21_f95/CMakeLists.txt

 # 
 # Note that the NIST license allows redistribution but requires
 # explicit acknowledgement of what has been changed. Please
 # amend README file if you modify the tests.
 
 # clean-up fort.* files otherwise one of the tests will fail
 llvm_test_prepare(rm -f %S/fort.*)
 
-# Test 509 requires this flag in more recent versions of GCC
+file(GLOB Source CONFIGURE_DEPENDS *.f)
+
+# Test 509 requires this flag in more recent versions of GCC.
+# If this flag is not supported, the test should be excluded.
 check_fortran_compiler_flag("-std=legacy" SUPPORTS_LEGACY)
 if (SUPPORTS_LEGACY)
   add_compile_options(-std=legacy)
-endif ()
+else()
+  # Regex because the GLOB returns the full path to each file
+  list(FILTER Source EXCLUDE REGEX "FM509\.f$")
+endif()
 
 set(FP_TOLERANCE 1.0e-11) # set by the most sensitive numerical test
-llvm_singlesource()
+llvm_singlesource(${Source})
 
 file(COPY lit.local.cfg DESTINATION "${CMAKE_CURRENT_BINARY_DIR}")

cmake/modules/SingleMultiSource.cmake

 ##===- SingleMultiSource.cmake --------------------------------------------===##
 #
 # This defines functions to be used by the SingleSource and MultiSource
 # directories. This style of defining benchmarks is considered obsolete now
 # and it is recommended to only use the functions in TestSuite.cmake and
 # TestFile.cmake instead.
 #
 # Defines helpers to add executables and tests. The entry points to this
 # file are:
-#   `llvm_singlesource([PREFIX p])`, and
+#   `llvm_singlesource([PREFIX p] [sources...])`, and
 #   `llvm_multisource()`
 #
 # Following convenience functions provide shortcuts for common test cases:
 #
-# llvm_singlesource([PREFIX p])
+# llvm_singlesource([PREFIX p] [sources...])
 #
-#   Invokes llvm_test_executable() for each c/c++ source file.  If
-#   'sources is emptyno sources are specified, creates test executables
-#   for all C/C++ files in current directory.
+#   Invokes llvm_test_executable() for each C/C++/Fortran source file.
+#   If sources is empty, creates test executables for all C/C++/Fortran
+#   files in current directory.
 #   Passes optional PREFIX parameter to llvm_test_executable().
 #
 # llvm_multisource(target)
 #   Invokes llvm_test_executable(${target} [sources...])
 #
 ##===----------------------------------------------------------------------===##
 
 include(TestSuite)
 
 # Configure the current directory as a SingleSource subdirectory - i.e. every
-# file in *.{c,cpp,cc} is treated as its own test.
+# C/C++/Fortran file is treated as its own test.
 function(llvm_singlesource)
+  # PREFIX is optional so it cannot be a named argument.
   cmake_parse_arguments(_LSARG "" "PREFIX" "" ${ARGN})
-  if(DEFINED Source)
-    set(sources ${Source})
-  else()
+  if(DEFINED _LSARG_PREFIX)
+    # Remove "PREFIX" and the associated value from ARGN so that it only
+    # contains sources.
+    list(REMOVE_AT ARGN 0)
+    list(REMOVE_AT ARGN 0)
+  endif()
+
+  set(sources ${ARGN})
+  if(NOT sources)
     file(GLOB sources *.c *.cpp *.cc *.f *.F *.f90 *.F90)
   endif()
+
   foreach(source ${sources})
     basename(name ${source})
     set(_target ${_LSARG_PREFIX}${name})
     llvm_test_executable_no_test(${_target} ${source})
     set_property(TARGET ${_target} PROPERTY TEST_NAME ${name})
     llvm_test_traditional(${_target})
     llvm_add_test_for_target(${_target})
   endforeach()