[Fortran/gfortran] Enable "compile" tests from the gfortran torture test suite

[tarunprabhu]

Summary

The gfortran testsuite consists of "compile" and "execute" tests. The former test the behavior of the compiler while the latter test the executable produced by the compiler. Most of the "compile" tests don't produce an executable that can be run at test-time.

This patch enables them by working around some of the expected behavior of this test suite.

The tests are compiled along with the rest of the tests at build-time. Some of the "compile" tests are expected to raise an error. To work around this, at build-time, the compiler is invoked through a wrapper script which saves any diagnostics emitted by the compiler and always returns success. If the compilation succeeded, any object files produced are deleted immediately.

At test-time, the diagnostics saved at build-time are compared against the expected output. .

In addition, since the test suite expects to be able to run an executable at test-time, a dummy executable that does nothing is generated at build-time which is invoked for all tests at run-time.

[kiranchandramohan]

Thanks @tarunprabhu for this patch. Is this only enabling the compile tests in the torture directory? Would we have to do something similar for compile tests in the regression/gomp directory for example.

[tarunprabhu]

In D149116#4298539, @kiranchandramohan wrote:

Thanks @tarunprabhu for this patch. Is this only enabling the compile tests in the torture directory? Would we have to do something similar for compile tests in the regression/gomp directory for example.

Yes. Enabling all the tests in the regression is what I intend to do next.

A major reason for just doing the "compile" torture tests first was to get feedback on the approach i.e. building the tests along with the "execute" tests, saving the compiler diagnostics (if any) and verifying those diagnostics at testing-time.

[luporl]

Thanks @tarunprabhu for this patch and for your work on porting gfortran
test-suite to LLVM.

It would really be better if llvm-test-suite could be improved to support
compile tests in a clean way, to avoid the need of workarounds that make
compile tests look like execute tests.
But I am not familiar with llvm-test-suite. Do you have an idea of
how much effort would be required?

With that said and assuming the required effort would be high, your
patch LGTM and IMHO we can use it while llvm-test-suite support isn't
ready, as it enables us to run several more tests.

It would also be nice to update Fortran/gfortran/README.md, unless you
are planning to do it later, when more compile tests are supported.

Fortran/gfortran/compile-save-diags.cmake

13

s/wells/well/

40

The if and elseif can be merged. Something like:
if (ALWAYS_SAVE_DIAGS OR NOT "${RETVAR}" EQUAL "0")

Fortran/gfortran/torture/compile/CMakeLists.txt

13

s/compier/compiler/

97

Do you mean "only saved on failure"?

159–162

Is it expected that UnsupportedTests/UnimplementedTests/SkippedTests will receive entries when more compile tests are enabled?

[tarunprabhu]

Added a note in Fortran/gfortran README.md indicating that the compile tests have been enabled for the torture suite.

[tarunprabhu]

Thanks for the review, @luporl

In D149116#4343386, @luporl wrote:

But I am not familiar with llvm-test-suite. Do you have an idea of
how much effort would be required?

Yes, I agree that modifying the test suite would be a better idea in the long run. However, I, too, am not very familiar with the llvm-test-suite so I am not sure just how much work it would be. I wanted to just get something working for now so we have a larger suite of tests that the community can use.

I could look into the modifying the test-suite itself, but I would like to prioritize getting most of the gfortran test-suite working first.

It would also be nice to update Fortran/gfortran/README.md, unless you are planning to do it later, when more compile tests are supported.

Done.

Fortran/gfortran/torture/compile/CMakeLists.txt

159–162

Yes, it is possible if gfortran adds more tests to this list and if they exercise features that we don't support, for instance. The idea is for these tests to be periodically synced with gfortran so we can take advantage of any more tests that they add.

[luporl]

In D149116#4352863, @tarunprabhu wrote:

In D149116#4343386, @luporl wrote:

But I am not familiar with llvm-test-suite. Do you have an idea of
how much effort would be required?

Yes, I agree that modifying the test suite would be a better idea in the long run. However, I, too, am not very familiar with the llvm-test-suite so I am not sure just how much work it would be. I wanted to just get something working for now so we have a larger suite of tests that the community can use.

I could look into the modifying the test-suite itself, but I would like to prioritize getting most of the gfortran test-suite working first.

I'm ok with getting most of the gfortran test-suite working first.

What happened to Fortran/gfortran/compile-save-diags.cmake file? It seems it was deleted in the last diff.

Fortran/gfortran/torture/compile/CMakeLists.txt

159–162

Ok, it makes sense.

[tarunprabhu]

Ugh! Sorry, was in a hurry and didn't notice that one of the files was missing in the last update. This is the correct diff.

[tblah]

Fortran/gfortran/README.md

45–50

After this patch, compile tests are supported in torture.

[tarunprabhu]

Fixed README to indicate that the compile tests are enabled in torture.

[luporl]

LGTM, but it's better to wait for someone with more experience in this area to approve.

[kiranchandramohan]

LG.

Just want to repeat the following I saw in the other patch.
I see two modules M1 and TEST that are repeated in torture/execute. One of the tests defining TEST is disabled so we might not be hitting the issue for this one but for M1 we might end up hitting the issue. Should we do something about the M1 tests?

$ grep -i 'module m1' llvm-test-suite/Fortran/gfortran/torture/execute/*
llvm-test-suite/Fortran/gfortran/torture/execute/function_module_1.f90:module M1
llvm-test-suite/Fortran/gfortran/torture/execute/module_init_1.f90:module m1

$ grep -i 'module test' llvm-test-suite/Fortran/gfortran/torture/execute/*
llvm-test-suite/Fortran/gfortran/torture/execute/pr32140.f90:MODULE TEST
llvm-test-suite/Fortran/gfortran/torture/execute/pr32604.f90:MODULE TEST

[kiranchandramohan]

In D149116#4368887, @kiranchandramohan wrote:

LG.

Just want to repeat the following I saw in the other patch.
I see two modules M1 and TEST that are repeated in torture/execute. One of the tests defining TEST is disabled so we might not be hitting the issue for this one but for M1 we might end up hitting the issue. Should we do something about the M1 tests?

$ grep -i 'module m1' llvm-test-suite/Fortran/gfortran/torture/execute/*
llvm-test-suite/Fortran/gfortran/torture/execute/function_module_1.f90:module M1
llvm-test-suite/Fortran/gfortran/torture/execute/module_init_1.f90:module m1

$ grep -i 'module test' llvm-test-suite/Fortran/gfortran/torture/execute/*
llvm-test-suite/Fortran/gfortran/torture/execute/pr32140.f90:MODULE TEST
llvm-test-suite/Fortran/gfortran/torture/execute/pr32604.f90:MODULE TEST

Disabling one of the module M1 tests is a reasonable workaround for now.

Fortran/gfortran/README.md

 ## Tests from the gfortran test suite ##
 
 This is the top-level directory for tests imported from
 [GCC](https://github.com/gcc-mirror/gcc). The test files are contained within
 two subdirectories:
 
     - `regression`: Contains the gfortran [tests](https://github.com/gcc-mirror/gcc/tree/master/gcc/testsuite/gfortran.dg).
     - `torture`: Contains the gfortran [torture tests](https://github.com/gcc-mirror/gcc/tree/master/gcc/testsuite/gfortran.fortran-torture).
 
 The tests in both `regression` (and its subdirectories) and `torture` can be
 classified roughly as _compile_ tests and _execute_ tests. The _compile_ tests
 generally check the compiler's error/warning messages and, in some cases,
 optimization logs. The _execute_ tests are end-to-end tests that check the
 behavior of the binary produced by the compiler.
 
-Currently, only the _execute_ tests are supported in both `regression` and
-`torture`.
+Currently, only the _execute_ tests are supported in `regression`. Both 
+`compile` and `execute` tests have been enabled in `torture`.
 
 Of the supported tests, a number of tests have been disabled. There are four
 categories of such tests:
 
     - *Unsupported*: These are tests that use non-standard extensions/intrinsics
     that are not currently supported by flang. Unless those non-standard
     features are supported in the future, these tests will never be enabled.
 

 
 Over time, the number of tests in the *unimplemented*, *skipped*, and *failing*
 categories should decrease. Eventually, only the *unsupported* category should
 remain.
 
 
 ### _Compile_ tests ###
 
-These tests are currently not supported.
+The _compile_ tests are "built" when the whole test suite is built at which 
+time a compilation log is saved. At testing time, the log is checked to 
+determine whether the test should pass or fail. If the test is expected to pass,
+but the compilation log contains errors, the test will be deemed to have failed
+and vice versa. The _compile_ test are supported in `torture`, but not in
+`regression`.
 
 
 ### _Execute tests_ ###
 
 The _execute_ tests are built when the whole test suite is built and executed
 when the tests are run. The *unsupported*, *unimplemented*, and *skipped* tests
 fail to build for the reasons described above. The *failing* tests do build.
 

Fortran/gfortran/compile-save-diags.cmake

+#===------------------------------------------------------------------------===#
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===------------------------------------------------------------------------===#
+
+# Run the compiler and save the compiler diagnostics to file. The diagnostics
+# are simply the messages printed to stdout and stderr. This is intended to be
+# used for the "compile" tests in this test. The "compile" tests exercise the
+# compiler's parser and semantic analyzer as well as the diagnostics.
+
+# Required parameters
+#
+#     COMPILER: STRING            Path to the compiler.
+#     COMPILER_FLAGS: STRING      Compiler flags.
+#     INPUT_FILES: STRING         Space separated list of files to compile.
+#     OUTPUT_FILE: STRING         The output diagnostics file.
+#     OBJECT_FILE: STRING         The object file produced by the compiler. This
+#                                 will be deleted before this script exits.
+#
+# Optional parameters
+#
+#     ALWAYS_SAVE_DIAGS: BOOLEAN  If true, diagnostics will be saved even if no
+#                                 error occured when compiling. Otherwise,
+#                                 diagnostics will only be saved if compilation
+#                                 failed for any reason.
+#
+
+execute_process(COMMAND ${COMPILER} ${COMPILER_FLAGS} ${INPUT_FILES} -o ${OBJECT_FILE}
+  RESULT_VARIABLE RETVAR
+  OUTPUT_VARIABLE OUTVAR
+  ERROR_VARIABLE ERRVAR
+  OUTPUT_STRIP_TRAILING_WHITESPACE
+  ERROR_STRIP_TRAILING_WHITESPACE)
+
+file(WRITE "${OUTPUT_FILE}" "")
+if (ALWAYS_SAVE_DIAGS OR NOT "${RETVAR}" EQUAL "0")
+  file(APPEND "${OUTPUT_FILE}" "${ERRVAR}")
+  file(APPEND "${OUTPUT_FILE}" "")
+  file(APPEND "${OUTPUT_FILE}" "${OUTVAR}")
+endif()
+
+file(REMOVE ${OBJECT_FILE})

Fortran/gfortran/torture/compile/CMakeLists.txt

 #===------------------------------------------------------------------------===#
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 #
 #===------------------------------------------------------------------------===#
 
-# It is not clear how we can use the compile tests within the test suite. It
-# would be good to modify the test suite to be able to use it somehow, but for
-# now, there is nothing we can do.
+# The test-suite expects an executable to be produced at build time and for
+# that executable to be run at test time. The result (in the form of the
+# return code or the output written to stdout/stderr) is used to determine
+# whether the test has succeeded. The "compile" tests are intended to exercise
+# the behavior of the compiler itself. There isn't a clean way of having the
+# compiler be executed at test time. Instead, the compiler is run at
+# build time and the diagnostics/errors saved to a file as needed. This file is
+# compared to a reference output at test time to determine success/failure of
+# the test. A dummy executable is also built. This does nothing, but provides
+# something that the test suite can "run" at test time.
+
+# Creates a test from each valid test file in the current source directory. Each
+# argument to a function is a list. If a test file is in any of the lists, a
+# test will not be created from it.
+function(add_tests UnsupportedTests UnimplementedTests SkippedTests FailingTests)
+  # This will just get all the Fortran source files in the directory. The tests
+  # in this directory are all single-source.
+  file(GLOB AllFiles CONFIGURE_DEPENDS LIST_DIRECTORIES false
+    *.f*
+    *.F*
+  )
+
+  set(TestsToSkip)
+
+  # There is still a chance that some of the unsupported tests may need to be
+  # enabled, for instance if the non-standard extensions that they exercise are
+  # supported due to user demand.
+  if (NOT TEST_SUITE_FORTRAN_FORCE_ALL_TESTS AND
+      NOT TEST_SUITE_FORTRAN_FORCE_UNSUPPORTED_TESTS)
+    list(APPEND TestsToSkip ${UnsupportedTests})
+  endif()
+
+  # For the remaining tests, there is cause to build and run the skipped, failing
+  # and unimplemented tests since some could be enabled once some feature is
+  # implemented. Eventually, all the TEST_SUITE_FORTRAN_FORCE_* options (perhaps
+  # with the exception of TEST_SUITE_FORTRAN_FORCE_UNSUPPORTED_TESTS) should
+  # become redundant and removed.
+  if (NOT TEST_SUITE_FORTRAN_FORCE_ALL_TESTS AND
+      NOT TEST_SUITE_FORTRAN_FORCE_UNIMPLEMENTED_TESTS)
+    list(APPEND TestsToSkip ${UnimplementedTests})
+  endif()
+
+  if (NOT TEST_SUITE_FORTRAN_FORCE_ALL_TESTS AND
+      NOT TEST_SUITE_FORTRAN_FORCE_FAILING_TESTS)
+    list(APPEND TestsToSkip ${FailingTests})
+  endif()
+
+  if (NOT TEST_SUITE_FORTRAN_FORCE_ALL_TESTS AND
+      NOT TEST_SUITE_FORTRAN_FORCE_SKIPPED_TESTS)
+    list(APPEND TestsToSkip ${SkippedTests})
+  endif()
+
+  foreach(TestToSkip ${TestsToSkip})
+    list(REMOVE_ITEM AllFiles ${TestToSkip})
+  endforeach()
+
+  # The program to be used to verify the results. The programs here should take
+  # two files as arguments, return 0 if the files are identical, non-zero
+  # otherwise.
+  set(DIFFPROG)
+  if (WIN32)
+    find_program(DIFFPROG
+      NAMES fc.exe
+      REQUIRED)
+  else ()
+    find_program(DIFFPROG
+      NAMES diff cmp
+      REQUIRED)
+  endif ()
+
+  # The file prefix is needed because there are several tests with the same
+  # file name across the gfortran test suite. cmake prefers all targets to be
+  # unique, so they get prefixed with this.
+  set(FilePrefix "gfortran-torture-compile")
+
+  # The test suite expects to be able to run something at testing time. For the
+  # compile tests, there is nothing to be run. While a better solution will be
+  # to modify the test suite to allow for cases like this, as a temporary
+  # measure, just create an empty executable that will be run for each test.
+  set(DummySrc ${CMAKE_CURRENT_BINARY_DIR}/dummy.f90)
+  set(Dummy "dummy")
+
+  file(WRITE ${DummySrc} "program test\nend program test")
+  add_executable(${Dummy} ${DummySrc})
+  # At some point, the -flang-experimental-exec flag will be removed.
+  target_link_options(${Dummy} PUBLIC "-flang-experimental-exec")
+
+  # All the "compile" tests in the gfortran torture tests are expected to
+  # pass. Since diagnostics are only saved on failure, the diagnostics
+  # file produced when compiling the test should be empty. An empty file can,
+  # therefore, be used as reference output.
+  set(Reference "${FilePrefix}-empty.reference.out")
+  add_custom_command(
+    OUTPUT ${Reference}
+    COMMAND ${CMAKE_COMMAND} -E touch ${CMAKE_CURRENT_BINARY_DIR}/${Reference}
+    VERBATIM
+    USES_TERMINAL
+    COMMENT "Creating reference output file"
+  )
+
+  # The compile script compiles the files and may save the diagnostics to file
+  # as needed (see the options that the script accepts).
+  set(COMPILE_SCRIPT
+    ${CMAKE_SOURCE_DIR}/Fortran/gfortran/compile-save-diags.cmake)
+
+  foreach(File ${AllFiles})
+    get_filename_component(FileName ${File} NAME)
+
+    set(Out ${FileName}.out)
+    set(Obj ${FileName}.o)
+
+    # ${Exe} is just used as a custom target name. Nevertheless, it needs to be
+    # unique. There are multiple files with the same name but different
+    # extensions in this directory. Retain the extension, but replace the
+    # final '.' with an '_'.
+    string(REPLACE "." "_" Exe "${FilePrefix}_${FileName}")
+
+    set(Compiler -DCOMPILER=${CMAKE_Fortran_COMPILER})
+    set(CompilerFlags -DCOMPILER_FLAGS=-c)
+    set(InputFiles -DINPUT_FILES=${File})
+    set(ObjFile -DOBJECT_FILE=${Obj})
+    set(OutputFile -DOUTPUT_FILE=${Out})
+    set(AlwaysSaveDiags -DALWAYS_SAVE_DIAGS=OFF)
+
+    add_custom_command(
+      OUTPUT ${Out}
+      COMMAND ${CMAKE_COMMAND} ${Compiler} ${CompilerFlags} ${InputFiles} ${ObjFile} ${OutputFile} ${AlwaysSaveDiags} -P ${COMPILE_SCRIPT}
+      VERBATIM
+      USES_TERMINAL
+      COMMENT "Compiling ${File}")
+
+    add_custom_target(${Exe}
+      ALL
+      DEPENDS ${Out} ${Reference} ${Dummy}
+      SOURCES ${File})
+
+    llvm_test_run(EXECUTABLE %S/${Dummy})
+    llvm_test_verify(${DIFFPROG} %S/${Reference} %S/${Out})
+    llvm_add_test(${Exe}.test %S/${Dummy})
+  endforeach()
+endfunction()
+
+# These tests are disabled because they fail, when they should pass.
+file(GLOB Failing CONFIGURE_DEPENDS
+  # For this, gfortran issues a warning while flang fails to compile. This is
+  # potentially an "unsupported" test if the reason for this difference in
+  # is because the standard allows for implementation-dependent behavior.
+  pr37236.f
+)
+
+list(APPEND UnsupportedTests "")
+list(APPEND UnimplementedTests "")
+list(APPEND SkippedTests "")
+list(APPEND FailingTests "${Failing}")
+
+add_tests("${UnsupportedTests}" "${UnimplementedTests}" "${SkippedTests}" "${FailingTests}")