This is an archive of the discontinued LLVM Phabricator instance.

Add tests from gfortran test suite [build files]
ClosedPublic

Authored by tarunprabhu on Mar 20 2023, 5:43 PM.

Details

Summary

Adds the build and license files for the gfortran test suite.

This patch does not contain the source files themselves. Based on discussions with @lenary here, it was decided that the actual test files will be committed separately with the proviso that the files will be copied as-is from the gfortran test suite without any modifications. Once this patch is approved, it will be commited first followed by a commit containing the test files themselves.

The test files to be imported are the "main" gfortran tests and the gfortran "torture" tests.

There are two broad classes of tests in the both the main and torture suites - compile tests which test the behavior of the compiler when given incorrect code and execute tests which test the resulting compiled executable. Currently, only the execute tests are enabled.

Of these, a number of tests have been disabled. I have made an attempt to classify them as follows:

  • Unsupported: Tests that use non-standard extensions/intrinsics that are not (currently) supported. Some of these may be supported in the future, but currently, there are no plans to do so.
  • Unimplemented: Tests that fail to build because they trigger a "not yet implemented" assertion in flang.
  • Skipped: Tests that fail to build for some reason other than triggering a "not yet implemented" assertion. These could fail because of some other unimplemented feature, a bug in the compiler, or even a bug in the test itself.
  • Failing: Tests that build but fail at runtime for one reason or another. This may be a bug somewhere in flang that manifests itself at runtime or a bug/unimplemented feature in the runtime.

Eventually, there should be no tests at all in the unimplemented, skipped and failing categories.

Configure-time options have been added to the enable these tests. These are intended to help with manually pruning the list of disabled tests as more language features are implemented and bugs fixed. These options are disabled by default, so all tests should pass by default.

Additionally, the tests are only currently enabled on non-Windows, non-Apple, x86 and AArch64 platforms. This is because:

  1. They haven't been tested on different architectures/OS'es.
  2. There are some tests that are intended to be run only on certain platforms, but the cmake files currently do not correctly deal with those.

Patches for the test files are here, but they have since been abandoned. The links are maintained here for reference:

  1. https://reviews.llvm.org/D146475
  2. https://reviews.llvm.org/D146477
  3. https://reviews.llvm.org/D146478
  4. https://reviews.llvm.org/D146479
  5. https://reviews.llvm.org/D146480
  6. https://reviews.llvm.org/D146481

Diff Detail

Repository
rT test-suite

Event Timeline

tarunprabhu created this revision.Mar 20 2023, 5:43 PM
tarunprabhu created this object with visibility "No One".
Herald added a project: Restricted Project. · View Herald TranscriptMar 20 2023, 5:43 PM
tarunprabhu requested review of this revision.Mar 20 2023, 5:43 PM
tarunprabhu edited the summary of this revision. (Show Details)Mar 20 2023, 6:33 PM
tarunprabhu set the repository for this revision to rT test-suite.
tarunprabhu changed the visibility from "No One" to "Public (No Login Required)".
tarunprabhu added a subscriber: flang-commits.

It looks like I have missed the license files. I will update the patch with those soon.

Added license files (GPL and GPLv3) and cmake file for the "compile" torture tests.

tarunprabhu edited the summary of this revision. (Show Details)Mar 21 2023, 6:23 AM

Fix incorrect include in top-level CMakeLists.txt.

Fix some comments.

lenary added a subscriber: lenary.

I only have one comment, beyond that this looks pretty good to me. Please don't forget the licensing info, it's fairly important to add either with or before the test sources are added, so we are totally clear on what is going on. Note with the C test suite, I landed the CMake first, and then the GPL-licensed test sources, rather than the other way around.

Fortran/gfortran/regression/CMakeLists.txt
54

If you look in SingleSource/Regression/C/gcc-c-torture/execute/CMakeLists.txt, i did try to separate out UnsupportedTests were ones that would never be supported, whereas FailingTests are ones that we should at some point fix. It better documents which of the lists should get smaller over time, and if we end up supporting XFAILs, then we would get even better information from the test suite.

@tarunprabhu, thank you for doing this. But when I try to download any of these patches, I get many error messages and the download fails. Do you need to rebase?

I only have one comment, beyond that this looks pretty good to me. Please don't forget the licensing info, it's fairly important to add either with or before the test sources are added, so we are totally clear on what is going on. Note with the C test suite, I landed the CMake first, and then the GPL-licensed test sources, rather than the other way around.

Yes, I have added the license files to this patch. My apologies for missing them when I first uploaded this. My plan is to commit both the test and the build files together along with the licenses. But failing that, the build and license files will be committed first followed by the tests themselves.

I will try to split the tests into "not yet supported", "unsupported" and "failing" categories.

But when I try to download any of these patches, I get many error messages and the download fails. Do you need to rebase?

Thanks for testing these. Could you tell me what errors you see? Do you get them in this patch, the ones which upload the actual test files, or both?

But when I try to download any of these patches, I get many error messages and the download fails. Do you need to rebase?

Thanks for testing these. Could you tell me what errors you see? Do you get them in this patch, the ones which upload the actual test files, or both?

I tried to download several patches and got similar errors with all of them. My method is to set up a git repository with a clone of the latest LLVM source. I then execute the command arc patch D146485. This is how I download all patches I review. But when I tried it for this patch, I got the following output:

Checking patch /tmp/git-blob-NSUY4b/TestSuite.cmake => cmake/modules/TestSuite.cmake...
error: /tmp/git-blob-NSUY4b/TestSuite.cmake: does not exist in index
Checking patch /tmp/git-blob-bXlOoM/SingleMultiSource.cmake => cmake/modules/SingleMultiSource.cmake...
error: /tmp/git-blob-bXlOoM/SingleMultiSource.cmake: does not exist in index
Checking patch dev/null => Fortran/gfortran/torture/execute/CMakeLists.txt...
error: dev/null: does not exist in index
Checking patch dev/null => Fortran/gfortran/torture/compile/CMakeLists.txt...
error: dev/null: does not exist in index
Checking patch dev/null => Fortran/gfortran/torture/CMakeLists.txt...
error: dev/null: does not exist in index
Checking patch dev/null => Fortran/gfortran/regression/vect/CMakeLists.txt...
error: dev/null: does not exist in index
Checking patch dev/null => Fortran/gfortran/regression/ubsan/CMakeLists.txt...
error: dev/null: does not exist in index
Checking patch dev/null => Fortran/gfortran/regression/prof/CMakeLists.txt...
error: dev/null: does not exist in index
Checking patch dev/null => Fortran/gfortran/regression/lto/CMakeLists.txt...
error: dev/null: does not exist in index
Checking patch dev/null => Fortran/gfortran/regression/ieee/CMakeLists.txt...
error: dev/null: does not exist in index
Checking patch dev/null => Fortran/gfortran/regression/gomp/appendix-a/CMakeLists.txt...
error: dev/null: does not exist in index
Checking patch dev/null => Fortran/gfortran/regression/gomp/CMakeLists.txt...
error: dev/null: does not exist in index
Checking patch dev/null => Fortran/gfortran/regression/goacc/CMakeLists.txt...
error: dev/null: does not exist in index
Checking patch dev/null => Fortran/gfortran/regression/goacc-gomp/CMakeLists.txt...
error: dev/null: does not exist in index
Checking patch dev/null => Fortran/gfortran/regression/g77/CMakeLists.txt...
error: dev/null: does not exist in index
Checking patch dev/null => Fortran/gfortran/regression/debug/CMakeLists.txt...
error: dev/null: does not exist in index
Checking patch dev/null => Fortran/gfortran/regression/coarray/CMakeLists.txt...
error: dev/null: does not exist in index
Checking patch dev/null => Fortran/gfortran/regression/c-interop/CMakeLists.txt...
error: dev/null: does not exist in index
Checking patch dev/null => Fortran/gfortran/regression/asan/CMakeLists.txt...
error: dev/null: does not exist in index
Checking patch dev/null => Fortran/gfortran/regression/analyzer/CMakeLists.txt...
error: dev/null: does not exist in index
Checking patch dev/null => Fortran/gfortran/regression/CMakeLists.txt...
error: dev/null: does not exist in index
Checking patch dev/null => Fortran/gfortran/COPYING3...
error: dev/null: does not exist in index
Checking patch dev/null => Fortran/gfortran/COPYING...
error: dev/null: does not exist in index
Checking patch dev/null => Fortran/gfortran/CMakeLists.txt...
error: dev/null: does not exist in index
Checking patch /tmp/git-blob-AtwfRi/CMakeLists.txt => Fortran/CMakeLists.txt...
error: /tmp/git-blob-AtwfRi/CMakeLists.txt: does not exist in index

 Patch Failed! 
Usage Exception: Unable to apply patch!
tarunprabhu edited the summary of this revision. (Show Details)

First pass at classifying the tests into:

  • Unsupported: Tests that use non-standard extensions/intrinsics that are not (currently) supported. Some of these may be supported in the future, but currently, there are no plans to do so.
  • Unimplemented: Tests that fail to build because they trigger a "not yet implemented" assertion in flang.
  • Skipped: Tests that fail to build for some reason other than triggering a "not yet implemented" assertion. These could fail because of some other unimplemented feature, a bug in the compiler, or even a bug in the test itself.
  • Failing: Tests that build but fail at runtime for one reason or another. This may be a bug somewhere in flang that manifests itself at runtime or a bug/unimplemented feature in the runtime.

Configure-time options have been added to enable the test in some or all of the categories above. These are disabled by default, so all tests should pass by default. The options are intended to be used to periodically prune the list of disabled tests as features are implemented and bugs fixed.

Only enable the gfortran tests for certain platforms and architectures. This is done for two reasons:

  1. They haven't been tested on different platforms.
  2. There are some tests that are intended to be run only on certain platforms, but the cmake files currently do not correctly deal with those.

License headers have been added to the cmake files to explicitly distinguish them from the rest of the files which are licensed under GPL.

lenary accepted this revision.Apr 6 2023, 8:41 AM

This LGTM, from an llvm-test-suite point of view. I would like someone from flang to take a look at it too.

This revision is now accepted and ready to land.Apr 6 2023, 8:41 AM
tarunprabhu added a comment.EditedApr 12 2023, 9:04 AM

High-level instructions to test this patch.

  1. Checkout the llvm-test-suite into a directory, say llvm-test-suite.
  2. Apply this patch to llvm-test-suite.
  3. Checkout/download the GCC source code, for instance from here into a directory, say gcc.
  4. Copy the *contents* of gcc/gcc/testsuite/gfortran.dg into llvm-test-suite/Fortran/gfortran/regression/
  5. Copy the *contents* of gcc/gcc/testsuite/gfortran.fortran-torture into llvm-test-suite/Fortran/gfortran/torture/

I use the following command to build the test-suite. It should only run the Fortran tests.

cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=/path/to/clang -DCMAKE_CXX_COMPILER=/path/to/clang++ -DCMAKE_Fortran_COMPILER=/path/to/flang-new -DCMAKE_Fortran_FLAGS=-flang-experimental-exec 
-DTEST_SUITE_FORTRAN=On -DTEST_SUITE_SUBDIRS=Fortran /path/to/source/of/llvm-test-suite

I use the following command to run the tests from the llvm-test-suite build directory.

NO_STOP_MESSAGE=1 llvm-lit -v  .

All the tests should pass.

EDIT: Fix link to GCC.

High-level instructions to test this patch.

  1. Checkout the llvm-test-suite into a directory, say llvm-test-suite.
  2. Apply this patch to llvm-test-suite.
  3. Checkout/download the GCC source code, for instance from here into a directory, say gcc.
  4. Copy the *contents* of gcc/gcc/testsuite/gfortran.dg into llvm-test-suite/Fortran/gfortran/regression/
  5. Copy the *contents* of gcc/gcc/testsuite/gfortran.fortran-torture into llvm-test-suite/Fortran/gfortran/torture/

I use the following command to build the test-suite. It should only run the Fortran tests.

cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=/path/to/clang -DCMAKE_CXX_COMPILER=/path/to/clang++ -DCMAKE_Fortran_COMPILER=/path/to/flang-new -DCMAKE_Fortran_FLAGS=-flang-experimental-exec 
-DTEST_SUITE_FORTRAN=On -DTEST_SUITE_SUBDIRS=Fortran /path/to/source/of/llvm-test-suite

I use the following command to run the tests from the llvm-test-suite build directory.

NO_STOP_MESSAGE=1 llvm-lit -v  .

All the tests should pass.

EDIT: Fix link to GCC.

Thanks @tarunprabhu for providing the instructions.

I am currently hitting the following error. Is the lit.local.cfg a generated file? Or is it missed in this patch?

CMake Error at Fortran/gfortran/regression/CMakeLists.txt:1839 (file):
  file COPY cannot find
  "/home/kircha02/llvm-test-suite/Fortran/gfortran/regression/lit.local.cfg":
  No such file or directory.

I am trying to run this on AArch64 by commenting out the check for X86.

tarunprabhu edited the summary of this revision. (Show Details)

@kiranchandramohan, Thanks for testing this.

Apologies, the lit.local.cfg files where missing from the patch. They have been added now.

If you believe that the AArch64 machine on which you are testing is representative, I can add it to the list of platforms on which these tests are enabled.

@kiranchandramohan, Thanks for testing this.

Apologies, the lit.local.cfg files where missing from the patch. They have been added now.

If you believe that the AArch64 machine on which you are testing is representative, I can add it to the list of platforms on which these tests are enabled.

I had to disable the following tests to get an all-pass. I think three of them are runtime failures, the rest seem to be compile-time failures. I don't know whether these are new tests or platform-specific but being mostly compile-time failures I doubt they are platform specific.

+  interface_49.f90
+  associate_60.f90
+  finalize_38.f90
+  finalize_38a.f90
+  finalize_39.f90
+  finalize_40.f90
+  finalize_41.f90
+  finalize_42.f90
+  finalize_43.f90
+  finalize_44.f90
+  finalize_45.f90
+  finalize_46.f90
+  finalize_47.f90
+  finalize_48.f90
+  finalize_50.f90
+  finalize_51.f90
+  pr105205.f90
+  pr106918.f90
+  internal_dummy_2.f08
+  leadz_trailz_1.f90
+  leadz_trailz_2.f90
+  leadz_trailz_3.f90
+  assign_2.f90
+  entry_23.f
+  findloc_8.f90

It seems that most of the failing tests were added since I last pulled from gcc's repo. I was unable to reproduce the failures of

  • entry_23.f
  • findloc_8.f90

There were a number of other failing tests which may be regressions, but I didn't look too closely.

I intend to use the test files from this commit when I merge this patch into the test suite.

It seems that most of the failing tests were added since I last pulled from gcc's repo. I was unable to reproduce the failures of

  • entry_23.f
  • findloc_8.f90

I think these were runtime failures. I will try again and see.

There were a number of other failing tests which may be regressions, but I didn't look too closely.

OK. It is probably because I did not run the tests from the subdirectories. I think initially you were not planning to add them. I will run the subdirectories and get back to you with an update.

I intend to use the test files from this commit when I merge this patch into the test suite.

Good to have a reference commit.

In the lit report I find the following:

Failed Tests (2):
  test-suite :: Fortran/gfortran/regression/gfortran-regression-execute-regression__entry_23_f.test
  test-suite :: Fortran/gfortran/regression/gfortran-regression-execute-regression__findloc_8_f90.test


Testing Time: 5.54s
  Passed: 1717
  Failed:    2

If I manually run these tests I get the following. The second one is probably a platform-specific error. If you can mark these as runtime failures then we can enable on AArch64 as well.

$ /home/kircha02/llvm-project/build_release/bin/flang-new ../Fortran/gfortran/regression/entry_23.f -flang-experimental-exec
$ ./a.out 
 CHAR_SUB
 len(carray(1)) 20
Segmentation fault (core dumped)
$ /home/kircha02/llvm-project/build_release/bin/flang-new ../Fortran/gfortran/regression/findloc_8.f90 -flang-experimental-exec
$ ./a.out 

fatal Fortran runtime error(/home/kircha02/llvm-test-suite/build/../Fortran/gfortran/regression/findloc_8.f90:9): not yet implemented: REAL/COMPLEX(KIND=10)
Aborted (core dumped)
Fortran/CMakeLists.txt
15

Would it work on Mac?

16

Please enable on AArch64 as well.

Fortran/gfortran/CMakeLists.txt
2

Ninja or cmake version?

We should have a cmake version check (CMake 3.24.0) for the version that enabled it. Or is that what the following check does?

19
63
187

My check-flang build did not have this and I had to copy it over. Is this copied over in the normal build or an install?
ninja check-flang vs ninja vs ninja install.

218
Fortran/gfortran/COPYING
59

Is this a real character?

Addressed several comments by @kiranchandramohan

tarunprabhu marked 8 inline comments as done.Apr 14 2023, 3:15 PM
tarunprabhu added inline comments.
Fortran/CMakeLists.txt
15

I haven't checked on a Mac. Added Apple to the blacklist.

Fortran/gfortran/CMakeLists.txt
2

That comment is completely wrong. I confused it with something else. Removed.

187

It is definitely available with ninja install. I am not sure with just Ninja, I will check.

Do you have any suggestions on how this can be dealt with? I think many of the flang developers only run check-flang builds, right?

Fortran/gfortran/COPYING
59

There were several of these in the file that I copied over from the SingleSource/Regression/C/gcc-c-torture. I have removed all instances.

tarunprabhu marked 3 inline comments as done.

Disable entry_23.f and findloc_8.f90 since they fail at runtime on AArch64. This should allow the test-suite to pass by default on AArch64 and x86.

LGTM. Thanks for your effort in setting this up.

Will the changes to https://github.com/llvm/llvm-test-suite/blob/main/LICENSE.TXT come in a separate patch. i.e listing gfortran tests as test having a different license?

The summary will need minor updates (like enabled on AArch64 as well).

Otherwise, the split into Unsupported, Unimplemented, Skipped, Failing looks OK. Not sure whether we need to rename Skipped/Failing to make it clear that Skipped is build failures and Failing is runtime failures.

BTW, the detailed characterisation (even though it is probably at a coarse-level) looks very good and possibly a good source for folks to look at and start fixing issues.

Fortran/gfortran/CMakeLists.txt
2

Can we add a minimum CMake version (3.24.0 I think) check? I had to download the latest cmake version to get this to work.
https://cmake.org/cmake/help/latest/command/cmake_minimum_required.html

187

Yes, I generally run check-flang. This is fine for now, we can check with others whether this needs to be moved. Possible option is to add this in the build step. I think similar copying of files to build directory is in flang/tools/f18/CMakeLists.txt.

tarunprabhu edited the summary of this revision. (Show Details)

Added a minimum cmake version for the gfortran tests.

Added entries for the gfortran tests into the top-level LICENSE.txt file indicating that the files in Fortran/gfortran are governed by a different license.

tarunprabhu marked 2 inline comments as done.Apr 17 2023, 7:44 AM

Otherwise, the split into Unsupported, Unimplemented, Skipped, Failing looks OK. Not sure whether we need to rename Skipped/Failing to make it clear that Skipped is build failures and Failing is runtime failures.

I am open to suggestions for renaming them. Eventually, the "Skipped" and "Failing" lists should be empty and removed. Everything should either pass or be moved to the "Unsupported" list. There are inline comments explaining the purpose of the various lists in Fortran/gfortran/regressions/CMakeLists.txt. but, I agree, it's not ideal.

Add an option for the user to provide a path to the directory containing ISO_Fortran_bindings.h that is needed to run the tests.

Added a README summarizing the current state of the gfortran tests.

@kiranchandramohan, could you take a look at the README please. Thanks for all the reviews so far :-)

I will be leaving this up for a few more days to give others a chance to test/review it.

tarunprabhu marked an inline comment as done.Apr 18 2023, 8:51 AM

Move the add_tests function in Fortran/gfortran/torture' to 'Fortran/gfortran/torture/exceute. That function is only relevant for the "execute" tests. The "compile" tests will probably need to be handled differently.

Add an option for the user to provide a path to the directory containing ISO_Fortran_bindings.h that is needed to run the tests.

Added a README summarizing the current state of the gfortran tests.

@kiranchandramohan, could you take a look at the README please. Thanks for all the reviews so far :-)

I will be leaving this up for a few more days to give others a chance to test/review it.

Readme looks good. Patch still LGTM.

Fortran/gfortran/regression/gomp/CMakeLists.txt
10

Is this because all the OpenMP tests are a compilation tests.

I guess we can provide an option (in later patches) to build all non-execute tests as well. Currently, no OpenMP tests are run.

tarunprabhu marked an inline comment as done.Apr 20 2023, 10:19 AM

I guess we can provide an option (in later patches) to build all non-execute tests as well. Currently, no OpenMP tests are run.

I have been working to enable the "compile" tests. I have a way to run the compile tests from the gfortran torture suite. I will put up a patch to do that once this goes in. If that approach is deemed acceptable, it should translate to the tests in the main gfortran test suite as well - including the OpenMP tests.

If there are no objections, I will merge this first thing next week.

Fortran/gfortran/regression/gomp/CMakeLists.txt
10

Yes, all OpenMP tests are compilation tests.

This revision was automatically updated to reflect the committed changes.
tarunprabhu marked an inline comment as done.

The gfortran tests don't build for me when I use the Ninja generator from CMake. It appears that multiple files in the regression directory have modules called m and so multiple m.mod files are generated by the compiler representing these modules, which causes ninja to error out.

I think this is actually a legitimate race condition that doesn't get caught by Make as it doesn't check the outputs in the same way, but this still may cause issues if one of the compiles generates an m.mod file and then tries to use it but it has been replaced by the m.mod from another compile.

The gfortran tests don't build for me when I use the Ninja generator from CMake. It appears that multiple files in the regression directory have modules called m and so multiple m.mod files are generated by the compiler representing these modules, which causes ninja to error out.

I think this is actually a legitimate race condition that doesn't get caught by Make as it doesn't check the outputs in the same way, but this still may cause issues if one of the compiles generates an m.mod file and then tries to use it but it has been replaced by the m.mod from another compile.

I see two modules M1 and TEST that are repeated in torture/execute. One of the tests defining TEST is disabled so we might not be hitting the issue for this one.

$ grep -i 'module m1' llvm-test-suite/Fortran/gfortran/torture/execute/*
llvm-test-suite/Fortran/gfortran/torture/execute/function_module_1.f90:module M1
llvm-test-suite/Fortran/gfortran/torture/execute/module_init_1.f90:module m1

$ grep -i 'module test' llvm-test-suite/Fortran/gfortran/torture/execute/*
llvm-test-suite/Fortran/gfortran/torture/execute/pr32140.f90:MODULE TEST
llvm-test-suite/Fortran/gfortran/torture/execute/pr32604.f90:MODULE TEST

But there are several tests in llvm-test-suite/Fortran/gfortran/regression that define modules with the same name.

 55 MODULE TEST
 80 MODULE M2
 83 MODULE FOO
 95 MODULE M1
384 MODULE M

The gfortran tests don't build for me when I use the Ninja generator from CMake. It appears that multiple files in the regression directory have modules called m and so multiple m.mod files are generated by the compiler representing these modules, which causes ninja to error out.

I think this is actually a legitimate race condition that doesn't get caught by Make as it doesn't check the outputs in the same way, but this still may cause issues if one of the compiles generates an m.mod file and then tries to use it but it has been replaced by the m.mod from another compile.

Do you get a failure at configure time or build time?

I get an error at build time when running ninja

I get an error at build time when running ninja

Ok. I'll look into this. Seems like there are several different issues all related to race conditions. I'll try to find a more general approach that will fix all of them.