This is an archive of the discontinued LLVM Phabricator instance.

[Fortran] adding a new test now that CMake supports LLVMFlang
ClosedPublic

Authored by cabreraam on Jul 28 2022, 3:42 PM.

Details

Summary

Adding a Non-trivial Fortran Test to llvm-test-suite

As of this CMake merge request, there is now CMake support for LLVMFlang. This allows for the integration of LLVMFlang into the llvm-test-suite. To this end, this request for review takes advantage of this new CMake functionality by adding a proxy application called SNAP, which aims "to model the performance of a modern discrete ordinates neutral particle transport application". This application was picked because of its importance to Flang developers, and because there were folks who were already using it to test Flang's correctness in producing end-to-end binaries.

A recipe for building the llvm-test-suite is listed below:

#!/usr/bin/env bash

LLVM_TS_BUILD_DIR="/path/to/where/you/want/llvm-test-suite/to/be/built"
FLANG_PATH="/path/to/your/build/of/flang"
FC="${FLANG_PATH}/bin/flang-new"

PATH_TO_LIBPGMATH="/path/to/libpgmath"
CMAKE_FORTRAN_FLAGS="{PATH_TO_LIBPGMATH}/libpgmath.so -flang-experimental-exec"

CMAKE_OPTIONS="-GNinja \
-DCMAKE_C_COMPILER=clang \
-DCMAKE_CXX_COMPILER=clang++ \
-DCMAKE_Fortran_COMPILER=${FC} \
-DTEST_SUITE_FORTRAN=on \
-DTEST_SUITE_SUBDIRS=Fortran \
-DCMAKE_BUILD_RPATH=${PATH_TO_LIBPGMATH}"


mkdir -p ${LLVM_TS_BUILD_DIR}
cd ${LLVM_TS_BUILD_DIR}
cmake ${CMAKE_OPTIONS} -DCMAKE_Fortran_FLAGS="${CMAKE_FORTRAN_FLAGS}" ..
cmake --build . -j64

NOTE: You can save yourself some build time by just specifying -DTEST_SUITE_SUBDIRS=Fortran so you don't have to build the entire llvm-test-suite, but this option can be omitted if you want to build the whole thing.

NOTE: An option called PGMATH is specified. For now, this is required, but may not be if we decide to remove libpgmath being built _as part_ of the llvm-test-suite.

The major components of this review request are:
+ adding the necessary source files from the SNAP project repository
+ CMakeLists.txt for the SNAP proxy application

  1. Adding SNAP Source Files
    1. Location

I have taken the SNAP project repo and placed it in llvm-test-suite/Fortran.

Files

From the SNAP project repo, I have taken the files in src and added them to the llvm-test-suite. I have also explicitly added a LICENSE.md file. Based on the license, we should be able to include the source in llvm-test-suite. I have removed some stuff files from the original SNAP repo that really shouldn't have been added -- thank you @rovka for the feedback! -- but there are some other files that I have included from the SNAP repo that I have kept in the revision (but ultimately probably shouldn't be there):

  • docs
  • qasnap (the input test files that aren't qasnap/mms_src/2d_mms_st.inp)
    • really, there's only one input that we test, which is listed above. It would be possible to test other inputs to act as additional regression tests, but I would agree with the sentiment that the other inputs can be removed and then subsequently added if folks wanted to try more inputs.

CMake

I've removed already modified some of the CMake stuff that @rovka pointed out wasn't really necessary, but here are the salient bits of the CMake build infrastructure that I've added.

There are two files: Fortran/SNAP/CMakeLists.txt and Fortran/SNAP/src/CMakeLists.txt.

  1. src/CMakeLists.txt
    1. Why not llvm_test_executable

There are methods that allow you to build targets that consist of multiple source files. In general, though, using this call does not give enough fine-grained control over the target and how it is compiled/linked.

Additionally, this question breaks down into:

  • Why doesn't llvm_test_executable_no_test work?
    • this CMake function only mutates the global CFLAGS, CPPFLAGS, and CXXFLAGS variables, which doesn't apply to Fortran. Not using llvm_test_executable_no_test frees me up to use functions like target_compile_options, target_link_libraries`, and the like.
  • Why doesn't llvm_add_tests_for_target work?
    • this function assumes that the name of the target is also the base name for the input file, which is not the case for SNAP. Since this assumption doesn't hold, I have to roll my own CMake for this bit, though admittedly there is a lot of overlap since the name is the only issue.

For future applications that require more fine-grained control of parameterizing the target, it might make sense to create a .cmake file geared for Fortran based on the work I've done here. But I'll call that future work and outside the scope of this PR.

TESTSCRIPT

The RUN: and VERIFY: commands that eventually get run are, in part, based on work done by @awarzynski and others. Specifically, I am running the application with the input that they have verified and validated to be correct. As noted above, I am using the SNAP/qasnap/mms_src/2d_mms_st.inp file.

libpgmath with ExternalProject_Add

Currently, using LLVMFlang means that you need to externally link libpgmath so that those symbols are defined. Since this is the current state of Flang, I have added the functionality to do this without the programmer/developer having to supply the locaiton of a libpgmath build themselves. As @rovka points out, I hope that this isn't always the case. But since it is, that's why I've made the design choice. I am happy to remove it in this PR, or to leave it and then remove it once the math intrinsics are properly handled within LLVMFlang. Let's keep it that way.

Diff Detail

Repository
rT test-suite

Event Timeline

cabreraam created this revision.Jul 28 2022, 3:42 PM
Herald added a project: Restricted Project. · View Herald Transcript
Herald added subscribers: wenlei, mgorny. · View Herald Transcript
cabreraam updated this revision to Diff 448506.Jul 28 2022, 8:58 PM

Updating D130734: [Fortran] adding a new test now that CMake supports LLVMFlang

rovka added a subscriber: rovka.Jul 29 2022, 1:50 AM

Hi, thanks for working on this! I only had a very cursory first glance and I have a few questions (sorry if some of them are silly, some of this was a bit TL;DR).

Fortran/CMakeLists.txt
11

This is interesting, but I wouldn't say it's the test-suite's job to install libpgmath. It's not needed by the tests themselves, but by the compiler (and hopefully not for long). I think it's reasonable to assume that whoever wants to run the test-suite with flang already has libpgmath installed and they can provide whatever CMAKE_Fortran_FLAGS are needed to make things work (see e.g. my experimental buildbot running the test-suite)

Fortran/SNAP/.travis.yml
1 ↗(On Diff #448506)

What's this file used for? Is it for Travis CI?

Fortran/SNAP/CMakeLists.txt
19

What about the other directories? Do we really need to add all those to the test-suite, or could we get away with adding just src?

Fortran/SNAP/src/CMakeLists.txt
19

These need more specific names, e.g. SNAP_USE_MPI etc. Otherwise people might misinterpret them as applying to the whole test-suite.

231

I'm a bit unfamiliar with this topic, so please bear with me: isn't this a common Fortran convention, i.e. something the compiler's driver should know how to handle? I would have expected the compiler itself to look at whether the extension is uppercase or lowercase and decide to preprocess or not accordingly. Alternatively, if the compiler can't handle this, I would expect maybe CMake itself to be able to handle it out of the box. It seems suspicious for every Fortran application to have to explicitly handle this in its build system...

236

Do users really need to be able to modify this? Or is there some other reason why it needs to be in the cache?

254

Any particular reason why these can't be passed directly to add_executable?

302

Could you be more specific about this? E.g. why can't you use llvm_test_executable?

390

What for?

cabreraam added inline comments.Aug 1 2022, 10:23 AM
Fortran/CMakeLists.txt
11

I agonized over this. My thought was that, if you're using the LLVM test suite, you're (probably?) trying to test Flang. And the current state of the practice is that you need to link in libpgmath. But to your point, I also hope that the compiler won't need to require external linking of libpgmath. I just wasn't sure how long it would need to be required. I could go either way -- removing it or leaving it -- so I'm happy to do whatever the majority of folks thinks is best!

Fortran/SNAP/.travis.yml
1 ↗(On Diff #448506)

Good question! This is left over from when I forked the SNAP repo initially. It is not actually required for llvm-test-suite. I will remove it!

Fortran/SNAP/CMakeLists.txt
19

Very good question. I'll have to think on this. ports can go. I like docs staying in so folks have an idea of what the code actually does. qasnap contains the directory for different inputs. Right now, only one file is used for input -- /qasnap/mms_src/2d_mms_st.inp -- but different files might be able to be used to run additional tests. However, I could envision just adding the input file that we do use, and then updating the repo if we decide to use different input files to run additional tests.

For now, I will go ahead and remove ports.

Fortran/SNAP/src/CMakeLists.txt
19

Yes, good point! I meant to do this but it slipped through the cracks. I'll work on it.

231

I will look into this. This is an artifact of the Makefile that I based the CMakeLists.txt on. I'm not sure if it's actually necessary.

236

Nope, no reason to cache this! Good catch.

254

Nope! I think I had a reason when I first did this, but I cannot remember it. Extra verbosity for no reason, it seems!

302

I will write this up in my pull request description!

390

This was part of the original Makefile. Part of the exercise of creating the CMake infra for SNAP was to preserve original functionality of the Makefile. There was a rule to create this file called Lines that counted lines of code. This is not actually necessary for the llvm-test-suite. I should probably remove this.

cabreraam added inline comments.Aug 3 2022, 11:48 AM
Fortran/SNAP/src/CMakeLists.txt
231

Yep, upon further inspection, LLVMFlang takes care of this. Here is a sketch of the command for compiling a file with the .F90 extension:

flang-new -cpp  -module-dirsrc -E src/plib.F90 > src/CMakeFiles/snap.dir/plib.F90-pp.f90 && cmake -E cmake_ninja_depends --tdi=src/CMakeFiles/snap.dir/FortranDependInfo.json --lang=Fortran --pp=src/CMakeFiles/snap.dir/plib.F90-pp.f90 --dep=src/CMakeFiles/snap.dir/plib.F90-pp.f90.d --obj=src/CMakeFiles/snap.dir/plib.F90.o --ddi=src/CMakeFiles/snap.dir/plib.F90.o.ddi

an intermediate file with the .f90 extension gets created! So all of the stuff that deals with the .F90 extension can be removed.

cabreraam updated this revision to Diff 449776.Aug 3 2022, 1:39 PM

Addressing Diana's comments

cabreraam updated this revision to Diff 449779.Aug 3 2022, 1:53 PM

Addressing Diana's comments

cabreraam added inline comments.Aug 3 2022, 2:01 PM
Fortran/SNAP/src/CMakeLists.txt
236

This will be getting removed because we don't need the preprocessing stuff.

cabreraam edited the summary of this revision. (Show Details)Aug 4 2022, 6:25 PM
cabreraam updated this revision to Diff 450372.Aug 5 2022, 1:15 PM

Removed more unnecessary files; added LICENSE.md; revised the README.md to show
that the original SNAP distribution had been modified for inclusion into
llvm-test-suite.

cabreraam edited the summary of this revision. (Show Details)Aug 5 2022, 1:17 PM
cabreraam published this revision for review.Aug 5 2022, 2:07 PM
cabreraam edited the summary of this revision. (Show Details)
cabreraam edited the summary of this revision. (Show Details)
cabreraam edited the summary of this revision. (Show Details)Aug 5 2022, 2:10 PM
rovka added a comment.Aug 11 2022, 1:47 AM

I'm only seeing the README, is this intentional?

I'm only seeing the README, is this intentional?

I think the changes are visible in the revision history, but I'm still new to phabricator, so any help would be appreciated @rovka!

Each time I wanted to update this diff, I'd issue the command

arc diff --update D130734

Did I clobber over my previous commits when I issued this command?

I'm only seeing the README, is this intentional?

I think the changes are visible in the revision history, but I'm still new to phabricator, so any help would be appreciated @rovka!

Each time I wanted to update this diff, I'd issue the command

arc diff --update D130734

Did I clobber over my previous commits when I issued this command?

I suspect that locally you are working on a branch and every change in that branch is a separate commit. And what you are uploading here is one commit from that branch? That's basically the GitHub workflow :) In Phabricator, you always upload the whole thing. In Git parlance, you'd squash your branch before updating the revision here. Put differently, every Diff in Phabricator should contain all your changes at the given time (rather than the delta between the "current" state and the "previous" state).

Hope this helps, but I might be totally wrong as well :)

I'm only seeing the README, is this intentional?

I think the changes are visible in the revision history, but I'm still new to phabricator, so any help would be appreciated @rovka!

Each time I wanted to update this diff, I'd issue the command

arc diff --update D130734

Did I clobber over my previous commits when I issued this command?

I suspect that locally you are working on a branch and every change in that branch is a separate commit. And what you are uploading here is one commit from that branch? That's basically the GitHub workflow :) In Phabricator, you always upload the whole thing. In Git parlance, you'd squash your branch before updating the revision here. Put differently, every Diff in Phabricator should contain all your changes at the given time (rather than the delta between the "current" state and the "previous" state).

Hope this helps, but I might be totally wrong as well :)

Okay, this is helpful @awarzynski! In this particular instance, I used arc diff --draft. Then I got some helpful feedback, and wanted to update this diff according to the feedback I got. In that case, how would I proceed in the future? I used arc diff --update D130734, but should I have done something else? Additionally, how would I fix this particular revision to show ALL of my changes instead of individual commits? Do I just have to close this one and create a new diff?

cabreraam edited the summary of this revision. (Show Details)

Reworked the commit history with some rebasing and some squashing to get the
correct changes to show up

Reworked the commit history with some rebasing and some squashing to get the correct changes to show up

Reworked the commit history with some rebasing and some squashing to get the correct changes to show up part 3

Apologies for the spam. I think I've finally gotten it in a state that shows the changes and reflects all of the comments that I'd gotten from @rovka. Thanks to @awarzynski for the phabricator advice!

Looking at the qasnap directory, I see that there are a ton of unnecessary files like I mentioned in the write-up, so I'm assuming I'll need to remove those!

cabreraam edited the summary of this revision. (Show Details)Aug 17 2022, 9:03 AM
cabreraam edited the summary of this revision. (Show Details)
cabreraam edited the summary of this revision. (Show Details)Aug 17 2022, 9:05 AM
rovka added a comment.Aug 18 2022, 3:56 AM

Hi, thanks for updating the patch! I have a few more comments / questions.

  • We really need to extract some helpers for adding fortran tests (or extend the existing ones to handle fortran as well, I haven't really thought through the whole thing). I can probably convince myself to leave that for the next fortran application that we add, but I'd like to hear other people's opinions about this.
  • How did you decide which input to use? Also, do you know if we have any applications already in the test-suite with more than one input available?
Fortran/CMakeLists.txt
11

I'd be more inclined to remove it. In the medium to long term, flang won't depend on libpgmath anymore (surely not on *that* libpgmath, we'll either stop needing libpgmath altogether or integrate it into the llvm-project repo and teach the driver about it). While it's possible that when that happens someone will come and clean these cmake files up, it's IMO more likely that people will forget and then it will just be (potentially confusing) clutter.

Fortran/SNAP/src/CMakeLists.txt
19

I've been thinking some more about this and these options probably shouldn't be set here like this. Usually people will want to run the whole test-suite with the same compilation options. There's a directory cmake/caches with various configuration files (such as O3.cmake etc) which can be passed to cmake -C. Obviously, those only set flags for C/C++ compilation at the moment, but I see no reason why you can't add Fortran options there as well. You can also add new cache files for Cray, Haswell, ifort etc if you feel so inclined. Could you give that a try? Those should probably be separate patches, this patch should only contain the bare minimum needed to run snap.

45
46
Fortran/SNAP/src/LineCount
1 ↗(On Diff #452231)

I thought we didn't need this anymore :)

Fortran/SNAP/src/LineReport
1 ↗(On Diff #452231)

Ditto.

Fortran/SNAP/src/Makefile
1 ↗(On Diff #452231)

Ditto.

cabreraam updated this revision to Diff 457107.Aug 31 2022, 3:17 PM

Addressing Diana's comments

cabreraam updated this revision to Diff 457110.Aug 31 2022, 3:20 PM

Forgot to remove ExternalProject_Add stuff

cabreraam updated this revision to Diff 457111.Aug 31 2022, 3:25 PM

a couple of other mistakes removed

cabreraam added inline comments.Aug 31 2022, 3:29 PM
Fortran/SNAP/src/CMakeLists.txt
19

Okay, this is helpful @rovka ! I understand the idea of cmake -C and cache scripts, but I'm not too familiar with how I would go about using that idea here. As you can see now, I've just commented out all of the stuff that had to do with OpenMP/MPI/architecture/u-architecture dependent options. My thought is I would just remove all of them? To your point, they aren't necessary. I'm trying to aim for the bare minimum that's needed for LLVMFlang and removing all of these would be okay. I've already tested and verified that these aren't needed anymore. What do you think?

cabreraam edited the summary of this revision. (Show Details)Aug 31 2022, 3:30 PM
cabreraam edited the summary of this revision. (Show Details)
cabreraam edited the summary of this revision. (Show Details)Aug 31 2022, 3:36 PM
rovka added a comment.Sep 1 2022, 3:02 AM

Thanks for getting back to this, it's starting to look very nice and slender! :) I have a few more comments, all pretty minor.

Fortran/SNAP/src/CMakeLists.txt
19

Removing all of it sounds good to me :)

59
61

This doesn't seem to be used.

324

I think you can remove this whole comment, most people won't know what the original Makefile looked like (not to mention that the Makefile itself might diverge in time, or SNAP itself might move to cmake, or whatever).

330
Fortran/SNAP/src/mainpage.dox
4 ↗(On Diff #457111)

Is this just duplicating the text from the README? I don't think we have other .dox files in the test-suite, so maybe we don't need it?

cabreraam updated this revision to Diff 457345.Sep 1 2022, 12:01 PM

[test-suite][Fortran] addressing Diana's changes RE: Fortran/src/CMakeLists.txt and removed mainpage.dox

cabreraam added a comment.EditedSep 1 2022, 12:06 PM

Okay! I've

  • Removed that giant block where I commented out all of the different options for various (unnecessary) architecture, microarchitectures, and vendors
  • Removed the setting of the unnecessary PGM_DIR variable
  • Changed fxn --> function
  • and removed the Fortran/SNAP/src/mainpage.dox

What do you think @rovka? Ready for upstreaming?

rovka accepted this revision.Sep 5 2022, 12:31 AM

LGTM! Thanks for putting up with so many rounds of review. I have 2 more nits, but you don't have to submit another diff. Do you have commit access?

Fortran/SNAP/CMakeLists.txt
17

You can remove these lines

Fortran/SNAP/src/CMakeLists.txt
18

I think this should be SNAP_USER_DEFS or something of the sort, since it only applies to SNAP and not the rest of the test-suite.

This revision is now accepted and ready to land.Sep 5 2022, 12:31 AM

I do not yet have commit access! How do I go about requesting that?

I will make sure to address your last couple of comments.

rovka added a comment.Sep 8 2022, 11:27 PM

I do not yet have commit access! How do I go about requesting that?

https://llvm.org/docs/DeveloperPolicy.html#obtaining-commit-access

If it's your first patch, I can commit on your behalf and then you can request access.

cabreraam updated this revision to Diff 459254.Sep 9 2022, 10:24 PM

Updating D130734: [Fortran] adding a new test now that CMake supports LLVMFlang

I've fixed the nits! Please commit away!

I'm sending an email now to request commit access for the future.

This revision was automatically updated to reflect the committed changes.