This is an archive of the discontinued LLVM Phabricator instance.

[flang][driver] Add support for `-D`, `-U`
ClosedPublic

Authored by FarisRehman on Dec 16 2020, 8:31 AM.

Details

Summary

Add support for options -D and -U in the Flang driver.

Summary of changes:

  • Create PreprocessorOptions, to be used by the driver then translated into Fortran::parser::Options
  • Create CompilerInvocation::setFortranOpts to pass preprocessor options into the parser options
  • Add a dedicated method, Flang::AddPreprocessingOptions, to extract preprocessing options from the driver arguments into the preprocessor command arguments

Macros specified like -DName will default to definition 1.

When defining macros, the new driver will drop anything after an end-of-line character. This is consistent with gfortran and clang, but different to what currently f18 does. However, flang (which is a bash wrapper for f18), also drops everything after an end-of-line character (at least in the bash on my Linux machine). So gfortran-like behaviour felt like the natural choice. Add a test to demonstrate this behaviour.

Diff Detail

Event Timeline

FarisRehman created this revision.Dec 16 2020, 8:31 AM
FarisRehman requested review of this revision.Dec 16 2020, 8:31 AM
Herald added a project: Restricted Project. · View Herald TranscriptDec 16 2020, 8:31 AM
Herald added a subscriber: cfe-commits. · View Herald Transcript
FarisRehman edited projects, added Restricted Project; removed Restricted Project.
Herald added a project: Restricted Project. · View Herald TranscriptDec 16 2020, 8:34 AM
Herald added a subscriber: jdoerfert. · View Herald Transcript
FarisRehman edited the summary of this revision. (Show Details)Dec 17 2020, 3:10 AM

@FarisRehman, thank you for working on this! This looks really good and I think that it's almost ready. I did leave a few comments, but most are a matter of style and should be easy to address.

clang/lib/Driver/ToolChains/Flang.cpp
74–75

Clang deals with the preprocessor options in a dedicated method, AddPreprocessingOptions. See here:
https://github.com/llvm/llvm-project/blob/eba09a2db9eab46832cb7ec7ef0d2c227747772a/clang/lib/Driver/ToolChains/Clang.cpp#L1077

I feel that it might be a good idea to follow that. The preprocessor options (and the corresponding logic) are bound to grow, so we may as well add a dedicated method sooner rather than later.

77

This change is not needed, is it?

77

AFAIK, when you call AddAllArgs the corresponding options are eventually _claimed_:
https://github.com/llvm/llvm-project/blob/eba09a2db9eab46832cb7ec7ef0d2c227747772a/llvm/lib/Option/ArgList.cpp#L112
So, IIUC, this line is not needed.

flang/include/flang/Frontend/CompilerInvocation.h
35
  1. Could you add a comment explaining what this is for?
  1. Could you follow the current semantics used in the file and make it a std::shared_ptr? Long term we can expect PreprocessorOptions to grow big and at that point we wouldn't want to copy it too often (especially when compiling a lot of files).
85–89

IIUC, you are renaming this method because it is finally translating the user provided options (i.e. preprocessorOpts()) into Frontend options (i.e. fortranOptions). However, with your changes this method:

  • sets some _sane_ predefined defaults
  • adds some stuff from the user (the preprocessor defines)

So it's somewhere in between SetDefaultFortranOpts and SetFortranOpts (i.e. both names are _almost_ accurate). Perhaps splitting this into two methods would be better? E.g.:

  • SetDefaultFortranOpts (leave as is)
  • SetFortranOpts (implements new functionality, to replace SetDefaultFortranOpts eventually)
flang/include/flang/Frontend/PreprocessorOptions.h
2
9

[nit] No need to repeat the name of the class.

13

[nit] IMO this would be preferred : /*isUndef=*/bool (https://llvm.org/docs/CodingStandards.html#comment-formatting) (currently you have bool /*isUndef*/)

flang/lib/Frontend/CompilerInvocation.cpp
158

I couldn't find any notes on naming static functions here: https://github.com/llvm/llvm-project/blob/main/flang/docs/C%2B%2Bstyle.md. This means that we fallow this: https://llvm.org/docs/CodingStandards.html#name-types-functions-variables-and-enumerators-properly. So, parsePreprocessorArgs instead of ParsePreprocessorArgs.

Also, could you add a comment briefly explaining what the method is for? E.g. Parses all preprocessor input arguments and populates the preprocessor options accordingly? Ideally doxygen :) See example here: https://github.com/llvm/llvm-project/blob/eba09a2db9eab46832cb7ec7ef0d2c227747772a/flang/include/flang/Frontend/CompilerInstance.h#L158-L163

161

Variable names should start with lower case.

[nit] Perhaps currentArg instead of A for the sake of being explicit? Or curArg?

246

This is not quite true, because you're collecting them into preprocessor options :)

247

static void instead (we don't need this method to be visible outside this translation unit).

249

All variables here have to be lower case:.

flang/test/Flang-Driver/macro.f90
1 ↗(On Diff #312220)

There are no tests for e.g. -DSOME_VAR=SOME_VAL. It would be good to test that SOME_VAL was parsed correctly.

[nit[ Perhaps macro_def_undef.f90 would be more descriptive? (re filename).

tskeith added inline comments.Dec 17 2020, 8:25 AM
flang/lib/Frontend/CompilerInvocation.cpp
269

This is a change in behavior from f18, right? If so, the commit message should mention it and why it's changing.

FarisRehman edited the summary of this revision. (Show Details)

Address review comments

Address comments with the following changes:

  • Add a dedicated method to adding preprocessing options in Flang.cpp
  • Change preprocessorOpts to use std::shared_ptr
  • Separate SetDefaultFortranOpts from SetFortranOpts in CompilerInvocation
  • Fix method and variable naming in CompilerInvocation.cpp
  • Add comments to new methods and fields
  • Add missing file header to PreprocessorOptions.h
  • Remove unnecessary changes and method calls
  • Add a regression test to check end-of-line character behaviour in a macro definition
FarisRehman marked 11 inline comments as done.Dec 21 2020, 7:40 AM
FarisRehman edited the summary of this revision. (Show Details)Dec 21 2020, 7:53 AM
FarisRehman marked an inline comment as done.Dec 21 2020, 7:57 AM
FarisRehman added inline comments.
flang/lib/Frontend/CompilerInvocation.cpp
269

Updated the summary to address this, thanks

FarisRehman marked 2 inline comments as done.Dec 21 2020, 8:02 AM
FarisRehman added inline comments.
flang/lib/Frontend/CompilerInvocation.cpp
246

Added a comment to clarify the input and output arguments in the latest revision 👍

@FarisRehman , thank you for updating this! I have two high-level comments:

  1. In your summary:

Change the way the driver handles end-of-line characters in macro definitions.

That's a a bit misleading - which driver do you mean? You're not changing the behavior of the current driver, flang. Also, this functionality doesn't yet exist in the new driver, flang-new, so you're not changing it there either. You can probably remove this sentence and just make it clear that you meant the new driver.

  1. Could you doxygen style comments for input and output parameters in the new methods that you introduce? Ta! https://www.doxygen.nl/manual/commands.html#cmdparam

More comments inline.

clang/lib/Driver/ToolChains/Flang.cpp
36

This is not used until line 73. Perhaps better to define it near to where it's used?

https://isocpp.org/wiki/faq/coding-standards#declare-near-first-use

clang/lib/Driver/ToolChains/Flang.h
27–31
  1. Could you clarify the difference between Args and CmdArgs?
  2. Could you use doxygen's syntax to specify the parameter _direction_? E.g.:
/// \param [in] Args The list of input arguments
/// \param [out] CmdArgs The list of output arguments

See https://www.doxygen.nl/manual/commands.html#cmdparam.

flang/include/flang/Frontend/PreprocessorOptions.h
26

Macros --> macros_ :)

flang/lib/Frontend/CompilerInvocation.cpp
213–214

[nit] Ideally this would use doxygen style: https://www.doxygen.nl/manual/commands.html#cmdparam. For example:

/// \param [in] ppOpts The preprocessor options
/// \param [out] opts The fortran options

Apologies for not making it clearer earlier.

234

I think that it would make sense to add a note in the commit message that -Dname predefines name as a macro with definition 1. Basically, a note that all macros default to 1.

flang/test/Flang-Driver/macro_def_undef.f90
19

[nit] Empty line not needed

37

[nit] Doesn't agree with the name of file. Also, I would just skip it.

flang/test/Flang-Driver/macro_multiline.f90
15

[nit] Empty line not needed

19

[nit] Repetition of the RUN line

24

[nit] Doesn't agree with the name of file. Also, I would just skip it.

25–26

I think that it would be useful to make this a bit stronger:

! CHECK: {{^start a end$}}
! CHECK-NOT: THIS_SHOULD_NOT_EXIST_IN_THE_OUTPUT
! CHECK-NOT: this_should_not_exist_in_the_output
START X END
FarisRehman edited the summary of this revision. (Show Details)Dec 23 2020, 3:51 AM
FarisRehman marked 5 inline comments as done.

Address review comments

This revision addresses comments by @awarzynski

Summary of changes:

  • Update commit summary
  • Use Doxygen style for input/output parameters
  • Make method and variable names lowercase where required
  • Remove unnecessary lines in added tests
  • Update multiline macro test to have a stronger test
FarisRehman marked 7 inline comments as done.Dec 23 2020, 4:19 AM
awarzynski accepted this revision.Jan 4 2021, 8:03 AM

Thank you for working on this @FarisRehman , LGTM! From what I can tell you also addressed the point that @tskeith raised (updated commit msg and added a dedicated test). Ccould you wait another day or two before merging? Just in case people have some other comments/suggestions.

clang/lib/Driver/ToolChains/Flang.h
27–31

[nit] One minor suggestion:

/// \param Args The list of input driver arguments
/// \param CmdArgs The list of output command arguments
This revision is now accepted and ready to land.Jan 4 2021, 8:03 AM
This revision was landed with ongoing or failed builds.Jan 6 2021, 8:17 AM
This revision was automatically updated to reflect the committed changes.
sameeranjoshi added inline comments.Jan 9 2021, 1:53 AM
flang/test/Flang-Driver/driver-help.f90
22

I see below crash report, when omitting the <value> but not omitting the = symbol.
Not sure if that's correct way of running hence instead of filing bug reporting here.

./bin/flang-new -E -DX= test.f90
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
Stack dump:
0.      Program arguments: /home/amd/f18_git/final_test/driver_build/bin/flang-new -fc1 -E -D X= -o - test.f90
 #0 0x00007f26185d0bc1 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/home/amd/f18_git/final_test/driver_build/bin/../lib/libLLVMSupport.so.12git+0x1a4bc1)
 #1 0x00007f26185ce9a4 llvm::sys::RunSignalHandlers() (/home/amd/f18_git/final_test/driver_build/bin/../lib/libLLVMSupport.so.12git+0x1a29a4)
 #2 0x00007f26185ceb10 SignalHandler(int) (/home/amd/f18_git/final_test/driver_build/bin/../lib/libLLVMSupport.so.12git+0x1a2b10)
 #3 0x00007f2617749470 (/lib/x86_64-linux-gnu/libc.so.6+0x46470)
 #4 0x00007f2617443498 Fortran::parser::OffsetToProvenanceMappings::Map(unsigned long) const (/home/amd/f18_git/final_test/driver_build/bin/../lib/../lib/libFortranParser.so.12git+0x571498)
 #5 0x00007f261744adf2 Fortran::parser::TokenSequence::GetProvenanceRange() const (/home/amd/f18_git/final_test/driver_build/bin/../lib/../lib/libFortranParser.so.12git+0x578df2)
 #6 0x00007f26173dd334 Fortran::parser::Preprocessor::MacroReplacement(Fortran::parser::TokenSequence const&, Fortran::parser::Prescanner&) (.localalias) (/home/amd/f18_git/final_test/driver_build/bin/../lib/../lib/libFortranParser.so.12git+0x50b334)
 #7 0x00007f26173e9dc1 Fortran::parser::Prescanner::Statement() (.localalias) (/home/amd/f18_git/final_test/driver_build/bin/../lib/../lib/libFortranParser.so.12git+0x517dc1)
 #8 0x00007f26173ea888 Fortran::parser::Prescanner::Prescan(Fortran::common::Interval<Fortran::parser::Provenance>) (.localalias) (/home/amd/f18_git/final_test/driver_build/bin/../lib/../lib/libFortranParser.so.12git+0x518888)
 #9 0x00007f26173d480b Fortran::parser::Parsing::Prescan(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Fortran::parser::Options) (/home/amd/f18_git/final_test/driver_build/bin/../lib/../lib/libFortranParser.so.12git+0x50280b)
#10 0x00007f2618424ddd Fortran::frontend::FrontendAction::Execute() (/home/amd/f18_git/final_test/driver_build/bin/../lib/libflangFrontend.so.12git+0xdddd)
#11 0x00007f261841fa5e Fortran::frontend::CompilerInstance::ExecuteAction(Fortran::frontend::FrontendAction&) (/home/amd/f18_git/final_test/driver_build/bin/../lib/libflangFrontend.so.12git+0x8a5e)
#12 0x00007f26184132fc Fortran::frontend::ExecuteCompilerInvocation(Fortran::frontend::CompilerInstance*) (/home/amd/f18_git/final_test/driver_build/bin/../lib/libflangFrontendTool.so.12git+0x12fc)
#13 0x000055bad4081c20 fc1_main(llvm::ArrayRef<char const*>, char const*) (/home/amd/f18_git/final_test/driver_build/bin/flang-new+0x3c20)
#14 0x000055bad4080e59 main (/home/amd/f18_git/final_test/driver_build/bin/flang-new+0x2e59)
#15 0x00007f261772a1e3 __libc_start_main /build/glibc-5mDdLG/glibc-2.30/csu/../csu/libc-start.c:342:3
#16 0x000055bad4080eae _start (/home/amd/f18_git/final_test/driver_build/bin/flang-new+0x2eae)
flang-new: error: unable to execute command: Segmentation fault (core dumped)
flang-new: error: flang frontend command failed due to signal (use -v to see invocation)
flang-new version 12.0.0 (https://github.com/llvm/llvm-project.git 2f9cb090cc6db1be5bf524eb0a32537503b3e786)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/amd/f18_git/final_test/driver_build/bin
flang-new: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.
FarisRehman added inline comments.Jan 11 2021, 3:29 AM
flang/test/Flang-Driver/driver-help.f90
22

Thanks for reporting this!
I was able to reproduce this error running:

  • flang-new -DX= file.f
  • flang-new -E -DX= file.f
  • flang-new -fc1 -E -DX= file.f
  • f18 -E -DX= file.f

However I did not receive an error when running: flang-new -fc1 -DX= file.f
I suspect this may be an issue in the frontend and I will look into this further.

FarisRehman marked an inline comment as done.Jan 15 2021, 1:51 AM
FarisRehman added inline comments.
flang/test/Flang-Driver/driver-help.f90
22

The bug seems to occur in the frontend and a bug report has been filed: https://bugs.llvm.org/show_bug.cgi?id=48747

clementval added inline comments.
flang/include/flang/Frontend/CompilerInvocation.h
89

@awarzynski I was looking at this file recently and I was wondering why we have different case style? Is there a particular reason?

flang/include/flang/Frontend/PreprocessorOptions.h
43

You have a couple of these on your patch.

awarzynski added inline comments.Jan 27 2022, 8:50 AM
flang/include/flang/Frontend/CompilerInvocation.h
89

Thanks for pointing this out! See:https://reviews.llvm.org/D118381

Is there a particular reason?

Tl;Dr This is a typo - I blame the reviewers :)

A bit longer discussion
The driver strives to follow Flang's C++ style. That was pretty much the only style used/available in LLVM Flang when the driver was introduced. Basically, it looked like the only available option. I didn't realise that in lowering and code-gen people decided to use the MLIR style. In hindsight, the driver should follow that too (to be better aligned with the rest of LLVM).

Flang's C++ style is very different to LLVM's, Clang's or MLIR's coding styles. Since the driver "glues" these projects together, it effectively needs to keep switching between the styles (i.e. LLVM and Clang's APIs fallow their style, MLIR's APIs fallow MLIR style, and Flang's parser/sema APIs follow Flang's style). To me personally, this has been very confusing and a major pain point.

I think that it would be good for the overall health of the project if the driver was refactored to follow the MLIR style. What are you thoughts?

flang/include/flang/Frontend/PreprocessorOptions.h
43
flang/test/Flang-Driver/macro_def_undef.f90
44