[flang][driver] Add support for `-D`, `-U`

[FarisRehman]

Summary

Add support for options -D and -U in the Flang driver.

Summary of changes:

• Create PreprocessorOptions, to be used by the driver then translated into Fortran::parser::Options
• Create CompilerInvocation::setFortranOpts to pass preprocessor options into the parser options
• Add a dedicated method, Flang::AddPreprocessingOptions, to extract preprocessing options from the driver arguments into the preprocessor command arguments

Macros specified like -DName will default to definition 1.

When defining macros, the new driver will drop anything after an end-of-line character. This is consistent with gfortran and clang, but different to what currently f18 does. However, flang (which is a bash wrapper for f18), also drops everything after an end-of-line character (at least in the bash on my Linux machine). So gfortran-like behaviour felt like the natural choice. Add a test to demonstrate this behaviour.

[awarzynski]

@FarisRehman, thank you for working on this! This looks really good and I think that it's almost ready. I did leave a few comments, but most are a matter of style and should be easy to address.

clang/lib/Driver/ToolChains/Flang.cpp

72–73

Clang deals with the preprocessor options in a dedicated method, AddPreprocessingOptions. See here:
https://github.com/llvm/llvm-project/blob/eba09a2db9eab46832cb7ec7ef0d2c227747772a/clang/lib/Driver/ToolChains/Clang.cpp#L1077

I feel that it might be a good idea to follow that. The preprocessor options (and the corresponding logic) are bound to grow, so we may as well add a dedicated method sooner rather than later.

75

AFAIK, when you call AddAllArgs the corresponding options are eventually _claimed_:
https://github.com/llvm/llvm-project/blob/eba09a2db9eab46832cb7ec7ef0d2c227747772a/llvm/lib/Option/ArgList.cpp#L112
So, IIUC, this line is not needed.

77

This change is not needed, is it?

flang/include/flang/Frontend/CompilerInvocation.h

35

1. Could you add a comment explaining what this is for?

2. Could you follow the current semantics used in the file and make it a std::shared_ptr? Long term we can expect PreprocessorOptions to grow big and at that point we wouldn't want to copy it too often (especially when compiling a lot of files).

85–89

IIUC, you are renaming this method because it is finally translating the user provided options (i.e. preprocessorOpts()) into Frontend options (i.e. fortranOptions). However, with your changes this method:

• sets some _sane_ predefined defaults
• adds some stuff from the user (the preprocessor defines)

So it's somewhere in between SetDefaultFortranOpts and SetFortranOpts (i.e. both names are _almost_ accurate). Perhaps splitting this into two methods would be better? E.g.:

• SetDefaultFortranOpts (leave as is)
• SetFortranOpts (implements new functionality, to replace SetDefaultFortranOpts eventually)

flang/include/flang/Frontend/PreprocessorOptions.h

2

Missing file header: https://llvm.org/docs/CodingStandards.html#file-headers

9

[nit] No need to repeat the name of the class.

13

[nit] IMO this would be preferred : /*isUndef=*/bool (https://llvm.org/docs/CodingStandards.html#comment-formatting) (currently you have bool /*isUndef*/)

flang/lib/Frontend/CompilerInvocation.cpp

161

I couldn't find any notes on naming static functions here: https://github.com/llvm/llvm-project/blob/main/flang/docs/C%2B%2Bstyle.md. This means that we fallow this: https://llvm.org/docs/CodingStandards.html#name-types-functions-variables-and-enumerators-properly. So, parsePreprocessorArgs instead of ParsePreprocessorArgs.

Also, could you add a comment briefly explaining what the method is for? E.g. Parses all preprocessor input arguments and populates the preprocessor options accordingly? Ideally doxygen :) See example here: https://github.com/llvm/llvm-project/blob/eba09a2db9eab46832cb7ec7ef0d2c227747772a/flang/include/flang/Frontend/CompilerInstance.h#L158-L163

164

Variable names should start with lower case.

[nit] Perhaps currentArg instead of A for the sake of being explicit? Or curArg?

249

This is not quite true, because you're collecting them into preprocessor options :)

250

static void instead (we don't need this method to be visible outside this translation unit).

252

All variables here have to be lower case:.

flang/test/Flang-Driver/macro.f90

1 ↗

(On Diff #312220)

There are no tests for e.g. -DSOME_VAR=SOME_VAL. It would be good to test that SOME_VAL was parsed correctly.

[nit[ Perhaps macro_def_undef.f90 would be more descriptive? (re filename).

[tskeith]

flang/lib/Frontend/CompilerInvocation.cpp

272

This is a change in behavior from f18, right? If so, the commit message should mention it and why it's changing.

[FarisRehman]

Address review comments

Address comments with the following changes:

• Add a dedicated method to adding preprocessing options in Flang.cpp
• Change preprocessorOpts to use std::shared_ptr
• Separate SetDefaultFortranOpts from SetFortranOpts in CompilerInvocation
• Fix method and variable naming in CompilerInvocation.cpp
• Add comments to new methods and fields
• Add missing file header to PreprocessorOptions.h
• Remove unnecessary changes and method calls
• Add a regression test to check end-of-line character behaviour in a macro definition

[FarisRehman]

flang/lib/Frontend/CompilerInvocation.cpp

272

Updated the summary to address this, thanks

[FarisRehman]

flang/lib/Frontend/CompilerInvocation.cpp

249

Added a comment to clarify the input and output arguments in the latest revision 👍

[awarzynski]

@FarisRehman , thank you for updating this! I have two high-level comments:

1. In your summary:

Change the way the driver handles end-of-line characters in macro definitions.

That's a a bit misleading - which driver do you mean? You're not changing the behavior of the current driver, flang. Also, this functionality doesn't yet exist in the new driver, flang-new, so you're not changing it there either. You can probably remove this sentence and just make it clear that you meant the new driver.

2. Could you doxygen style comments for input and output parameters in the new methods that you introduce? Ta! https://www.doxygen.nl/manual/commands.html#cmdparam

More comments inline.

clang/lib/Driver/ToolChains/Flang.cpp

36

This is not used until line 73. Perhaps better to define it near to where it's used?

https://isocpp.org/wiki/faq/coding-standards#declare-near-first-use

clang/lib/Driver/ToolChains/Flang.h

27–31

1. Could you clarify the difference between Args and CmdArgs?
2. Could you use doxygen's syntax to specify the parameter _direction_? E.g.:

/// \param [in] Args The list of input arguments
/// \param [out] CmdArgs The list of output arguments

See https://www.doxygen.nl/manual/commands.html#cmdparam.

flang/include/flang/Frontend/PreprocessorOptions.h

27

Macros --> macros_ :)

flang/lib/Frontend/CompilerInvocation.cpp

216–217

[nit] Ideally this would use doxygen style: https://www.doxygen.nl/manual/commands.html#cmdparam. For example:

/// \param [in] ppOpts The preprocessor options
/// \param [out] opts The fortran options

Apologies for not making it clearer earlier.

237

I think that it would make sense to add a note in the commit message that -Dname predefines name as a macro with definition 1. Basically, a note that all macros default to 1.

flang/test/Flang-Driver/macro_def_undef.f90

20

[nit] Empty line not needed

38

[nit] Doesn't agree with the name of file. Also, I would just skip it.

flang/test/Flang-Driver/macro_multiline.f90

16

[nit] Empty line not needed

20

[nit] Repetition of the RUN line

25

[nit] Doesn't agree with the name of file. Also, I would just skip it.

26–27

I think that it would be useful to make this a bit stronger:

! CHECK: {{^start a end$}}
! CHECK-NOT: THIS_SHOULD_NOT_EXIST_IN_THE_OUTPUT
! CHECK-NOT: this_should_not_exist_in_the_output
START X END

[FarisRehman]

Address review comments

This revision addresses comments by @awarzynski

Summary of changes:

• Update commit summary
• Use Doxygen style for input/output parameters
• Make method and variable names lowercase where required
• Remove unnecessary lines in added tests
• Update multiline macro test to have a stronger test

[awarzynski]

Thank you for working on this @FarisRehman , LGTM! From what I can tell you also addressed the point that @tskeith raised (updated commit msg and added a dedicated test). Ccould you wait another day or two before merging? Just in case people have some other comments/suggestions.

clang/lib/Driver/ToolChains/Flang.h

27–31

[nit] One minor suggestion:

/// \param Args The list of input driver arguments
/// \param CmdArgs The list of output command arguments

[sameeranjoshi]

flang/test/Flang-Driver/driver-help.f90

22

I see below crash report, when omitting the <value> but not omitting the = symbol.
Not sure if that's correct way of running hence instead of filing bug reporting here.

./bin/flang-new -E -DX= test.f90

PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
Stack dump:
0.      Program arguments: /home/amd/f18_git/final_test/driver_build/bin/flang-new -fc1 -E -D X= -o - test.f90
 #0 0x00007f26185d0bc1 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/home/amd/f18_git/final_test/driver_build/bin/../lib/libLLVMSupport.so.12git+0x1a4bc1)
 #1 0x00007f26185ce9a4 llvm::sys::RunSignalHandlers() (/home/amd/f18_git/final_test/driver_build/bin/../lib/libLLVMSupport.so.12git+0x1a29a4)
 #2 0x00007f26185ceb10 SignalHandler(int) (/home/amd/f18_git/final_test/driver_build/bin/../lib/libLLVMSupport.so.12git+0x1a2b10)
 #3 0x00007f2617749470 (/lib/x86_64-linux-gnu/libc.so.6+0x46470)
 #4 0x00007f2617443498 Fortran::parser::OffsetToProvenanceMappings::Map(unsigned long) const (/home/amd/f18_git/final_test/driver_build/bin/../lib/../lib/libFortranParser.so.12git+0x571498)
 #5 0x00007f261744adf2 Fortran::parser::TokenSequence::GetProvenanceRange() const (/home/amd/f18_git/final_test/driver_build/bin/../lib/../lib/libFortranParser.so.12git+0x578df2)
 #6 0x00007f26173dd334 Fortran::parser::Preprocessor::MacroReplacement(Fortran::parser::TokenSequence const&, Fortran::parser::Prescanner&) (.localalias) (/home/amd/f18_git/final_test/driver_build/bin/../lib/../lib/libFortranParser.so.12git+0x50b334)
 #7 0x00007f26173e9dc1 Fortran::parser::Prescanner::Statement() (.localalias) (/home/amd/f18_git/final_test/driver_build/bin/../lib/../lib/libFortranParser.so.12git+0x517dc1)
 #8 0x00007f26173ea888 Fortran::parser::Prescanner::Prescan(Fortran::common::Interval<Fortran::parser::Provenance>) (.localalias) (/home/amd/f18_git/final_test/driver_build/bin/../lib/../lib/libFortranParser.so.12git+0x518888)
 #9 0x00007f26173d480b Fortran::parser::Parsing::Prescan(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Fortran::parser::Options) (/home/amd/f18_git/final_test/driver_build/bin/../lib/../lib/libFortranParser.so.12git+0x50280b)
#10 0x00007f2618424ddd Fortran::frontend::FrontendAction::Execute() (/home/amd/f18_git/final_test/driver_build/bin/../lib/libflangFrontend.so.12git+0xdddd)
#11 0x00007f261841fa5e Fortran::frontend::CompilerInstance::ExecuteAction(Fortran::frontend::FrontendAction&) (/home/amd/f18_git/final_test/driver_build/bin/../lib/libflangFrontend.so.12git+0x8a5e)
#12 0x00007f26184132fc Fortran::frontend::ExecuteCompilerInvocation(Fortran::frontend::CompilerInstance*) (/home/amd/f18_git/final_test/driver_build/bin/../lib/libflangFrontendTool.so.12git+0x12fc)
#13 0x000055bad4081c20 fc1_main(llvm::ArrayRef<char const*>, char const*) (/home/amd/f18_git/final_test/driver_build/bin/flang-new+0x3c20)
#14 0x000055bad4080e59 main (/home/amd/f18_git/final_test/driver_build/bin/flang-new+0x2e59)
#15 0x00007f261772a1e3 __libc_start_main /build/glibc-5mDdLG/glibc-2.30/csu/../csu/libc-start.c:342:3
#16 0x000055bad4080eae _start (/home/amd/f18_git/final_test/driver_build/bin/flang-new+0x2eae)
flang-new: error: unable to execute command: Segmentation fault (core dumped)
flang-new: error: flang frontend command failed due to signal (use -v to see invocation)
flang-new version 12.0.0 (https://github.com/llvm/llvm-project.git 2f9cb090cc6db1be5bf524eb0a32537503b3e786)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/amd/f18_git/final_test/driver_build/bin
flang-new: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.

[FarisRehman]

flang/test/Flang-Driver/driver-help.f90

22

Thanks for reporting this!
I was able to reproduce this error running:

• flang-new -DX= file.f
• flang-new -E -DX= file.f
• flang-new -fc1 -E -DX= file.f
• f18 -E -DX= file.f

However I did not receive an error when running: flang-new -fc1 -DX= file.f
I suspect this may be an issue in the frontend and I will look into this further.

[FarisRehman]

flang/test/Flang-Driver/driver-help.f90

22

The bug seems to occur in the frontend and a bug report has been filed: https://bugs.llvm.org/show_bug.cgi?id=48747

[clementval]

flang/include/flang/Frontend/CompilerInvocation.h

89

@awarzynski I was looking at this file recently and I was wondering why we have different case style? Is there a particular reason?

flang/include/flang/Frontend/PreprocessorOptions.h

43

You have a couple of these on your patch.

[awarzynski]

flang/include/flang/Frontend/CompilerInvocation.h

89

Thanks for pointing this out! See:https://reviews.llvm.org/D118381

Is there a particular reason?

Tl;Dr This is a typo - I blame the reviewers :)

A bit longer discussion
The driver strives to follow Flang's C++ style. That was pretty much the only style used/available in LLVM Flang when the driver was introduced. Basically, it looked like the only available option. I didn't realise that in lowering and code-gen people decided to use the MLIR style. In hindsight, the driver should follow that too (to be better aligned with the rest of LLVM).

Flang's C++ style is very different to LLVM's, Clang's or MLIR's coding styles. Since the driver "glues" these projects together, it effectively needs to keep switching between the styles (i.e. LLVM and Clang's APIs fallow their style, MLIR's APIs fallow MLIR style, and Flang's parser/sema APIs follow Flang's style). To me personally, this has been very confusing and a major pain point.

I think that it would be good for the overall health of the project if the driver was refactored to follow the MLIR style. What are you thoughts?

flang/include/flang/Frontend/PreprocessorOptions.h

43

Fixed in https://reviews.llvm.org/D97080

flang/test/Flang-Driver/macro_def_undef.f90

39

@clementval Fixed in https://reviews.llvm.org/D99292

clang/include/clang/Driver/Options.td

def B : JoinedOrSeparate<["-"], "B">, MetaVarName<"<dir>">,
     HelpText<"Add <dir> to search path for binaries and object files used implicitly">;
 def CC : Flag<["-"], "CC">, Flags<[CC1Option]>, Group<Preprocessor_Group>,
     HelpText<"Include comments from within macros in preprocessed output">,
     MarshallingInfoFlag<"PreprocessorOutputOpts.ShowMacroComments">;
 def C : Flag<["-"], "C">, Flags<[CC1Option]>, Group<Preprocessor_Group>,
     HelpText<"Include comments in preprocessed output">,
     MarshallingInfoFlag<"PreprocessorOutputOpts.ShowComments">;
 def D : JoinedOrSeparate<["-"], "D">, Group<Preprocessor_Group>,
-    Flags<[CC1Option]>, MetaVarName<"<macro>=<value>">,
+    Flags<[CC1Option, FlangOption, FC1Option]>, MetaVarName<"<macro>=<value>">,
     HelpText<"Define <macro> to <value> (or 1 if <value> omitted)">;
 def E : Flag<["-"], "E">, Flags<[NoXarchOption,CC1Option, FlangOption, FC1Option]>, Group<Action_Group>,
     HelpText<"Only run the preprocessor">;
 def F : JoinedOrSeparate<["-"], "F">, Flags<[RenderJoined,CC1Option]>,
     HelpText<"Add directory to framework include search path">;
 def G : JoinedOrSeparate<["-"], "G">, Flags<[NoXarchOption]>, Group<m_Group>,
     MetaVarName<"<size>">, HelpText<"Put objects of at most <size> bytes "
     "into small data section (MIPS / Hexagon)">;
def Tbss : JoinedOrSeparate<["-"], "Tbss">, Group<T_Group>,
   MetaVarName<"<addr>">, HelpText<"Set starting address of BSS to <addr>">;
 def Tdata : JoinedOrSeparate<["-"], "Tdata">, Group<T_Group>,
   MetaVarName<"<addr>">, HelpText<"Set starting address of DATA to <addr>">;
 def Ttext : JoinedOrSeparate<["-"], "Ttext">, Group<T_Group>,
   MetaVarName<"<addr>">, HelpText<"Set starting address of TEXT to <addr>">;
 def T : JoinedOrSeparate<["-"], "T">, Group<T_Group>,
   MetaVarName<"<script>">, HelpText<"Specify <script> as linker script">;
 def U : JoinedOrSeparate<["-"], "U">, Group<Preprocessor_Group>,
-  Flags<[CC1Option]>, MetaVarName<"<macro>">, HelpText<"Undefine macro <macro>">;
+  Flags<[CC1Option, FlangOption, FC1Option]>, MetaVarName<"<macro>">, HelpText<"Undefine macro <macro>">;
 def V : JoinedOrSeparate<["-"], "V">, Flags<[NoXarchOption, Unsupported]>;
 def Wa_COMMA : CommaJoined<["-"], "Wa,">,
   HelpText<"Pass the comma separated arguments in <arg> to the assembler">,
   MetaVarName<"<arg>">;
 def Wall : Flag<["-"], "Wall">, Group<W_Group>, Flags<[CC1Option, HelpHidden]>;
 def WCL4 : Flag<["-"], "WCL4">, Group<W_Group>, Flags<[CC1Option, HelpHidden]>;
 def Wdeprecated : Flag<["-"], "Wdeprecated">, Group<W_Group>, Flags<[CC1Option]>,
   HelpText<"Enable warnings for deprecated constructs and define __DEPRECATED">;

clang/lib/Driver/ToolChains/Flang.h

 
 namespace clang {
 namespace driver {
 
 namespace tools {
 
 /// Flang compiler tool.
 class LLVM_LIBRARY_VISIBILITY Flang : public Tool {
+private:
+  /// Extract preprocessing options from the driver arguments and add them to
+  /// the preprocessor command arguments.
+  ///
+  /// \param [in] Args The list of input driver arguments
+  /// \param [out] CmdArgs The list of output command arguments
+  void AddPreprocessingOptions(const llvm::opt::ArgList &Args,
+                               llvm::opt::ArgStringList &CmdArgs) const;
+
 public:
   Flang(const ToolChain &TC);
   ~Flang() override;
 
   bool hasGoodDiagnostics() const override { return true; }
   bool hasIntegratedAssembler() const override { return true; }
   bool hasIntegratedCPP() const override { return true; }
   bool canEmitIR() const override { return true; }

clang/lib/Driver/ToolChains/Flang.cpp

 
 #include <cassert>
 
 using namespace clang::driver;
 using namespace clang::driver::tools;
 using namespace clang;
 using namespace llvm::opt;
 
+void Flang::AddPreprocessingOptions(const ArgList &Args,
+                                    ArgStringList &CmdArgs) const {
+  Args.AddAllArgs(CmdArgs, {options::OPT_D, options::OPT_U});
+}
+
 void Flang::ConstructJob(Compilation &C, const JobAction &JA,
                          const InputInfo &Output, const InputInfoList &Inputs,
                          const ArgList &Args, const char *LinkingOutput) const {
   const auto &TC = getToolChain();
   const llvm::Triple &Triple = TC.getEffectiveTriple();
   const std::string &TripleStr = Triple.getTriple();
 
   ArgStringList CmdArgs;
 
   CmdArgs.push_back("-fc1");
 
   // TODO: Eventually all actions will require a triple (e.g. `-triple
   // aarch64-unknown-linux-gnu`). However, `-triple` is currently not supported
   // by `flang-new -fc1`, so we only add it selectively to actions that we
   // don't support/execute just yet.
   if (isa<PreprocessJobAction>(JA)) {
     if (C.getArgs().hasArg(options::OPT_test_io))
       CmdArgs.push_back("-test-io");
void Flang::ConstructJob(Compilation &C, const JobAction &JA,
   } else if (isa<AssembleJobAction>(JA)) {
     CmdArgs.push_back("-triple");
     CmdArgs.push_back(Args.MakeArgString(TripleStr));
     CmdArgs.push_back("-emit-obj");
   } else {
     assert(false && "Unexpected action class for Flang tool.");
   }
 
+  const InputInfo &Input = Inputs[0];
+  types::ID InputType = Input.getType();
+
+  // Add preprocessing options like -I, -D, etc. if we are using the
+  // preprocessor (i.e. skip when dealing with e.g. binary files).
+  if (types::getPreprocessedType(InputType) != types::TY_INVALID)
+    AddPreprocessingOptions(Args, CmdArgs);
+
   if (Output.isFilename()) {
     CmdArgs.push_back("-o");
     CmdArgs.push_back(Output.getFilename());
   } else {
     assert(Output.isNothing() && "Invalid output.");
   }
 
-  const InputInfo &Input = Inputs[0];
   assert(Input.isFilename() && "Invalid input.");
   CmdArgs.push_back(Input.getFilename());
 
   const auto& D = C.getDriver();
   // TODO: Replace flang-new with flang once the new driver replaces the
   // throwaway driver
   const char *Exec = Args.MakeArgString(D.GetProgramPath("flang-new", TC));
   C.addCommand(std::make_unique<Command>(JA, *this,
                                          ResponseFileSupport::AtFileUTF8(),
                                          Exec, CmdArgs, Inputs, Output));
 }
 
 Flang::Flang(const ToolChain &TC) : Tool("flang-new", "flang frontend", TC) {}
 
 Flang::~Flang() {}

flang/include/flang/Frontend/CompilerInstance.h

 //===-- CompilerInstance.h - Flang Compiler Instance ------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 #ifndef LLVM_FLANG_FRONTEND_COMPILERINSTANCE_H
 #define LLVM_FLANG_FRONTEND_COMPILERINSTANCE_H
 
 #include "flang/Frontend/CompilerInvocation.h"
 #include "flang/Frontend/FrontendAction.h"
+#include "flang/Frontend/PreprocessorOptions.h"
 #include "flang/Parser/parsing.h"
 #include "flang/Parser/provenance.h"
 #include "flang/Semantics/semantics.h"
 #include "llvm/Support/raw_ostream.h"
 
 namespace Fortran::frontend {
 
 class CompilerInstance {
const clang::DiagnosticOptions &GetDiagnosticOpts() const {
     return invocation_->GetDiagnosticOpts();
   }
 
   FrontendOptions &frontendOpts() { return invocation_->frontendOpts(); }
   const FrontendOptions &frontendOpts() const {
     return invocation_->frontendOpts();
   }
 
+  PreprocessorOptions &preprocessorOpts() {
+    return invocation_->preprocessorOpts();
+  }
+  const PreprocessorOptions &preprocessorOpts() const {
+    return invocation_->preprocessorOpts();
+  }
+
   /// }
   /// @name Diagnostics Engine
   /// {
 
   bool HasDiagnostics() const { return diagnostics_ != nullptr; }
 
   /// Get the current diagnostics engine.
   clang::DiagnosticsEngine &diagnostics() const {

flang/include/flang/Frontend/CompilerInvocation.h

 //===- CompilerInvocation.h - Compiler Invocation Helper Data ---*- C -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 #ifndef LLVM_FLANG_FRONTEND_COMPILERINVOCATION_H
 #define LLVM_FLANG_FRONTEND_COMPILERINVOCATION_H
 
 #include "flang/Frontend/FrontendOptions.h"
+#include "flang/Frontend/PreprocessorOptions.h"
 #include "flang/Parser/parsing.h"
 #include "clang/Basic/Diagnostic.h"
 #include "clang/Basic/DiagnosticOptions.h"
 #include "llvm/Option/ArgList.h"
+#include <memory>
 
 namespace Fortran::frontend {
 
 /// Fill out Opts based on the options given in Args.
 ///
 /// When errors are encountered, return false and, if Diags is non-null,
 /// report the error(s).
 bool ParseDiagnosticArgs(clang::DiagnosticOptions &opts,
     llvm::opt::ArgList &args, bool defaultDiagColor = true);
 
 class CompilerInvocationBase {
 public:
   /// Options controlling the diagnostic engine.
   llvm::IntrusiveRefCntPtr<clang::DiagnosticOptions> diagnosticOpts_;
+  /// Options for the preprocessor.
+  std::shared_ptr<Fortran::frontend::PreprocessorOptions> preprocessorOpts_;
 
   CompilerInvocationBase();
   CompilerInvocationBase(const CompilerInvocationBase &x);
   ~CompilerInvocationBase();
 
   clang::DiagnosticOptions &GetDiagnosticOpts() {
     return *diagnosticOpts_.get();
   }
   const clang::DiagnosticOptions &GetDiagnosticOpts() const {
     return *diagnosticOpts_.get();
   }
+
+  PreprocessorOptions &preprocessorOpts() { return *preprocessorOpts_; }
+  const PreprocessorOptions &preprocessorOpts() const {
+    return *preprocessorOpts_;
+  }
 };
 
 class CompilerInvocation : public CompilerInvocationBase {
   /// Options for the frontend driver
   // TODO: Merge with or translate to parserOpts_. We shouldn't need two sets of
   // options.
   FrontendOptions frontendOpts_;
 
static bool CreateFromArgs(CompilerInvocation &res,
       llvm::ArrayRef<const char *> commandLineArgs,
       clang::DiagnosticsEngine &diags);
 
   /// Set the Fortran options to predifined defaults. These defaults are
   /// consistend with f18/f18.cpp.
   // TODO: We should map frontendOpts_ to parserOpts_ instead. For that, we
   // need to extend frontendOpts_ first. Next, we need to add the corresponding
   // compiler driver options in libclangDriver.
   void SetDefaultFortranOpts();
+
+  /// Set the Fortran options to user-specified values.
+  /// These values are found in the preprocessor options.
+  void setFortranOpts();
 };
 
 } // end namespace Fortran::frontend
 #endif // LLVM_FLANG_FRONTEND_COMPILERINVOCATION_H

flang/include/flang/Frontend/PreprocessorOptions.h

+//===- PreprocessorOptions.h ------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// This file contains the declaration of the PreprocessorOptions class, which
+/// is the class for all preprocessor options.
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_FLANG_PREPROCESSOROPTIONS_H
+#define LLVM_FLANG_PREPROCESSOROPTIONS_H
+
+#include "llvm/ADT/StringRef.h"
+
+namespace Fortran::frontend {
+
+/// This class is used for passing the various options used
+/// in preprocessor initialization to the parser options.
+class PreprocessorOptions {
+public:
+  std::vector<std::pair<std::string, /*isUndef*/ bool>> macros;
+
+public:
+  PreprocessorOptions() {}
+
+  void addMacroDef(llvm::StringRef name) {
+    macros.emplace_back(std::string(name), false);
+  }
+
+  void addMacroUndef(llvm::StringRef name) {
+    macros.emplace_back(std::string(name), true);
+  }
+};
+
+} // namespace Fortran::frontend
+
+#endif // LLVM_FLANG_PREPROCESSOROPTIONS_H
 No newline at end of file

flang/lib/Frontend/CompilerInstance.cpp

void CompilerInstance::ClearOutputFiles(bool eraseFiles) {
   nonSeekStream_.reset();
 }
 
 bool CompilerInstance::ExecuteAction(FrontendAction &act) {
   // Set some sane defaults for the frontend.
   // TODO: Instead of defaults we should be setting these options based on the
   // user input.
   this->invocation().SetDefaultFortranOpts();
+  // Set the fortran options to user-based input.
+  this->invocation().setFortranOpts();
 
   // Connect Input to a CompileInstance
   for (const FrontendInputFile &fif : frontendOpts().inputs_) {
     if (act.BeginSourceFile(*this, fif)) {
       if (llvm::Error err = act.Execute()) {
         consumeError(std::move(err));
       }
       act.EndSourceFile();

flang/lib/Frontend/CompilerInvocation.cpp

 //===- CompilerInvocation.cpp ---------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "flang/Frontend/CompilerInvocation.h"
+#include "flang/Frontend/PreprocessorOptions.h"
 #include "clang/Basic/AllDiagnostics.h"
 #include "clang/Basic/DiagnosticDriver.h"
 #include "clang/Basic/DiagnosticOptions.h"
 #include "clang/Driver/DriverDiagnostic.h"
 #include "clang/Driver/Options.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/StringSwitch.h"
 #include "llvm/Option/Arg.h"
 #include "llvm/Option/ArgList.h"
 #include "llvm/Option/OptTable.h"
 #include "llvm/Support/Process.h"
 #include "llvm/Support/raw_ostream.h"
 
 using namespace Fortran::frontend;
 
 //===----------------------------------------------------------------------===//
 // Initialization.
 //===----------------------------------------------------------------------===//
 CompilerInvocationBase::CompilerInvocationBase()
-    : diagnosticOpts_(new clang::DiagnosticOptions()) {}
+    : diagnosticOpts_(new clang::DiagnosticOptions()),
+      preprocessorOpts_(new PreprocessorOptions()) {}
 
 CompilerInvocationBase::CompilerInvocationBase(const CompilerInvocationBase &x)
-    : diagnosticOpts_(new clang::DiagnosticOptions(x.GetDiagnosticOpts())) {}
+    : diagnosticOpts_(new clang::DiagnosticOptions(x.GetDiagnosticOpts())),
+      preprocessorOpts_(new PreprocessorOptions(x.preprocessorOpts())) {}
 
 CompilerInvocationBase::~CompilerInvocationBase() = default;
 
 //===----------------------------------------------------------------------===//
 // Deserialization (from args)
 //===----------------------------------------------------------------------===//
 static bool parseShowColorsArgs(
     const llvm::opt::ArgList &args, bool defaultColor) {
if (ik.IsUnknown()) {
         dashX = ik;
     }
 
     opts.inputs_.emplace_back(std::move(inputs[i]), ik);
   }
   return dashX;
 }
 
+/// Parses all preprocessor input arguments and populates the preprocessor
+/// options accordingly.
+///
+/// \param [in] opts The preprocessor options instance
+/// \param [out] args The list of input arguments
+static void parsePreprocessorArgs(
+    Fortran::frontend::PreprocessorOptions &opts, llvm::opt::ArgList &args) {
+  // Add macros from the command line.
+  for (const auto *currentArg : args.filtered(
+           clang::driver::options::OPT_D, clang::driver::options::OPT_U)) {
+    if (currentArg->getOption().matches(clang::driver::options::OPT_D)) {
+      opts.addMacroDef(currentArg->getValue());
+    } else {
+      opts.addMacroUndef(currentArg->getValue());
+    }
+  }
+}
+
 bool CompilerInvocation::CreateFromArgs(CompilerInvocation &res,
     llvm::ArrayRef<const char *> commandLineArgs,
     clang::DiagnosticsEngine &diags) {
 
   bool success = true;
 
   // Parse the arguments
   const llvm::opt::OptTable &opts = clang::driver::getDriverOptTable();
for (const auto *a : args.filtered(clang::driver::options::OPT_UNKNOWN)) {
     else
       diags.Report(clang::diag::err_drv_unknown_argument_with_suggestion)
           << argString << nearest;
     success = false;
   }
 
   // Parse the frontend args
   ParseFrontendArgs(res.frontendOpts(), args, diags);
+  // Parse the preprocessor args
+  parsePreprocessorArgs(res.preprocessorOpts(), args);
 
   return success;
 }
 
+/// Collect the macro definitions provided by the given preprocessor
+/// options into the parser options.
+///
+/// \param [in] ppOpts The preprocessor options
+/// \param [out] opts The fortran options
+static void collectMacroDefinitions(
+    const PreprocessorOptions &ppOpts, Fortran::parser::Options &opts) {
+  for (unsigned i = 0, n = ppOpts.macros.size(); i != n; ++i) {
+    llvm::StringRef macro = ppOpts.macros[i].first;
+    bool isUndef = ppOpts.macros[i].second;
+
+    std::pair<llvm::StringRef, llvm::StringRef> macroPair = macro.split('=');
+    llvm::StringRef macroName = macroPair.first;
+    llvm::StringRef macroBody = macroPair.second;
+
+    // For an #undef'd macro, we only care about the name.
+    if (isUndef) {
+      opts.predefinitions.emplace_back(
+          macroName.str(), std::optional<std::string>{});
+      continue;
+    }
+
+    // For a #define'd macro, figure out the actual definition.
+    if (macroName.size() == macro.size())
+      macroBody = "1";
+    else {
+      // Note: GCC drops anything following an end-of-line character.
+      llvm::StringRef::size_type End = macroBody.find_first_of("\n\r");
+      macroBody = macroBody.substr(0, End);
+    }
+    opts.predefinitions.emplace_back(
+        macroName, std::optional<std::string>(macroBody.str()));
+  }
+}
+
 void CompilerInvocation::SetDefaultFortranOpts() {
   auto &fortranOptions = fortranOpts();
 
   // These defaults are based on the defaults in f18/f18.cpp.
   std::vector<std::string> searchDirectories{"."s};
   fortranOptions.searchDirectories = searchDirectories;
   fortranOptions.isFixedForm = false;
 }
+
+void CompilerInvocation::setFortranOpts() {
+  auto &fortranOptions = fortranOpts();
+  const auto &preprocessorOptions = preprocessorOpts();
+
+  collectMacroDefinitions(preprocessorOptions, fortranOptions);
+}

flang/test/Flang-Driver/driver-help-hidden.f90

 
 !----------------------------------------------------
 ! EXPECTED OUTPUT FOR FLANG DRIVER (flang-new)
 !----------------------------------------------------
 ! CHECK:USAGE: flang-new
 ! CHECK-EMPTY:
 ! CHECK-NEXT:OPTIONS:
 ! CHECK-NEXT: -###      Print (but do not run) the commands to run for this compilation
+! CHECK-NEXT: -D <macro>=<value>     Define <macro> to <value> (or 1 if <value> omitted)
 ! CHECK-NEXT: -E        Only run the preprocessor
 ! CHECK-NEXT: -fcolor-diagnostics    Enable colors in diagnostics
 ! CHECK-NEXT: -fno-color-diagnostics Disable colors in diagnostics
 ! CHECK-NEXT: -help     Display available options
 ! CHECK-NEXT: -o <file> Write output to <file>
 ! CHECK-NEXT: -test-io  Run the InputOuputTest action. Use for development and testing only.
+! CHECK-NEXT: -U <macro>             Undefine macro <macro>
 ! CHECK-NEXT: --version Print version information
 
 !-------------------------------------------------------------
 ! EXPECTED OUTPUT FOR FLANG DRIVER (flang-new)
 !-------------------------------------------------------------
 ! ERROR-FLANG: error: unknown argument '-help-hidden'; did you mean '--help-hidden'?
 
 !-------------------------------------------------------------
 ! EXPECTED OUTPUT FOR FLANG FRONTEND DRIVER (flang-new -fc1)
 !-------------------------------------------------------------
 ! Frontend driver -help-hidden is not supported
 ! ERROR-FLANG-FC1: error: unknown argument: '{{.*}}'
 

flang/test/Flang-Driver/driver-help.f90

 
 !----------------------------------------------------
 ! EXPECTED OUTPUT FOR FLANG DRIVER (flang-new)
 !----------------------------------------------------
 ! HELP:USAGE: flang-new
 ! HELP-EMPTY:
 ! HELP-NEXT:OPTIONS:
 ! HELP-NEXT: -###                   Print (but do not run) the commands to run for this compilation
+! HELP-NEXT: -D <macro>=<value>     Define <macro> to <value> (or 1 if <value> omitted)
 ! HELP-NEXT: -E                     Only run the preprocessor
 ! HELP-NEXT: -fcolor-diagnostics    Enable colors in diagnostics
 ! HELP-NEXT: -fno-color-diagnostics Disable colors in diagnostics
 ! HELP-NEXT: -help                  Display available options
 ! HELP-NEXT: -o <file>              Write output to <file>
+! HELP-NEXT: -U <macro>             Undefine macro <macro>
 ! HELP-NEXT: --version              Print version information
 
 !-------------------------------------------------------------
 ! EXPECTED OUTPUT FOR FLANG FRONTEND DRIVER (flang-new -fc1)
 !-------------------------------------------------------------
 ! HELP-FC1:USAGE: flang-new
 ! HELP-FC1-EMPTY:
 ! HELP-FC1-NEXT:OPTIONS:
-! HELP-FC1-NEXT: -E        Only run the preprocessor
-! HELP-FC1-NEXT: -help     Display available options
-! HELP-FC1-NEXT: -o <file> Write output to <file>
-! HELP-FC1-NEXT: --version Print version information
+! HELP-FC1-NEXT: -D <macro>=<value>     Define <macro> to <value> (or 1 if <value> omitted)
+! HELP-FC1-NEXT: -E                     Only run the preprocessor
+! HELP-FC1-NEXT: -help                  Display available options
+! HELP-FC1-NEXT: -o <file>              Write output to <file>
+! HELP-FC1-NEXT: -U <macro>             Undefine macro <macro>
+! HELP-FC1-NEXT: --version              Print version information
 
 !---------------
 ! EXPECTED ERROR
 !---------------
 ! ERROR: error: unknown argument '-helps'; did you mean '-help'

flang/test/Flang-Driver/macro_def_undef.f90

+! Ensure arguments -D and -U work as expected.
+
+! REQUIRES: new-flang-driver
+
+!--------------------------
+! FLANG DRIVER (flang-new)
+!--------------------------
+! RUN: %flang-new -E %s  2>&1 | FileCheck %s --check-prefix=UNDEFINED
+! RUN: %flang-new -E -DX=A %s  2>&1 | FileCheck %s --check-prefix=DEFINED
+! RUN: %flang-new -E -DX=A -UX %s  2>&1 | FileCheck %s --check-prefix=UNDEFINED
+
+!-----------------------------------------
+! FRONTEND FLANG DRIVER (flang-new -fc1)
+!-----------------------------------------
+! RUN: %flang-new -fc1 -E %s  2>&1 | FileCheck %s --check-prefix=UNDEFINED
+! RUN: %flang-new -fc1 -E -DX=A %s  2>&1 | FileCheck %s --check-prefix=DEFINED
+! RUN: %flang-new -fc1 -E -DX -UX %s  2>&1 | FileCheck %s --check-prefix=UNDEFINED
+
+!--------------------------------------------
+! EXPECTED OUTPUT FOR AN UNDEFINED MACRO
+!--------------------------------------------
+! UNDEFINED:program b
+! UNDEFINED-NOT:program x
+! UNDEFINED-NEXT:end
+
+!--------------------------------------------
+! EXPECTED OUTPUT FOR MACRO 'X' DEFINED AS A
+!--------------------------------------------
+! DEFINED:program a
+! DEFINED-NOT:program b
+! DEFINED-NEXT:end
+
+#ifdef X
+program X
+#else
+program B
+#endif
+end
 No newline at end of file

flang/test/Flang-Driver/macro_multiline.f90

+! Ensure the end-of-line character and anything that follows after in a macro definition (-D) is ignored.
+
+! REQUIRES: new-flang-driver
+
+!--------------------------
+! FLANG DRIVER (flang-new)
+!--------------------------
+! RUN: printf -- "-DX=A\\\\\nTHIS_SHOULD_NOT_EXIST_IN_THE_OUTPUT\n" | xargs %flang-new -E %s  2>&1 | FileCheck --strict-whitespace --match-full-lines %s
+
+!-----------------------------------------
+! FRONTEND FLANG DRIVER (flang-new -fc1)
+!-----------------------------------------
+! RUN: printf -- "-DX=A\\\\\nTHIS_SHOULD_NOT_EXIST_IN_THE_OUTPUT\n" | xargs %flang-new -fc1 -E %s  2>&1 | FileCheck --strict-whitespace --match-full-lines %s
+
+!-------------------------------
+! EXPECTED OUTPUT FOR MACRO 'X'
+!-------------------------------
+! CHECK:start a end
+! CHECK-NOT:THIS_SHOULD_NOT_EXIST_IN_THE_OUTPUT
+! CHECK-NOT:this_should_not_exist_in_the_output
+
+START X END
 No newline at end of file