[flang][driver] Add support for `-cpp/-nocpp`

[awarzynski]

Summary

This patch adds support for the -cpp and -nocpp flags. The
implemented semantics match f18 (i.e. the "throwaway" driver), but are
different to gfortran. In Flang the preprocessor is always run and
-cpp/-nocpp are only used to control whether standard macro
predefinitions are added or not. In practice this is sufficient to model
gfortran`s -cpp/-nocpp.

In the absence of -cpp/-nocpp, the driver will use the extension of
the input file to decide whether to include the standard macro
predefinitions. gfortran's documentation [1] was used to decide which
file extension to use for this. The extension for:

• predefined-macros-compiler-version.f90

had to be updated accordingly.

The logic mentioned above was added in FrontendAction::BeginSourceFile.
That's relatively late in the driver set-up, but this roughly where the
name of the input file becomes available. The logic for deciding between
fixed and free form works in an a similar way and was also moved to
FrontendAction::BeginSourceFile for consistency (and to reduce
code-duplication).

The -cpp/-nocpp flags are respected also when the input is read from
stdin. This is different to:

• gfortran (behaves as if -cpp was used)
• f18 (behaves as if -nocpp was used)

[1] https://gcc.gnu.org/onlinedocs/gcc/Overall-Options.html

[awarzynski]

flang/include/flang/Frontend/CompilerInvocation.h

149–152

DELETEME

[kiranchandramohan]

minor comments.

flang/lib/Frontend/FrontendAction.cpp

73

Nit: prefernce -> preference
below also.

flang/lib/Frontend/FrontendOptions.cpp

23

Unrelated comment: f77 is probably not free form.

30

Consider adding fpp and making it match the behaviour of gfortran as given below.

"The preprocessor is automatically invoked if the file extension is .fpp, .FPP, .F, .FOR, .FTN, .F90, .F95, .F03 or .F08."
https://gcc.gnu.org/onlinedocs/gfortran/Preprocessing-Options.html

[awarzynski]

Address PR comments + updated comments + rebase

I refined some comments and added -fpp to the list of file extensions. Thank you @kiranchandramohan for pointing this out!

[tskeith]

clang/include/clang/Driver/Options.td

4305

This option affects command line macro definitions too. So maybe something like:
Enable predefined and command line preprocessor macros

[awarzynski]

Make sure that -cpp\-nocpp controls command line macro definitions too

As @tskeith pointed out, -cpp\-nocpp should also affect command line macro
definitions (on top of standard macro predefinitions). With this change, the
file extension becomes significant in all tests that depend on macros. I've
renamed some of them accordingly. Few other tests are updated to
demonstrate that the new flags work as expected.

[awarzynski]

clang/include/clang/Driver/Options.td

4305

I've actually missed this case altogether - thank you for pointing it out. Updated accordingly.

flang/lib/Frontend/FrontendOptions.cpp

23

Thanks for noticing and pointing out! https://reviews.llvm.org/D99494

[kiranchandramohan]

LGTM.

flang/include/flang/Frontend/CompilerInvocation.h

142

Nit: Misplaced? Don't see any params here.

151

Nit: Should this have a param in?

flang/include/flang/Frontend/FrontendOptions.h

147

Nit: implment -> implement

flang/lib/Frontend/FrontendActions.cpp

68

Nit: Looks similar to function above.

[awarzynski]

Update comments

flang/include/flang/Frontend/CompilerInvocation.h

142

Yes, needs updating.

151

Should be deleted, sorry!

flang/lib/Frontend/FrontendActions.cpp

68

Yes, they are identical up to L85 (in which Parsing starts):

// Parse. In case of failure, report and return.
ci.parsing().Parse(llvm::outs());

I agree that this is not great and should be refactored at some point. But it will be much easier once we have more actions and a better understanding of what is needed. I think that https://reviews.llvm.org/D99645 is an important step in this direction. It adds a new abstract class for frontend actions. Once it's merged, we will have 3 classes of actions:

• prescan
• prescan + parse
• prescan + parse + sema

So there will be a very clear split into 3 groups.

[awarzynski]

As there have been no new comments and this patch has already been accepted, I decided to merge it without waiting for more reviews. Please leave post-commit comments if I missed anything and I will address them. Thank you all for reviewing!

clang/include/clang/Driver/Options.td

 defm web : BooleanFFlag<"web">, Group<clang_ignored_gcc_optimization_f_Group>;
 defm whole_program : BooleanFFlag<"whole-program">, Group<clang_ignored_gcc_optimization_f_Group>;
 defm devirtualize : BooleanFFlag<"devirtualize">, Group<clang_ignored_gcc_optimization_f_Group>;
 defm devirtualize_speculatively : BooleanFFlag<"devirtualize-speculatively">,
     Group<clang_ignored_gcc_optimization_f_Group>;
 
 // Generic gfortran options.
 def A_DASH : Joined<["-"], "A-">, Group<gfortran_Group>;
-def cpp : Flag<["-"], "cpp">, Group<gfortran_Group>;
-def nocpp : Flag<["-"], "nocpp">, Group<gfortran_Group>;
 def static_libgfortran : Flag<["-"], "static-libgfortran">, Group<gfortran_Group>;
 
 // "f" options with values for gfortran.
 def fblas_matmul_limit_EQ : Joined<["-"], "fblas-matmul-limit=">, Group<gfortran_Group>;
 def fcheck_EQ : Joined<["-"], "fcheck=">, Group<gfortran_Group>;
 def fcoarray_EQ : Joined<["-"], "fcoarray=">, Group<gfortran_Group>;
 def fconvert_EQ : Joined<["-"], "fconvert=">, Group<gfortran_Group>;
 def ffpe_trap_EQ : Joined<["-"], "ffpe-trap=">, Group<gfortran_Group>;
def Xflang : Separate<["-"], "Xflang">,
   Flags<[NoXarchOption, CoreOption]>, Group<CompileOnly_Group>;
 }
 
 //===----------------------------------------------------------------------===//
 // FlangOption and FC1 Options
 //===----------------------------------------------------------------------===//
 let Flags = [FC1Option, FlangOption, FlangOnlyOption] in {
 
+def cpp : Flag<["-"], "cpp">, Group<f_Group>,
+  HelpText<"Enable predefined and command line preprocessor macros">;
+def nocpp : Flag<["-"], "nocpp">, Group<f_Group>,
+  HelpText<"Disable predefined and command line preprocessor macros">;
 def module_dir : Separate<["-"], "module-dir">, MetaVarName<"<dir>">,
   HelpText<"Put MODULE files in <dir>">,
   DocBrief<[{This option specifies where to put .mod files for compiled modules.
 It is also added to the list of directories to be searched by an USE statement.
 The default is the current directory.}]>;
 
 def ffixed_form : Flag<["-"], "ffixed-form">, Group<f_Group>,
   HelpText<"Process source files in fixed form">;

clang/lib/Driver/ToolChains/Flang.cpp

Args.AddAllArgs(
                 options::OPT_fxor_operator, options::OPT_fno_xor_operator,
                 options::OPT_falternative_parameter_statement,
                 options::OPT_fdefault_real_8, options::OPT_fdefault_integer_8,
                 options::OPT_fdefault_double_8, options::OPT_flarge_sizes});
 }
 
 void Flang::AddPreprocessingOptions(const ArgList &Args,
                                     ArgStringList &CmdArgs) const {
-  Args.AddAllArgs(CmdArgs, {options::OPT_D, options::OPT_U, options::OPT_I});
+  Args.AddAllArgs(CmdArgs, {options::OPT_D, options::OPT_U, options::OPT_I,
+                            options::OPT_cpp, options::OPT_nocpp});
 }
 
 void Flang::AddOtherOptions(const ArgList &Args, ArgStringList &CmdArgs) const {
   Args.AddAllArgs(CmdArgs,
                   {options::OPT_module_dir, options::OPT_fdebug_module_writer,
                    options::OPT_fintrinsic_modules_path, options::OPT_pedantic,
                    options::OPT_std_EQ});
 }

flang/include/flang/Frontend/CompilerInvocation.h

 //===- CompilerInvocation.h - Compiler Invocation Helper Data ---*- C -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 #ifndef LLVM_FLANG_FRONTEND_COMPILERINVOCATION_H
 #define LLVM_FLANG_FRONTEND_COMPILERINVOCATION_H
 
 #include "flang/Frontend/FrontendOptions.h"
 #include "flang/Frontend/PreprocessorOptions.h"
 #include "flang/Parser/parsing.h"
 #include "flang/Semantics/semantics.h"
 #include "clang/Basic/Diagnostic.h"
 #include "clang/Basic/DiagnosticOptions.h"
 #include "llvm/Option/ArgList.h"
 #include <memory>
 
public:
   // TODO: We should map frontendOpts_ to parserOpts_ instead. For that, we
   // need to extend frontendOpts_ first. Next, we need to add the corresponding
   // compiler driver options in libclangDriver.
   void SetDefaultFortranOpts();
 
   /// Set the default predefinitions.
   void setDefaultPredefinitions();
 
+  /// Collect the macro definitions from preprocessorOpts_ and prepare them for
+  /// the parser (i.e. copy into parserOpts_)
+  void collectMacroDefinitions();
+
   /// Set the Fortran options to user-specified values.
   /// These values are found in the preprocessor options.
   void setFortranOpts();
 
   /// Set the Semantic Options
   void setSemanticsOpts(Fortran::parser::AllCookedSources &);
 };
 
 } // end namespace Fortran::frontend
 #endif // LLVM_FLANG_FRONTEND_COMPILERINVOCATION_H

flang/include/flang/Frontend/FrontendOptions.h

 //===- FrontendOptions.h ----------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 #ifndef LLVM_FLANG_FRONTEND_FRONTENDOPTIONS_H
 #define LLVM_FLANG_FRONTEND_FRONTENDOPTIONS_H
 
 #include "flang/Common/Fortran-features.h"
 #include "flang/Parser/characters.h"
 #include "flang/Parser/unparse.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/MemoryBuffer.h"
 
 #include <cstdint>
 #include <string>
 

 // TODO: Find a more suitable location for this. Added for compability with
 // f18.cpp (this is equivalent to `asFortran` defined there).
 Fortran::parser::AnalyzedObjectsAsFortran getBasicAsFortran();
 
 /// \param suffix The file extension
 /// \return True if the file extension should be processed as free form
 bool isFreeFormSuffix(llvm::StringRef suffix);
 
+/// \param suffix The file extension
+/// \return True if the file should be preprocessed
+bool mustBePreprocessed(llvm::StringRef suffix);
+
 enum class Language : uint8_t {
   Unknown,
 
   /// LLVM IR: we accept this so that we can run the optimizer on it,
   /// and compile it to assembly or object code.
   LLVM_IR,
 
   /// @{ Languages that the frontend can parse and compile.
class FrontendInputFile {
   /// The kind of input, atm it contains language
   InputKind kind_;
 
   /// Is this input file in fixed-form format? This is simply derived from the
   /// file extension and should not be altered by consumers. For input from
   /// stdin this is never modified.
   bool isFixedForm_ = false;
 
+  /// Must this file be preprocessed? Note that in Flang the preprocessor is
+  /// always run. This flag is used to control whether macro standard
+  /// predefinitions are added or not. In practice, this is sufficient to
+  /// implement gfortran`s logic controlled with `-cpp/-nocpp` flags.
+  unsigned mustBePreprocessed_ : 1;
+
 public:
   FrontendInputFile() = default;
   FrontendInputFile(llvm::StringRef file, InputKind kind)
       : file_(file.str()), kind_(kind) {
 
     // Based on the extension, decide whether this is a fixed or free form
     // file.
     auto pathDotIndex{file.rfind(".")};
     std::string pathSuffix{file.substr(pathDotIndex + 1)};
     isFixedForm_ = isFixedFormSuffix(pathSuffix);
+    mustBePreprocessed_ = mustBePreprocessed(pathSuffix);
   }
+
   FrontendInputFile(const llvm::MemoryBuffer *buffer, InputKind kind)
       : buffer_(buffer), kind_(kind) {}
 
   InputKind kind() const { return kind_; }
 
   bool IsEmpty() const { return file_.empty() && buffer_ == nullptr; }
   bool IsFile() const { return !IsBuffer(); }
   bool IsBuffer() const { return buffer_ != nullptr; }
   bool IsFixedForm() const { return isFixedForm_; }
+  bool MustBePreprocessed() const { return mustBePreprocessed_; }
 
   llvm::StringRef file() const {
     assert(IsFile());
     return file_;
   }
 
   const llvm::MemoryBuffer *buffer() const {
     assert(IsBuffer() && "Requested buffer_, but it is empty!");

flang/include/flang/Frontend/PreprocessorOptions.h

 
 #ifndef LLVM_FLANG_PREPROCESSOROPTIONS_H
 #define LLVM_FLANG_PREPROCESSOROPTIONS_H
 
 #include "llvm/ADT/StringRef.h"
 
 namespace Fortran::frontend {
 
+/// Communicates whether to include/exclude standard macro predefinitions
+enum class StdMacroPredefs : uint8_t {
+  /// Use the file extension to decide
+  Unknown,
+
+  Include,
+  Exclude
+};
+
 /// This class is used for passing the various options used
 /// in preprocessor initialization to the parser options.
 class PreprocessorOptions {
 public:
   std::vector<std::pair<std::string, /*isUndef*/ bool>> macros;
   // Search directories specified by the user with -I
   // TODO: When adding support for more options related to search paths,
   // consider collecting them in a separate aggregate. For now we keep it here
   // as there is no point creating a class for just one field.
   std::vector<std::string> searchDirectoriesFromDashI;
   // Search directories specified by the user with -fintrinsic-modules-path
   std::vector<std::string> searchDirectoriesFromIntrModPath;
 
+  StdMacroPredefs predefs_ = StdMacroPredefs::Unknown;
+
 public:
   PreprocessorOptions() {}
 
   void addMacroDef(llvm::StringRef name) {
     macros.emplace_back(std::string(name), false);
   }
 
   void addMacroUndef(llvm::StringRef name) {
     macros.emplace_back(std::string(name), true);
   }
 };
 
 } // namespace Fortran::frontend
 
 #endif // LLVM_FLANG_PREPROCESSOROPTIONS_H

flang/lib/Frontend/CompilerInstance.cpp

void CompilerInstance::ClearOutputFiles(bool eraseFiles) {
   nonSeekStream_.reset();
 }
 
 bool CompilerInstance::ExecuteAction(FrontendAction &act) {
   auto &invoc = this->invocation();
 
   // Set some sane defaults for the frontend.
   invoc.SetDefaultFortranOpts();
-  invoc.setDefaultPredefinitions();
   // Update the fortran options based on user-based input.
   invoc.setFortranOpts();
   // Set the encoding to read all input files in based on user input.
   allSources_->set_encoding(invoc.fortranOpts().encoding);
   // Create the semantics context and set semantic options.
   invoc.setSemanticsOpts(*this->allCookedSources_);
 
   // Run the frontend action `act` for every input file.

flang/lib/Frontend/CompilerInvocation.cpp

 //===- CompilerInvocation.cpp ---------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "flang/Frontend/CompilerInvocation.h"
 #include "flang/Common/Fortran-features.h"
 #include "flang/Frontend/PreprocessorOptions.h"
 #include "flang/Semantics/semantics.h"
 #include "flang/Version.inc"
 #include "clang/Basic/AllDiagnostics.h"
 #include "clang/Basic/DiagnosticDriver.h"
 #include "clang/Basic/DiagnosticOptions.h"
 #include "clang/Driver/DriverDiagnostic.h"
static void parsePreprocessorArgs(
   for (const auto *currentArg : args.filtered(clang::driver::options::OPT_I))
     opts.searchDirectoriesFromDashI.emplace_back(currentArg->getValue());
 
   // Prepend the ordered list of -intrinsic-modules-path
   // to the default location to search.
   for (const auto *currentArg :
       args.filtered(clang::driver::options::OPT_fintrinsic_modules_path))
     opts.searchDirectoriesFromIntrModPath.emplace_back(currentArg->getValue());
+
+  // -cpp/-nocpp
+  if (const auto *currentArg = args.getLastArg(
+          clang::driver::options::OPT_cpp, clang::driver::options::OPT_nocpp))
+    opts.predefs_ =
+        (currentArg->getOption().matches(clang::driver::options::OPT_cpp))
+        ? StdMacroPredefs::Include
+        : StdMacroPredefs::Exclude;
 }
 
 /// Parses all semantic related arguments and populates the variables
 /// options accordingly.
 static void parseSemaArgs(CompilerInvocation &res, llvm::opt::ArgList &args,
     clang::DiagnosticsEngine &diags) {
 
   // -J/module-dir option
bool CompilerInvocation::CreateFromArgs(CompilerInvocation &res,
   // Parse semantic args
   parseSemaArgs(res, args, diags);
   // Parse dialect arguments
   parseDialectArgs(res, args, diags);
 
   return success;
 }
 
-/// Collect the macro definitions provided by the given preprocessor
-/// options into the parser options.
-///
-/// \param [in] ppOpts The preprocessor options
-/// \param [out] opts The fortran options
-static void collectMacroDefinitions(
-    const PreprocessorOptions &ppOpts, Fortran::parser::Options &opts) {
+void CompilerInvocation::collectMacroDefinitions() {
+  auto &ppOpts = this->preprocessorOpts();
+
   for (unsigned i = 0, n = ppOpts.macros.size(); i != n; ++i) {
     llvm::StringRef macro = ppOpts.macros[i].first;
     bool isUndef = ppOpts.macros[i].second;
 
     std::pair<llvm::StringRef, llvm::StringRef> macroPair = macro.split('=');
     llvm::StringRef macroName = macroPair.first;
     llvm::StringRef macroBody = macroPair.second;
 
     // For an #undef'd macro, we only care about the name.
     if (isUndef) {
-      opts.predefinitions.emplace_back(
+      parserOpts_.predefinitions.emplace_back(
           macroName.str(), std::optional<std::string>{});
       continue;
     }
 
     // For a #define'd macro, figure out the actual definition.
     if (macroName.size() == macro.size())
       macroBody = "1";
     else {
       // Note: GCC drops anything following an end-of-line character.
       llvm::StringRef::size_type End = macroBody.find_first_of("\n\r");
       macroBody = macroBody.substr(0, End);
     }
-    opts.predefinitions.emplace_back(
+    parserOpts_.predefinitions.emplace_back(
         macroName, std::optional<std::string>(macroBody.str()));
   }
 }
 
 void CompilerInvocation::SetDefaultFortranOpts() {
   auto &fortranOptions = fortranOpts();
 
   // These defaults are based on the defaults in f18/f18.cpp.
if (frontendOptions.fortranForm_ != FortranForm::Unknown) {
     fortranOptions.isFixedForm =
         frontendOptions.fortranForm_ == FortranForm::FixedForm;
   }
   fortranOptions.fixedFormColumns = frontendOptions.fixedFormColumns_;
 
   fortranOptions.features = frontendOptions.features_;
   fortranOptions.encoding = frontendOptions.encoding_;
 
-  collectMacroDefinitions(preprocessorOptions, fortranOptions);
-
   // Adding search directories specified by -I
   fortranOptions.searchDirectories.insert(
       fortranOptions.searchDirectories.end(),
       preprocessorOptions.searchDirectoriesFromDashI.begin(),
       preprocessorOptions.searchDirectoriesFromDashI.end());
 
   // Add the ordered list of -intrinsic-modules-path
   fortranOptions.searchDirectories.insert(

flang/lib/Frontend/FrontendAction.cpp

 //===--- FrontendAction.cpp -----------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "flang/Frontend/FrontendAction.h"
 #include "flang/Frontend/CompilerInstance.h"
 #include "flang/Frontend/FrontendActions.h"
 #include "flang/Frontend/FrontendOptions.h"
 #include "flang/FrontendTool/Utils.h"
 #include "clang/Basic/DiagnosticFrontend.h"
 #include "llvm/Support/Errc.h"
 #include "llvm/Support/VirtualFileSystem.h"
 
bool FrontendAction::BeginSourceFile(
   set_currentInput(realInput);
   set_instance(&ci);
 
   if (!ci.HasAllSources()) {
     BeginSourceFileCleanUp(*this, ci);
     return false;
   }
 
+  auto &invoc = ci.invocation();
+
+  // Include command-line and predefined preprocessor macros. Use either:
+  //  * `-cpp/-nocpp`, or
+  //  * the file extension (if the user didn't express any preference)
+  // to decide whether to include them or not.
+  if ((invoc.preprocessorOpts().predefs_ == StdMacroPredefs::Include) ||
+      (invoc.preprocessorOpts().predefs_ == StdMacroPredefs::Unknown &&
+          currentInput().MustBePreprocessed())) {
+    invoc.setDefaultPredefinitions();
+    invoc.collectMacroDefinitions();
+  }
+
+  // Decide between fixed and free form (if the user didn't express any
+  // preference, use the file extension to decide)
+  if (invoc.frontendOpts().fortranForm_ == FortranForm::Unknown) {
+    invoc.fortranOpts().isFixedForm = currentInput().IsFixedForm();
+  }
+
   if (!BeginSourceFileAction(ci)) {
     BeginSourceFileCleanUp(*this, ci);
     return false;
   }
 
   return true;
 }
 

flang/lib/Frontend/FrontendActions.cpp

 //===--- FrontendActions.cpp ----------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "flang/Frontend/FrontendActions.h"
 #include "flang/Common/default-kinds.h"
 #include "flang/Frontend/CompilerInstance.h"
 #include "flang/Frontend/FrontendOptions.h"
+#include "flang/Frontend/PreprocessorOptions.h"
 #include "flang/Lower/PFTBuilder.h"
 #include "flang/Parser/dump-parse-tree.h"
 #include "flang/Parser/parsing.h"
 #include "flang/Parser/provenance.h"
 #include "flang/Parser/source.h"
 #include "flang/Parser/unparse.h"
 #include "flang/Semantics/runtime-type-info.h"
 #include "flang/Semantics/semantics.h"
if (semantics.AnyFatalError()) {
     diags.Report(DiagID) << bufferName;
     return true;
   }
   return false;
 }
 
 bool PrescanAction::BeginSourceFileAction(CompilerInstance &c1) {
   CompilerInstance &ci = this->instance();
-
   std::string currentInputPath{GetCurrentFileOrBufferName()};
-
   Fortran::parser::Options parserOptions = ci.invocation().fortranOpts();
 
-  if (ci.invocation().frontendOpts().fortranForm_ == FortranForm::Unknown) {
-    // Switch between fixed and free form format based on the input file
-    // extension.
-    //
-    // Ideally we should have all Fortran options set before entering this
-    // method (i.e. before processing any specific input files). However, we
-    // can't decide between fixed and free form based on the file extension
-    // earlier than this.
-    parserOptions.isFixedForm = currentInput().IsFixedForm();
-  }
 
   // Prescan. In case of failure, report and return.
   ci.parsing().Prescan(currentInputPath, parserOptions);
 
   if (ci.parsing().messages().AnyFatalError()) {
     const unsigned diagID = ci.diagnostics().getCustomDiagID(
         clang::DiagnosticsEngine::Error, "Could not scan %0");
     ci.diagnostics().Report(diagID) << GetCurrentFileOrBufferName();
     ci.parsing().messages().Emit(llvm::errs(), ci.allCookedSources());
 
     return false;
   }
 
   return true;
 }
 
 bool PrescanAndSemaAction::BeginSourceFileAction(CompilerInstance &c1) {
   CompilerInstance &ci = this->instance();
-
   std::string currentInputPath{GetCurrentFileOrBufferName()};
-
   Fortran::parser::Options parserOptions = ci.invocation().fortranOpts();
 
-  if (ci.invocation().frontendOpts().fortranForm_ == FortranForm::Unknown) {
-    // Switch between fixed and free form format based on the input file
-    // extension.
-    //
-    // Ideally we should have all Fortran options set before entering this
-    // method (i.e. before processing any specific input files). However, we
-    // can't decide between fixed and free form based on the file extension
-    // earlier than this.
-    parserOptions.isFixedForm = currentInput().IsFixedForm();
-  }
-
   // Prescan. In case of failure, report and return.
   ci.parsing().Prescan(currentInputPath, parserOptions);
 
   if (ci.parsing().messages().AnyFatalError()) {
     const unsigned diagID = ci.diagnostics().getCustomDiagID(
         clang::DiagnosticsEngine::Error, "Could not scan %0");
     ci.diagnostics().Report(diagID) << GetCurrentFileOrBufferName();
     ci.parsing().messages().Emit(llvm::errs(), ci.allCookedSources());

flang/lib/Frontend/FrontendOptions.cpp

 //===- FrontendOptions.cpp ------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "flang/Frontend/FrontendOptions.h"
 #include "flang/Evaluate/expression.h"
 
 using namespace Fortran::frontend;
 
 bool Fortran::frontend::isFixedFormSuffix(llvm::StringRef suffix) {
   // Note: Keep this list in-sync with flang/test/lit.cfg.py
   return suffix == "f" || suffix == "F" || suffix == "ff" || suffix == "for" ||
       suffix == "FOR" || suffix == "fpp" || suffix == "FPP";
 }
 
 bool Fortran::frontend::isFreeFormSuffix(llvm::StringRef suffix) {
   // Note: Keep this list in-sync with flang/test/lit.cfg.py
   // TODO: Add Cuda Fortan files (i.e. `*.cuf` and `*.CUF`).
   return suffix == "f77" || suffix == "f90" || suffix == "F90" ||
       suffix == "ff90" || suffix == "f95" || suffix == "F95" ||
       suffix == "ff95" || suffix == "f03" || suffix == "F03" ||
       suffix == "f08" || suffix == "F08" || suffix == "f18" || suffix == "F18";
 }
 
+bool Fortran::frontend::mustBePreprocessed(llvm::StringRef suffix) {
+  return suffix == "F" || suffix == "FOR" || suffix == "fpp" ||
+      suffix == "FPP" || suffix == "F90" || suffix == "F95" ||
+      suffix == "F03" || suffix == "F08" || suffix == "F18";
+}
+
 // TODO: This is a copy of `asFortran` from f18.cpp and is added here for
 // compatiblity. It doesn't really belong here, but I couldn't find a better
 // place. We should decide whether to add it to the Evaluate or Parse/Unparse
 // APIs or some dedicated utility library in the driver.
 Fortran::parser::AnalyzedObjectsAsFortran
 Fortran::frontend::getBasicAsFortran() {
   return Fortran::parser::AnalyzedObjectsAsFortran{
       [](llvm::raw_ostream &o, const Fortran::evaluate::GenericExprWrapper &x) {

flang/test/Driver/cpp-nocpp-command-line-macro.f90

+!-----------
+! RUN lines
+!-----------
+! RUN: %flang_fc1 -E %s 2>&1 | FileCheck %s --check-prefix=UNDEFINED
+! RUN: %flang_fc1 -E -cpp -DX=A %s 2>&1 | FileCheck %s --check-prefix=DEFINED
+! RUN: %flang_fc1 -E -nocpp -DX=A %s 2>&1 | FileCheck %s --check-prefix=UNDEFINED
+
+!-----------------
+! EXPECTED OUTPUT 
+!-----------------
+! UNDEFINED:program b
+! UNDEFINED-NOT:program a
+
+! DEFINED:program a
+! DEFINED-NOT:program b
+
+#ifdef X
+program X
+#else
+program B
+#endif
+end

flang/test/Driver/cpp-nocpp-predefined-macro.F90

+!-----------
+! RUN lines
+!-----------
+! RUN: %flang_fc1 -E %s 2>&1 | FileCheck %s --check-prefix=DEFINED
+! RUN: %flang_fc1 -E -cpp %s 2>&1 | FileCheck %s --check-prefix=DEFINED
+! RUN: %flang_fc1 -E -nocpp %s 2>&1 | FileCheck %s --check-prefix=NOT_DEFINED
+
+!-----------------
+! EXPECTED OUTPUT
+!-----------------
+! DEFINED: flang = 1
+! DEFINED-NEXT: flang_major = {{[1-9][0-9]*$}}
+
+! NOT_DEFINED: flang = __flang__
+! NOT_DEFINED-NEXT: flang_major = __flang_major__
+
+integer, parameter :: flang = __flang__
+integer, parameter :: flang_major = __flang_major__

flang/test/Driver/driver-help-hidden.f90

 
 !----------------------------------------------------
 ! EXPECTED OUTPUT FOR FLANG DRIVER (flang-new)
 !----------------------------------------------------
 ! CHECK:USAGE: flang-new
 ! CHECK-EMPTY:
 ! CHECK-NEXT:OPTIONS:
 ! CHECK-NEXT: -###      Print (but do not run) the commands to run for this compilation
+! CHECK-NEXT: -cpp      Enable predefined and command line preprocessor macros
 ! CHECK-NEXT: -c        Only run preprocess, compile, and assemble steps
 ! CHECK-NEXT: -D <macro>=<value>     Define <macro> to <value> (or 1 if <value> omitted)
 ! CHECK-NEXT: -E        Only run the preprocessor
 ! CHECK-NEXT: -falternative-parameter-statement
 ! CHECK-NEXT: Enable the old style PARAMETER statement
 ! CHECK-NEXT: -fbackslash            Specify that backslash in string introduces an escape character
 ! CHECK-NEXT: -fcolor-diagnostics    Enable colors in diagnostics
 ! CHECK-NEXT: -fdefault-double-8     Set the default double precision kind to an 8 byte wide type

 ! CHECK-NEXT: -flogical-abbreviations Enable logical abbreviations
 ! CHECK-NEXT: -fno-color-diagnostics Disable colors in diagnostics
 ! CHECK-NEXT: -fopenacc              Enable OpenACC
 ! CHECK-NEXT: -fopenmp               Parse OpenMP pragmas and generate parallel code.
 ! CHECK-NEXT: -fxor-operator         Enable .XOR. as a synonym of .NEQV.
 ! CHECK-NEXT: -help     Display available options
 ! CHECK-NEXT: -I <dir>               Add directory to the end of the list of include search paths
 ! CHECK-NEXT: -module-dir <dir>      Put MODULE files in <dir>
+! CHECK-NEXT: -nocpp                 Disable predefined and command line preprocessor macros
 ! CHECK-NEXT: -o <file> Write output to <file>
 ! CHECK-NEXT: -pedantic              Warn on language extensions
 ! CHECK-NEXT: -std=<value>           Language standard to compile for
 ! CHECK-NEXT: -U <macro>             Undefine macro <macro>
 ! CHECK-NEXT: --version Print version information
 ! CHECK-NEXT: -Xflang <arg>          Pass <arg> to the flang compiler
 
 !-------------------------------------------------------------

flang/test/Driver/driver-help.f90

 
 !----------------------------------------------------
 ! EXPECTED OUTPUT FOR FLANG DRIVER (flang-new)
 !----------------------------------------------------
 ! HELP:USAGE: flang-new
 ! HELP-EMPTY:
 ! HELP-NEXT:OPTIONS:
 ! HELP-NEXT: -###                   Print (but do not run) the commands to run for this compilation
+! HELP-NEXT: -cpp                   Enable predefined and command line preprocessor macros
 ! HELP-NEXT: -c                     Only run preprocess, compile, and assemble steps
 ! HELP-NEXT: -D <macro>=<value>     Define <macro> to <value> (or 1 if <value> omitted)
 ! HELP-NEXT: -E                     Only run the preprocessor
 ! HELP-NEXT: -falternative-parameter-statement
 ! HELP-NEXT: Enable the old style PARAMETER statement
 ! HELP-NEXT: -fbackslash            Specify that backslash in string introduces an escape character
 ! HELP-NEXT: -fcolor-diagnostics    Enable colors in diagnostics
 ! HELP-NEXT: -fdefault-double-8     Set the default double precision kind to an 8 byte wide type

 ! HELP-NEXT: -flogical-abbreviations Enable logical abbreviations
 ! HELP-NEXT: -fno-color-diagnostics Disable colors in diagnostics
 ! HELP-NEXT: -fopenacc              Enable OpenACC
 ! HELP-NEXT: -fopenmp               Parse OpenMP pragmas and generate parallel code.
 ! HELP-NEXT: -fxor-operator         Enable .XOR. as a synonym of .NEQV.
 ! HELP-NEXT: -help                  Display available options
 ! HELP-NEXT: -I <dir>               Add directory to the end of the list of include search paths
 ! HELP-NEXT: -module-dir <dir>      Put MODULE files in <dir>
+! HELP-NEXT: -nocpp                 Disable predefined and command line preprocessor macros
 ! HELP-NEXT: -o <file>              Write output to <file>
 ! HELP-NEXT: -pedantic              Warn on language extensions
 ! HELP-NEXT: -std=<value>           Language standard to compile for
 ! HELP-NEXT: -U <macro>             Undefine macro <macro>
 ! HELP-NEXT: --version              Print version information
 ! HELP-NEXT: -Xflang <arg>          Pass <arg> to the flang compiler
 
 !-------------------------------------------------------------
 ! EXPECTED OUTPUT FOR FLANG FRONTEND DRIVER (flang-new -fc1)
 !-------------------------------------------------------------
 ! HELP-FC1:USAGE: flang-new
 ! HELP-FC1-EMPTY:
 ! HELP-FC1-NEXT:OPTIONS:
+! HELP-FC1-NEXT: -cpp                   Enable predefined and command line preprocessor macros
 ! HELP-FC1-NEXT: -D <macro>=<value>     Define <macro> to <value> (or 1 if <value> omitted)
 ! HELP-FC1-NEXT: -emit-obj Emit native object files
 ! HELP-FC1-NEXT: -E                     Only run the preprocessor
 ! HELP-FC1-NEXT: -falternative-parameter-statement
 ! HELP-FC1-NEXT: Enable the old style PARAMETER statement
 ! HELP-FC1-NEXT: -fbackslash            Specify that backslash in string introduces an escape character
 ! HELP-FC1-NEXT: -fdebug-dump-parse-tree Dump the parse tree
 ! HELP-FC1-NEXT: -fdebug-dump-parsing-log

 ! HELP-FC1-NEXT: -flarge-sizes          Use INTEGER(KIND=8) for the result type in size-related intrinsics
 ! HELP-FC1-NEXT: -flogical-abbreviations Enable logical abbreviations
 ! HELP-FC1-NEXT: -fopenacc              Enable OpenACC
 ! HELP-FC1-NEXT: -fopenmp               Parse OpenMP pragmas and generate parallel code.
 ! HELP-FC1-NEXT: -fxor-operator         Enable .XOR. as a synonym of .NEQV.
 ! HELP-FC1-NEXT: -help                  Display available options
 ! HELP-FC1-NEXT: -I <dir>               Add directory to the end of the list of include search paths
 ! HELP-FC1-NEXT: -module-dir <dir>      Put MODULE files in <dir>
+! HELP-FC1-NEXT: -nocpp                 Disable predefined and command line preprocessor macros
 ! HELP-FC1-NEXT: -o <file>              Write output to <file>
 ! HELP-FC1-NEXT: -pedantic              Warn on language extensions
 ! HELP-FC1-NEXT: -std=<value>           Language standard to compile for
 ! HELP-FC1-NEXT: -test-io               Run the InputOuputTest action. Use for development and testing only.
 ! HELP-FC1-NEXT: -U <macro>             Undefine macro <macro>
 ! HELP-FC1-NEXT: --version              Print version information
 
 !---------------
 ! EXPECTED ERROR
 !---------------
 ! ERROR: error: unknown argument '-helps'; did you mean '-help'

flang/test/Driver/input-from-stdin.f90

 ! Verify that reading from stdin works as expected
 
 ! REQUIRES: new-flang-driver
 
 !--------------------------
 ! FLANG DRIVER (flang)
 !--------------------------
 ! Input type is implicit
-! RUN: cat %s | %flang -E - | FileCheck %s --check-prefix=PP-NOT-DEFINED
-! RUN: cat %s | %flang -DNEW -E - | FileCheck %s --check-prefix=PP-DEFINED
+! RUN: cat %s | %flang -E -cpp - | FileCheck %s --check-prefix=PP-NOT-DEFINED
+! RUN: cat %s | %flang -DNEW -E -cpp - | FileCheck %s --check-prefix=PP-DEFINED
+! RUN: cat %s | %flang -DNEW -E - | FileCheck %s --check-prefix=PP-NOT-DEFINED
+! RUN: cat %s | %flang -DNEW -E -nocpp - | FileCheck %s --check-prefix=PP-NOT-DEFINED
 
 ! Input type is explicit
-! RUN: cat %s | %flang -E -x f95-cpp-input - | FileCheck %s --check-prefix=PP-NOT-DEFINED
-! RUN: cat %s | %flang -DNEW -E -x f95-cpp-input - | FileCheck %s --check-prefix=PP-DEFINED
+! RUN: cat %s | %flang -E -cpp -x f95-cpp-input - | FileCheck %s --check-prefix=PP-NOT-DEFINED
+! RUN: cat %s | %flang -DNEW -E -cpp -x f95-cpp-input - | FileCheck %s --check-prefix=PP-DEFINED
 
 !---------------------------------------
 ! FLANG FRONTEND DRIVER (flang -fc1)
 !---------------------------------------
 ! Test `-E`: for the corresponding frontend actions the driver relies on the prescanner API to handle file I/O
-! RUN: cat %s | %flang -fc1 -E | FileCheck %s --check-prefix=PP-NOT-DEFINED
-! RUN: cat %s | %flang -fc1 -DNEW -E | FileCheck %s --check-prefix=PP-DEFINED
+! RUN: cat %s | %flang -fc1 -E -cpp | FileCheck %s --check-prefix=PP-NOT-DEFINED
+! RUN: cat %s | %flang -fc1 -DNEW -E -cpp | FileCheck %s --check-prefix=PP-DEFINED
 
 ! Test `-test-io`: for the corresponding frontend action (`InputOutputTestAction`) the driver handles the file I/O on its own
 ! the corresponding action (`PrintPreprocessedAction`)
-! RUN: cat %s | %flang -fc1 -test-io | FileCheck %s --check-prefix=IO --match-full-lines
-! RUN: cat %s | %flang -fc1 -DNEW -test-io | FileCheck %s --check-prefix=IO --match-full-lines
+! RUN: cat %s | %flang -fc1 -test-io -cpp | FileCheck %s --check-prefix=IO --match-full-lines
+! RUN: cat %s | %flang -fc1 -DNEW -cpp -test-io | FileCheck %s --check-prefix=IO --match-full-lines
 
 !-------------------------
 ! EXPECTED OUTPUT for `-E`
 !-------------------------
 ! PP-NOT-DEFINED: program b
 ! PP-DEFINED: program a
 
 !-------------------------------

flang/test/Driver/macro-def-undef.F90

 ! Ensure arguments -D and -U work as expected.
 
-! REQUIRES: new-flang-driver
-
 !--------------------------
 ! FLANG DRIVER (flang-new)
 !--------------------------
-! RUN: %flang-new -E %s  2>&1 | FileCheck %s --check-prefix=UNDEFINED
-! RUN: %flang-new -E -DX=A %s  2>&1 | FileCheck %s --check-prefix=DEFINED
-! RUN: %flang-new -E -DX=A -UX %s  2>&1 | FileCheck %s --check-prefix=UNDEFINED
+! RUN: %flang -E %s  2>&1 | FileCheck %s --check-prefix=UNDEFINED
+! RUN: %flang -E -DX=A %s  2>&1 | FileCheck %s --check-prefix=DEFINED
+! RUN: %flang -E -DX=A -UX %s  2>&1 | FileCheck %s --check-prefix=UNDEFINED
 
 !-----------------------------------------
 ! FRONTEND FLANG DRIVER (flang-new -fc1)
 !-----------------------------------------
-! RUN: %flang-new -fc1 -E %s  2>&1 | FileCheck %s --check-prefix=UNDEFINED
-! RUN: %flang-new -fc1 -E -DX=A %s  2>&1 | FileCheck %s --check-prefix=DEFINED
-! RUN: %flang-new -fc1 -E -DX -UX %s  2>&1 | FileCheck %s --check-prefix=UNDEFINED
+! RUN: %flang_fc1 -E %s  2>&1 | FileCheck %s --check-prefix=UNDEFINED
+! RUN: %flang_fc1 -E -DX=A %s  2>&1 | FileCheck %s --check-prefix=DEFINED
+! RUN: %flang_fc1 -E -DX -UX %s  2>&1 | FileCheck %s --check-prefix=UNDEFINED
 
 !--------------------------------------------
 ! EXPECTED OUTPUT FOR AN UNDEFINED MACRO
 !--------------------------------------------
 ! UNDEFINED:program b
 ! UNDEFINED-NOT:program x
 ! UNDEFINED-NEXT:end
 
 !--------------------------------------------
 ! EXPECTED OUTPUT FOR MACRO 'X' DEFINED AS A
 !--------------------------------------------
 ! DEFINED:program a
 ! DEFINED-NOT:program b
 ! DEFINED-NEXT:end
 
 #ifdef X
 program X
 #else
 program B
 #endif
-end
+end
 

flang/test/Driver/macro-multiline.F90

 ! Ensure the end-of-line character and anything that follows after in a macro definition (-D) is ignored.
 
 ! REQUIRES: new-flang-driver
 
 !--------------------------
 ! FLANG DRIVER (flang-new)
 !--------------------------
 ! RUN: printf -- "-DX=A\\\\\nTHIS_SHOULD_NOT_EXIST_IN_THE_OUTPUT\n" | xargs %flang-new -E %s  2>&1 | FileCheck --strict-whitespace --match-full-lines %s
 
 !-----------------------------------------
 ! FRONTEND FLANG DRIVER (flang-new -fc1)
 !-----------------------------------------
 ! RUN: printf -- "-DX=A\\\\\nTHIS_SHOULD_NOT_EXIST_IN_THE_OUTPUT\n" | xargs %flang-new -fc1 -E %s  2>&1 | FileCheck --strict-whitespace --match-full-lines %s
 
 !-------------------------------
 ! EXPECTED OUTPUT FOR MACRO 'X'
 !-------------------------------
 ! CHECK:start a end
 ! CHECK-NOT:THIS_SHOULD_NOT_EXIST_IN_THE_OUTPUT
 ! CHECK-NOT:this_should_not_exist_in_the_output
 
 START X END
 No newline at end of file

flang/test/Driver/predefined-macros-compiler-version.F90

 ! Check that the driver correctly defines macros with the compiler version
 
 ! REQUIRES: new-flang-driver
 
 !--------------------------
 ! FLANG DRIVER (flang-new)
 !--------------------------
 ! RUN: %flang-new -E %s  2>&1 | FileCheck %s --ignore-case
 
 !-----------------------------------------
 ! FRONTEND FLANG DRIVER (flang-new -fc1)
 !-----------------------------------------
 ! RUN: %flang-new -fc1 -E %s  2>&1 | FileCheck %s --ignore-case
 
 !-----------------
 ! EXPECTED OUTPUT
 !-----------------
 ! CHECK: flang = 1
 ! CHECK: flang_major = {{[1-9][0-9]*$}}
 ! CHECK: flang_minor = {{[0-9]+$}}
 ! CHECK: flang_patchlevel = {{[0-9]+$}}
 
 integer, parameter :: flang = __flang__
 integer, parameter :: flang_major = __flang_major__
 integer, parameter :: flang_minor = __flang_minor__
 integer, parameter :: flang_patchlevel = __flang_patchlevel__