[flang][driver] Add debug options not requiring semantic checks

[awarzynski]

Summary

This patch adds two debugging options in the new Flang driver (flang-new):

• -fdebug-unparse-no-sema
• -fdebug-dump-parse-tree-no-sema

Each of these options combines two options from the "throwaway" driver (left: f18, right: flang-new):

• -fdebug-uparse -fdebug-no-semantics --> -fdebug-unparse-no-sema
• -fdebug-dump-parse-tree -fdebug-no-semantics --> -fdebug-dump-parse-tree-no-sema

There are no plans to implement -fdebug-no-semantics in the new driver. Such option would be too powerful. Also, it would only make sense when combined with specific frontend actions (-fdebug-unparse and -fdebug-dump-parse-tree). Instead, this patch adds 2 options that are more specialised and 2 dedicated FrontendActions.

The new frontend actions are implemented in terms of a new abstract base action: PrescanAndSemaAction. This new base class was required so that we can have finer control over what steps within the frontend are executed:

• PrescanAction: run the _prescanner_
• PrescanAndSemaAction: run the _prescanner_ and the _parser_ (new in this patch)
• PrescanAndSemaAction: run the _prescanner_, _parser_ and run the _semantic checks_

This patch also introduces PrescanAndParseAction::BeginSourceFileAction. Apart from the semantic checks removed at the end, it is similar to PrescanAndSemaAction::BeginSourceFileAction.

[arnamoy10]

flang/test/Parser/omp-allocate-unparse.f90

1

We probably need to add an alias to f18 to make the two tests succeed.

[awarzynski]

Update f18.cpp and driver-help.f90 to improve testing

• added -fdebug-unparse-no-sema and -fdebug-dump-parse-tree-no-sema in f18.cpp (for compatibility with flang-new)
• updated driver-help.f90 with the new options

[ashermancinelli]

flang/lib/Frontend/FrontendActions.cpp

116

Is this variable unused?

[awarzynski]

flang/test/Parser/omp-allocate-unparse.f90

1

Updated, thank you!

[awarzynski]

Remove unused variable

flang/lib/Frontend/FrontendActions.cpp

116

How did it get here? :) Thank you, updated!

[kiranchandramohan]

clang/include/clang/Driver/Options.td

4380

Does this flag actually mean, parse and construct the parse-tree and unparse and stop?

While the current name is OK now, when the rest of the pipeline (codegen, linking etc) is there will it be an issue?

[awarzynski]

clang/include/clang/Driver/Options.td

4380

Does this flag actually mean, parse and construct the parse-tree and unparse and stop?

Yes

While the current name is OK now, when the rest of the pipeline (codegen, linking etc) is there will it be an issue?

IIUC, there shouldn't be any. Currently, the -fdebug-unparse options have dedicated frontend actions

• -fdebug-unparse runs DebugUnparseAction
• -fdebug-unparse-no-sema will run DebugUnparseNoSemaAction (introduced here)

I don't see why we'd change these actions once codegen or linking are available. That's something that will be implemented separately.

Does this help?

[awarzynski]

Add DocBrief descriptions in Options.td

I added the descriptions so that it's documented what the options are intended for and to avoid confusion in the future.

[awarzynski]

clang/include/clang/Driver/Options.td

4380

I added DocBrief descriptions to clarify the purpose of these options. Please note that currently these are not used anywhere. In Clang DocBrief descriptions are used to generate https://clang.llvm.org/docs/ClangCommandLineReference.html. We could set-up something similar for Flang.

[kiranchandramohan]

LGTM.

Might be good to have a test for fdebug-dump-parse-tree-no-sema.

flang/include/flang/Frontend/FrontendOptions.h

54

Nit: semantics are not run here, i guess.

[awarzynski]

In D99645#2674496, @kiranchandramohan wrote:

LGTM.

Might be good to have a test for fdebug-dump-parse-tree-no-sema.

Good point, added before merging into main! Thank you for reviewing.

[• walli99]

انت

clang/include/clang/Driver/Options.td

 
 //===----------------------------------------------------------------------===//
 // FC1 Options
 //===----------------------------------------------------------------------===//
 let Flags = [FC1Option, FlangOnlyOption] in {
 
 def test_io : Flag<["-"], "test-io">, Group<Action_Group>,
   HelpText<"Run the InputOuputTest action. Use for development and testing only.">;
+def fdebug_unparse_no_sema : Flag<["-"], "fdebug-unparse-no-sema">, Group<Action_Group>,
+  HelpText<"Unparse and stop (skips the semantic checks)">,
+  DocBrief<[{Only run the parser, then unparse the parse-tree and output the
+generated Fortran source file. Semantic checks are disabled.}]>;
 def fdebug_unparse : Flag<["-"], "fdebug-unparse">, Group<Action_Group>,
-  HelpText<"Unparse and stop.">;
+  HelpText<"Unparse and stop.">,
+  DocBrief<[{Run the parser and the semantic checks. Then unparse the
+parse-tree and output the generated Fortran source file.}]>;
 def fdebug_unparse_with_symbols : Flag<["-"], "fdebug-unparse-with-symbols">, Group<Action_Group>,
   HelpText<"Unparse and stop.">;
 def fdebug_dump_symbols : Flag<["-"], "fdebug-dump-symbols">, Group<Action_Group>,
   HelpText<"Dump symbols after the semantic analysis">;
 def fdebug_dump_parse_tree : Flag<["-"], "fdebug-dump-parse-tree">, Group<Action_Group>,
-  HelpText<"Dump the parse tree">;
+  HelpText<"Dump the parse tree">,
+  DocBrief<[{Run the Parser and the semantic checks, and then output the
+parse tree.}]>;
+def fdebug_dump_parse_tree_no_sema : Flag<["-"], "fdebug-dump-parse-tree-no-sema">, Group<Action_Group>,
+  HelpText<"Dump the parse tree (skips the semantic checks)">,
+  DocBrief<[{Run the Parser and then output the parse tree. Semantic
+checks are disabled.}]>;
 def fdebug_dump_provenance : Flag<["-"], "fdebug-dump-provenance">, Group<Action_Group>,
   HelpText<"Dump provenance">;
 def fdebug_dump_parsing_log : Flag<["-"], "fdebug-dump-parsing-log">, Group<Action_Group>,
   HelpText<"Run instrumented parse and dump the parsing log">;
 def fdebug_measure_parse_tree : Flag<["-"], "fdebug-measure-parse-tree">, Group<Action_Group>,
   HelpText<"Measure the parse tree">;
 def fdebug_pre_fir_tree : Flag<["-"], "fdebug-pre-fir-tree">, Group<Action_Group>,
   HelpText<"Dump the pre-FIR tree">;

flang/include/flang/Frontend/FrontendActions.h

class DebugDumpParsingLogAction : public PrescanAction {
   void ExecuteAction() override;
 };
 
 class DebugMeasureParseTreeAction : public PrescanAction {
   void ExecuteAction() override;
 };
 
 //===----------------------------------------------------------------------===//
+// PrescanAndParse Actions
+//===----------------------------------------------------------------------===//
+class PrescanAndParseAction : public FrontendAction {
+  void ExecuteAction() override = 0;
+  bool BeginSourceFileAction(CompilerInstance &ci) override;
+};
+
+class DebugUnparseNoSemaAction : public PrescanAndParseAction {
+  void ExecuteAction() override;
+};
+
+class DebugDumpParseTreeNoSemaAction : public PrescanAndParseAction {
+  void ExecuteAction() override;
+};
+
+//===----------------------------------------------------------------------===//
 // PrescanAndSema Actions
 //===----------------------------------------------------------------------===//
 class PrescanAndSemaAction : public FrontendAction {
   std::unique_ptr<Fortran::semantics::Semantics> semantics_;
 
   void ExecuteAction() override = 0;
   bool BeginSourceFileAction(CompilerInstance &ci) override;
 

flang/include/flang/Frontend/FrontendOptions.h

enum ActionKind {
   ParseSyntaxOnly,
 
   /// Emit a .o file.
   EmitObj,
 
   /// Parse, unparse the parse-tree and output a Fortran source file
   DebugUnparse,
 
+  /// Parse, unparse the parse-tree and output a Fortran source file, skip the
+  /// semantic checks
+  DebugUnparseNoSema,
+
   /// Parse, resolve the sybmols, unparse the parse-tree and then output a
   /// Fortran source file
   DebugUnparseWithSymbols,
 
   /// Parse, run semantics and then output symbols from semantics
   DebugDumpSymbols,
 
   /// Parse, run semantics and then output the parse tree
   DebugDumpParseTree,
 
+  /// Parse and then output the parse tree, skip the semantic checks
+  DebugDumpParseTreeNoSema,
+
   /// Dump provenance
   DebugDumpProvenance,
 
   /// Parse then output the parsing log
   DebugDumpParsingLog,
 
   /// Parse then output the number of objects in the parse tree and the overall
   /// size

flang/lib/Frontend/CompilerInvocation.cpp

case clang::driver::options::OPT_fsyntax_only:
       opts.programAction_ = ParseSyntaxOnly;
       break;
     case clang::driver::options::OPT_emit_obj:
       opts.programAction_ = EmitObj;
       break;
     case clang::driver::options::OPT_fdebug_unparse:
       opts.programAction_ = DebugUnparse;
       break;
+    case clang::driver::options::OPT_fdebug_unparse_no_sema:
+      opts.programAction_ = DebugUnparseNoSema;
+      break;
     case clang::driver::options::OPT_fdebug_unparse_with_symbols:
       opts.programAction_ = DebugUnparseWithSymbols;
       break;
     case clang::driver::options::OPT_fdebug_dump_symbols:
       opts.programAction_ = DebugDumpSymbols;
       break;
     case clang::driver::options::OPT_fdebug_dump_parse_tree:
       opts.programAction_ = DebugDumpParseTree;
       break;
+    case clang::driver::options::OPT_fdebug_dump_parse_tree_no_sema:
+      opts.programAction_ = DebugDumpParseTreeNoSema;
+      break;
     case clang::driver::options::OPT_fdebug_dump_provenance:
       opts.programAction_ = DebugDumpProvenance;
       break;
     case clang::driver::options::OPT_fdebug_dump_parsing_log:
       opts.programAction_ = DebugDumpParsingLog;
       break;
     case clang::driver::options::OPT_fdebug_measure_parse_tree:
       opts.programAction_ = DebugMeasureParseTree;

flang/lib/Frontend/FrontendActions.cpp

if (!ci.parsing().messages().empty() &&
     ci.parsing().messages().Emit(llvm::errs(), ci.allCookedSources());
 
     return false;
   }
 
   return true;
 }
 
+bool PrescanAndParseAction::BeginSourceFileAction(CompilerInstance &c1) {
+  CompilerInstance &ci = this->instance();
+
+  std::string currentInputPath{GetCurrentFileOrBufferName()};
+
+  Fortran::parser::Options parserOptions = ci.invocation().fortranOpts();
+
+  if (ci.invocation().frontendOpts().fortranForm_ == FortranForm::Unknown) {
+    // Switch between fixed and free form format based on the input file
+    // extension.
+    //
+    // Ideally we should have all Fortran options set before entering this
+    // method (i.e. before processing any specific input files). However, we
+    // can't decide between fixed and free form based on the file extension
+    // earlier than this.
+    parserOptions.isFixedForm = currentInput().IsFixedForm();
+  }
+
+  // Prescan. In case of failure, report and return.
+  ci.parsing().Prescan(currentInputPath, parserOptions);
+
+  if (ci.parsing().messages().AnyFatalError()) {
+    const unsigned diagID = ci.diagnostics().getCustomDiagID(
+        clang::DiagnosticsEngine::Error, "Could not scan %0");
+    ci.diagnostics().Report(diagID) << GetCurrentFileOrBufferName();
+    ci.parsing().messages().Emit(llvm::errs(), ci.allCookedSources());
+
+    return false;
+  }
+
+  // Parse. In case of failure, report and return.
+  ci.parsing().Parse(llvm::outs());
+
+  if (ci.parsing().messages().AnyFatalError()) {
+    unsigned diagID = ci.diagnostics().getCustomDiagID(
+        clang::DiagnosticsEngine::Error, "Could not parse %0");
+    ci.diagnostics().Report(diagID) << GetCurrentFileOrBufferName();
+
+    ci.parsing().messages().Emit(
+        llvm::errs(), this->instance().allCookedSources());
+    return false;
+  }
+
+  // Report the diagnostics from parsing
+  ci.parsing().messages().Emit(llvm::errs(), ci.allCookedSources());
+
+  return true;
+}
+
 bool PrescanAndSemaAction::BeginSourceFileAction(CompilerInstance &c1) {
   CompilerInstance &ci = this->instance();
   std::string currentInputPath{GetCurrentFileOrBufferName()};
   Fortran::parser::Options parserOptions = ci.invocation().fortranOpts();
 
   // Prescan. In case of failure, report and return.
   ci.parsing().Prescan(currentInputPath, parserOptions);
 
setSemantics(std::make_unique<Fortran::semantics::Semantics>(
       ci.parsing().cooked().AsCharBlock(), ci.invocation().debugModuleDir()));
   auto &semantics = this->semantics();
 
   // Run semantic checks
   semantics.Perform();
 
   // Report the diagnostics from the semantic checks
   semantics.EmitMessages(ci.semaOutputStream());
+
   return true;
 }
 
 void InputOutputTestAction::ExecuteAction() {
   CompilerInstance &ci = instance();
 
   // Create a stream for errors
   std::string buf;
void DebugDumpProvenanceAction::ExecuteAction() {
   this->instance().parsing().DumpProvenance(llvm::outs());
 }
 
 void ParseSyntaxOnlyAction::ExecuteAction() {
   reportFatalSemanticErrors(semantics(), this->instance().diagnostics(),
       GetCurrentFileOrBufferName());
 }
 
+void DebugUnparseNoSemaAction::ExecuteAction() {
+  auto &parseTree{instance().parsing().parseTree()};
+
+  Fortran::parser::AnalyzedObjectsAsFortran asFortran =
+      Fortran::frontend::getBasicAsFortran();
+
+  // TODO: Options should come from CompilerInvocation
+  Unparse(llvm::outs(), *parseTree,
+      /*encoding=*/Fortran::parser::Encoding::UTF_8,
+      /*capitalizeKeywords=*/true, /*backslashEscapes=*/false,
+      /*preStatement=*/nullptr, &asFortran);
+}
+
 void DebugUnparseAction::ExecuteAction() {
   auto &parseTree{instance().parsing().parseTree()};
   Fortran::parser::AnalyzedObjectsAsFortran asFortran =
       Fortran::frontend::getBasicAsFortran();
 
   // TODO: Options should come from CompilerInvocation
   Unparse(llvm::outs(), *parseTree,
       /*encoding=*/Fortran::parser::Encoding::UTF_8,
void DebugDumpSymbolsAction::ExecuteAction() {
 
   // Dump symbols
   semantics.DumpSymbols(llvm::outs());
   // Report fatal semantic errors
   reportFatalSemanticErrors(
       semantics, this->instance().diagnostics(), GetCurrentFileOrBufferName());
 }
 
+void DebugDumpParseTreeNoSemaAction::ExecuteAction() {
+  auto &parseTree{instance().parsing().parseTree()};
+  Fortran::parser::AnalyzedObjectsAsFortran asFortran =
+      Fortran::frontend::getBasicAsFortran();
+
+  // Dump parse tree
+  Fortran::parser::DumpTree(llvm::outs(), parseTree, &asFortran);
+}
+
 void DebugDumpParseTreeAction::ExecuteAction() {
   auto &parseTree{instance().parsing().parseTree()};
   Fortran::parser::AnalyzedObjectsAsFortran asFortran =
       Fortran::frontend::getBasicAsFortran();
 
   // Dump parse tree
   Fortran::parser::DumpTree(llvm::outs(), parseTree, &asFortran);
   // Report fatal semantic errors

flang/lib/FrontendTool/ExecuteCompilerInvocation.cpp

static std::unique_ptr<FrontendAction> CreateFrontendBaseAction(
   case ParseSyntaxOnly:
     return std::make_unique<ParseSyntaxOnlyAction>();
   case EmitObj:
     return std::make_unique<EmitObjAction>();
     break;
   case DebugUnparse:
     return std::make_unique<DebugUnparseAction>();
     break;
+  case DebugUnparseNoSema:
+    return std::make_unique<DebugUnparseNoSemaAction>();
+    break;
   case DebugUnparseWithSymbols:
     return std::make_unique<DebugUnparseWithSymbolsAction>();
     break;
   case DebugDumpSymbols:
     return std::make_unique<DebugDumpSymbolsAction>();
     break;
   case DebugDumpParseTree:
     return std::make_unique<DebugDumpParseTreeAction>();
     break;
+  case DebugDumpParseTreeNoSema:
+    return std::make_unique<DebugDumpParseTreeNoSemaAction>();
+    break;
   case DebugDumpProvenance:
     return std::make_unique<DebugDumpProvenanceAction>();
     break;
   case DebugDumpParsingLog:
     return std::make_unique<DebugDumpParsingLogAction>();
     break;
   case DebugMeasureParseTree:
     return std::make_unique<DebugMeasureParseTreeAction>();

flang/test/Driver/driver-help.f90

 ! HELP-FC1-NEXT:OPTIONS:
 ! HELP-FC1-NEXT: -cpp                   Enable predefined and command line preprocessor macros
 ! HELP-FC1-NEXT: -D <macro>=<value>     Define <macro> to <value> (or 1 if <value> omitted)
 ! HELP-FC1-NEXT: -emit-obj Emit native object files
 ! HELP-FC1-NEXT: -E                     Only run the preprocessor
 ! HELP-FC1-NEXT: -falternative-parameter-statement
 ! HELP-FC1-NEXT: Enable the old style PARAMETER statement
 ! HELP-FC1-NEXT: -fbackslash            Specify that backslash in string introduces an escape character
+! HELP-FC1-NEXT: -fdebug-dump-parse-tree-no-sema
+! HELP-FC1-NEXT:                        Dump the parse tree (skips the semantic checks)
 ! HELP-FC1-NEXT: -fdebug-dump-parse-tree Dump the parse tree
 ! HELP-FC1-NEXT: -fdebug-dump-parsing-log
 ! HELP-FC1-NEXT:                   Run instrumented parse and dump the parsing log
 ! HELP-FC1-NEXT: -fdebug-dump-provenance Dump provenance
 ! HELP-FC1-NEXT: -fdebug-dump-symbols    Dump symbols after the semantic analysis
 ! HELP-FC1-NEXT: -fdebug-measure-parse-tree
 ! HELP-FC1-NEXT:                         Measure the parse tree
 ! HELP-FC1-NEXT: -fdebug-module-writer   Enable debug messages while writing module files
 ! HELP-FC1-NEXT: -fdebug-pre-fir-tree    Dump the pre-FIR tree
+! HELP-FC1-NEXT: -fdebug-unparse-no-sema Unparse and stop (skips the semantic checks)
 ! HELP-FC1-NEXT: -fdebug-unparse-with-symbols
 ! HELP-FC1-NEXT:                        Unparse and stop.
 ! HELP-FC1-NEXT: -fdebug-unparse        Unparse and stop.
 ! HELP-FC1-NEXT: -fdefault-double-8  Set the default double precision kind to an 8 byte wide type
 ! HELP-FC1-NEXT: -fdefault-integer-8 Set the default integer kind to an 8 byte wide type
 ! HELP-FC1-NEXT: -fdefault-real-8    Set the default real kind to an 8 byte wide type
 ! HELP-FC1-NEXT: -ffixed-form           Process source files in fixed form
 ! HELP-FC1-NEXT: -ffixed-line-length=<value>

flang/test/Driver/dump-parse-tree-no-sema.f90

+!----------
+! RUN lines
+!----------
+! RUN: %flang_fc1 -fdebug-dump-parse-tree %s 2>&1 | FileCheck %s --check-prefix=SEMA_ON
+! RUN: %flang_fc1 -fdebug-dump-parse-tree-no-sema %s 2>&1 | FileCheck %s --check-prefix=SEMA_OFF
+
+!-----------------
+! EXPECTEED OUTPUT
+!-----------------
+! SEMA_ON: | | | NamedConstant -> Name = 'i'
+! SEMA_ON-NEXT: | | | Constant -> Expr = '1_4'
+! SEMA_ON-NEXT: | | | | LiteralConstant -> IntLiteralConstant = '1'
+
+! SEMA_OFF: | | | NamedConstant -> Name = 'i'
+! SEMA_OFF-NEXT: | | | Constant -> Expr -> LiteralConstant -> IntLiteralConstant = '1'
+
+!-------
+! INPUT
+!-------
+parameter(i=1)
+integer :: j
+end program

flang/test/Parser/omp-allocate-unparse.f90

-! RUN: %f18 -fdebug-no-semantics -funparse -fopenmp %s | FileCheck %s
+! RUN: %flang_fc1 -fdebug-unparse-no-sema -fopenmp %s | FileCheck %s
 ! Check Unparsing of OpenMP Allocate directive
 
 program allocate_unparse
 use omp_lib
 
 real, dimension (:,:), allocatable :: darray
 integer :: a, b, m, n, t, x, y, z
 

flang/test/Parser/omp-atomic-unparse.f90

-! RUN: %f18 -fdebug-no-semantics -funparse -fopenmp %s | FileCheck %s
+! RUN: %flang_fc1 -fdebug-unparse-no-sema -fopenmp %s | FileCheck %s
 
 program main
    implicit none
    integer :: i, j = 10
 !READ
 !$omp atomic read
    i = j
 !$omp atomic seq_cst read

flang/tools/f18/f18.cpp

if (arg.empty()) {
       driver.debugModuleWriter = true;
     } else if (arg == "-fdebug-measure-parse-tree") {
       driver.measureTree = true;
     } else if (arg == "-fdebug-instrumented-parse" ||
         arg == "-fdebug-dump-parsing-log") {
       options.instrumentedParse = true;
     } else if (arg == "-fdebug-no-semantics") {
       driver.debugNoSemantics = true;
+    } else if (arg == "-fdebug-unparse-no-sema") {
+      driver.debugNoSemantics = true;
+      driver.dumpUnparse = true;
+    } else if (arg == "-fdebug-dump-parse-tree-no-sema") {
+      driver.debugNoSemantics = true;
+      driver.dumpParseTree = true;
+      driver.syntaxOnly = true;
     } else if (arg == "-funparse" || arg == "-fdebug-unparse") {
       driver.dumpUnparse = true;
     } else if (arg == "-funparse-with-symbols" ||
         arg == "-fdebug-unparse-with-symbols") {
       driver.dumpUnparseWithSymbols = true;
     } else if (arg == "-funparse-typed-exprs-to-f18-fc") {
       driver.unparseTypedExprsToF18_FC = true;
     } else if (arg == "-fparse-only" || arg == "-fsyntax-only") {