[flang][driver] Add debug measure-parse-tree and pre-fir-tree options

[FarisRehman]

Summary

Add the following options:

• -fdebug-measure-parse-tree
• -fdebug-pre-fir-tree

Summary of changes:

• Add 2 new frontend actions: DebugMeasureParseTreeAction and DebugPreFIRTreeAction
• Add MeasurementVisitor to FrontendActions.h
• Make reportFatalSemanticErrors return true if there are any fatal errors
• Port most of the -fdebug-pre-fir-tree tests to use the new driver if built, otherwise use f18

[FarisRehman]

Rebase off main

Rebase off main and add a comment to MeasurementVisitor.

[awarzynski]

Thank you for working on this @FarisRehman !

This semantics implemented here are identical to what f18 does. -fdebug-instrumented-parse from f18 becomes -fdebug-parsing-log in flang-new -fc1. This was proposed (and accepted) in the RFC.

I've left a few comments inline, but nothing major and otherwise this is ready to land.

flang/include/flang/Frontend/FrontendActions.h

18

This should be shared with f18.cpp, but currently we don't have a good place for that (and we probably want to understand how much is going to be shared first). So IMO it is fine to keep it here.

Could you add a comment similar to this so that it is clear that this should be moved at some point?

flang/lib/Frontend/FrontendActions.cpp

313

Could this use diagnostics instead? For example:

unsigned diagID = ci.diagnostics().getCustomDiagID(clang::DiagnosticsEngine::Error, "Pre FIR Tree is NULL.");
ci.diagnostics().Report(diagID);

flang/test/Flang-Driver/debug-measure-parse-tree.f90

24–25

The output for this test is:

Parse tree comprises 18 objects and occupies 1496 total bytes.

I think that the actual size (i.e. 18 and 1496) is not that relevant here and also prone to change (which may cause the test to become fragile).

Instead, I suggest that we verify that -fdebug-measure-parse-tree indeed instructs the driver to measure the parse and print the result. So, perhaps this instead:

Parse tree comprises {{[0-9]+}} objects and occupies {{[0-9]+}} total bytes.

flang/test/Flang-Driver/debug-parsing-log.f90

23 ↗

(On Diff #324362)

IIUC, this output is not really specific to -fdebug-parsing-log (-fsyntax-only prints this too). I think that this would be more specific to this flag:

! FRONTEN: fail 1
! FRONTEN: fail 2
! FRONTEN: fail 3

This is still not great, but looking at the log itself I struggle to notice anything that would stand out.

[awarzynski]

I've just realized that for -fdebug-parsing-log we need to set Fortran::parser::instrumentedParse. I don't see a good place for that just yet.

It's probably best extracting that option into a separate patch.

[FarisRehman]

Remove -fdebug-parsing-log

Remove -fdebug-parsing-log from this patch as it requires additional work that can be implemented in a separate patch.
Also address review comments by @awarzynski

[awarzynski]

Thank you for updating this @FarisRehman !

The original implementation of `-fdebug-parsing-log submitted here was not 100% compatible with -fdebug-instrumented-parse from f18 as we originally thought. I think that that's fine, but we need something that gives us such compatibility. We probably need another flag for that, e.g. -finstrumented-parse, that would toggle Fortran::parser::instrumentedParse. This way we will have this:

• -fdebug-parsing-log (flang-new -fc1) != -fdebug-instrumented-parse (f18)
• -fdebug-parsing-log -finstrumented-parse (flang-new -fc1) == -fdebug-instrumented-parse (f18)

But that's quite a few new scenarios and it's worth submitting as a separate patch.

You've also changed the order of flags in tests. This is also fine and guarantees that tests work with both f18 and flang-new -fc1:

• in flang-new -fc1 only the last _action_ option is take into account, so -fsyntax-only is effectively ignored (and not needed)
• in f18 we still need -fsyntax-only to make sure that code-generation is not run (e.g. to make f18 exit _after_ running the semantic checks), but the order is irrelevant

All this makes a lot of sense, thank you for adding these! LGTM!

clang/include/clang/Driver/Options.td

 def fdebug_unparse_with_symbols : Flag<["-"], "fdebug-unparse-with-symbols">, Group<Action_Group>,
   HelpText<"Unparse and stop.">;
 def fdebug_dump_symbols : Flag<["-"], "fdebug-dump-symbols">, Group<Action_Group>,
   HelpText<"Dump symbols after the semantic analysis">;
 def fdebug_dump_parse_tree : Flag<["-"], "fdebug-dump-parse-tree">, Group<Action_Group>,
   HelpText<"Dump the parse tree">;
 def fdebug_dump_provenance : Flag<["-"], "fdebug-dump-provenance">, Group<Action_Group>,
   HelpText<"Dump provenance">;
+def fdebug_measure_parse_tree : Flag<["-"], "fdebug-measure-parse-tree">, Group<Action_Group>,
+  HelpText<"Measure the parse tree">;
+def fdebug_pre_fir_tree : Flag<["-"], "fdebug-pre-fir-tree">, Group<Action_Group>,
+  HelpText<"Dump the pre-FIR tree">;
 
 }
 
 //===----------------------------------------------------------------------===//
 // CC1 Options
 //===----------------------------------------------------------------------===//
 
 let Flags = [CC1Option, NoDriverOption] in {

flang/include/flang/Frontend/FrontendActions.h

 //===- FrontendActions.h -----------------------------------------*- C++-*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_FLANG_FRONTEND_FRONTENDACTIONS_H
 #define LLVM_FLANG_FRONTEND_FRONTENDACTIONS_H
 
 #include "flang/Frontend/FrontendAction.h"
 #include "flang/Semantics/semantics.h"
 #include <memory>
 
 namespace Fortran::frontend {
 
+// TODO: This is a copy from f18.cpp. It doesn't really belong here and should
+// be moved to a more suitable place in future.
+struct MeasurementVisitor {
+  template <typename A> bool Pre(const A &) { return true; }
+  template <typename A> void Post(const A &) {
+    ++objects;
+    bytes += sizeof(A);
+  }
+  size_t objects{0}, bytes{0};
+};
+
 //===----------------------------------------------------------------------===//
 // Custom Consumer Actions
 //===----------------------------------------------------------------------===//
 
 class InputOutputTestAction : public FrontendAction {
   void ExecuteAction() override;
 };
 

 class PrintPreprocessedAction : public PrescanAction {
   void ExecuteAction() override;
 };
 
 class DebugDumpProvenanceAction : public PrescanAction {
   void ExecuteAction() override;
 };
 
+class DebugMeasureParseTreeAction : public PrescanAction {
+  void ExecuteAction() override;
+};
+
 //===----------------------------------------------------------------------===//
 // PrescanAndSema Actions
 //===----------------------------------------------------------------------===//
 class PrescanAndSemaAction : public FrontendAction {
   std::unique_ptr<Fortran::semantics::Semantics> semantics_;
 
   void ExecuteAction() override = 0;
   bool BeginSourceFileAction(CompilerInstance &ci) override;

 class DebugDumpSymbolsAction : public PrescanAndSemaAction {
   void ExecuteAction() override;
 };
 
 class DebugDumpParseTreeAction : public PrescanAndSemaAction {
   void ExecuteAction() override;
 };
 
+class DebugPreFIRTreeAction : public PrescanAndSemaAction {
+  void ExecuteAction() override;
+};
+
 class ParseSyntaxOnlyAction : public PrescanAndSemaAction {
   void ExecuteAction() override;
 };
 
 } // namespace Fortran::frontend
 
 #endif // LLVM_FLANG_FRONTEND_FRONTENDACTIONS_H

flang/include/flang/Frontend/FrontendOptions.h

 //===- FrontendOptions.h ----------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 #ifndef LLVM_FLANG_FRONTEND_FRONTENDOPTIONS_H
 #define LLVM_FLANG_FRONTEND_FRONTENDOPTIONS_H
 
 #include "flang/Common/Fortran-features.h"
 #include "flang/Parser/characters.h"
 #include "flang/Parser/unparse.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/MemoryBuffer.h"
 
 #include <cstdint>
 #include <string>
 
enum ActionKind {
 
   /// Parse, run semantics and then output symbols from semantics
   DebugDumpSymbols,
 
   /// Parse, run semantics and then output the parse tree
   DebugDumpParseTree,
 
   /// Dump provenance
-  DebugDumpProvenance
+  DebugDumpProvenance,
+
+  /// Parse then output the number of objects in the parse tree and the overall
+  /// size
+  DebugMeasureParseTree,
+
+  /// Parse, run semantics and then output the pre-FIR tree
+  DebugPreFIRTree
 
   /// TODO: RunPreprocessor, EmitLLVM, EmitLLVMOnly,
   /// EmitCodeGenOnly, EmitAssembly, (...)
 };
 
 /// \param suffix The file extension
 /// \return True if the file extension should be processed as fixed form
 bool isFixedFormSuffix(llvm::StringRef suffix);

flang/lib/Frontend/CMakeLists.txt

add_flang_library(flangFrontend
   DEPENDS
   clangBasic
 
   LINK_LIBS
   FortranParser
   FortranSemantics
   FortranEvaluate
   FortranCommon
+  FortranLower
   clangBasic
   clangDriver
 
   LINK_COMPONENTS
   Option
   Support
   FrontendOpenACC
   FrontendOpenMP
 )

flang/lib/Frontend/CompilerInvocation.cpp

 //===- CompilerInvocation.cpp ---------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "flang/Frontend/CompilerInvocation.h"
 #include "flang/Common/Fortran-features.h"
 #include "flang/Frontend/PreprocessorOptions.h"
 #include "flang/Semantics/semantics.h"
 #include "flang/Version.inc"
 #include "clang/Basic/AllDiagnostics.h"
 #include "clang/Basic/DiagnosticDriver.h"
 #include "clang/Basic/DiagnosticOptions.h"
 #include "clang/Driver/DriverDiagnostic.h"
case clang::driver::options::OPT_fdebug_dump_symbols:
       opts.programAction_ = DebugDumpSymbols;
       break;
     case clang::driver::options::OPT_fdebug_dump_parse_tree:
       opts.programAction_ = DebugDumpParseTree;
       break;
     case clang::driver::options::OPT_fdebug_dump_provenance:
       opts.programAction_ = DebugDumpProvenance;
       break;
+    case clang::driver::options::OPT_fdebug_measure_parse_tree:
+      opts.programAction_ = DebugMeasureParseTree;
+      break;
+    case clang::driver::options::OPT_fdebug_pre_fir_tree:
+      opts.programAction_ = DebugPreFIRTree;
+      break;
 
       // TODO:
       // case calng::driver::options::OPT_emit_llvm:
       // case clang::driver::options::OPT_emit_llvm_only:
       // case clang::driver::options::OPT_emit_codegen_only:
       // case clang::driver::options::OPT_emit_module:
       // (...)
     }

flang/lib/Frontend/FrontendActions.cpp

 //===--- FrontendActions.cpp ----------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "flang/Frontend/FrontendActions.h"
 #include "flang/Common/default-kinds.h"
 #include "flang/Frontend/CompilerInstance.h"
 #include "flang/Frontend/FrontendOptions.h"
+#include "flang/Lower/PFTBuilder.h"
 #include "flang/Parser/dump-parse-tree.h"
 #include "flang/Parser/parsing.h"
 #include "flang/Parser/provenance.h"
 #include "flang/Parser/source.h"
 #include "flang/Parser/unparse.h"
 #include "flang/Semantics/semantics.h"
 #include "flang/Semantics/unparse-with-symbols.h"
 #include "llvm/ADT/StringRef.h"
 #include <clang/Basic/Diagnostic.h>
 #include <memory>
 
 using namespace Fortran::frontend;
 
-void reportFatalSemanticErrors(const Fortran::semantics::Semantics &semantics,
+/// Report fatal semantic errors if present.
+///
+/// \param semantics The semantics instance
+/// \param diags The diagnostics engine instance
+/// \param bufferName The file or buffer name
+///
+/// \return True if fatal semantic errors are present, false if not
+bool reportFatalSemanticErrors(const Fortran::semantics::Semantics &semantics,
     clang::DiagnosticsEngine &diags, const llvm::StringRef &bufferName) {
   if (semantics.AnyFatalError()) {
     unsigned DiagID = diags.getCustomDiagID(
         clang::DiagnosticsEngine::Error, "Semantic errors in %0");
     diags.Report(DiagID) << bufferName;
+    return true;
   }
+  return false;
 }
 
 bool PrescanAction::BeginSourceFileAction(CompilerInstance &c1) {
   CompilerInstance &ci = this->instance();
 
   std::string currentInputPath{GetCurrentFileOrBufferName()};
 
   Fortran::parser::Options parserOptions = ci.invocation().fortranOpts();
void DebugDumpParseTreeAction::ExecuteAction() {
 
   // Dump parse tree
   Fortran::parser::DumpTree(llvm::outs(), parseTree, &asFortran);
   // Report fatal semantic errors
   reportFatalSemanticErrors(semantics(), this->instance().diagnostics(),
       GetCurrentFileOrBufferName());
 }
 
+void DebugMeasureParseTreeAction::ExecuteAction() {
+  CompilerInstance &ci = this->instance();
+
+  // Parse. In case of failure, report and return.
+  ci.parsing().Parse(llvm::outs());
+
+  if (ci.parsing().messages().AnyFatalError()) {
+    unsigned diagID = ci.diagnostics().getCustomDiagID(
+        clang::DiagnosticsEngine::Error, "Could not parse %0");
+    ci.diagnostics().Report(diagID) << GetCurrentFileOrBufferName();
+
+    ci.parsing().messages().Emit(
+        llvm::errs(), this->instance().allCookedSources());
+    return;
+  }
+
+  // Report the diagnostics from parsing
+  ci.parsing().messages().Emit(llvm::errs(), ci.allCookedSources());
+
+  auto &parseTree{*ci.parsing().parseTree()};
+
+  // Measure the parse tree
+  MeasurementVisitor visitor;
+  Fortran::parser::Walk(parseTree, visitor);
+  llvm::outs() << "Parse tree comprises " << visitor.objects
+               << " objects and occupies " << visitor.bytes
+               << " total bytes.\n";
+}
+
+void DebugPreFIRTreeAction::ExecuteAction() {
+  CompilerInstance &ci = this->instance();
+  // Report and exit if fatal semantic errors are present
+  if (reportFatalSemanticErrors(
+          semantics(), ci.diagnostics(), GetCurrentFileOrBufferName())) {
+    return;
+  }
+
+  auto &parseTree{*ci.parsing().parseTree()};
+
+  // Dump pre-FIR tree
+  if (auto ast{Fortran::lower::createPFT(
+          parseTree, ci.invocation().semanticsContext())}) {
+    Fortran::lower::dumpPFT(llvm::outs(), *ast);
+  } else {
+    unsigned diagID = ci.diagnostics().getCustomDiagID(
+        clang::DiagnosticsEngine::Error, "Pre FIR Tree is NULL.");
+    ci.diagnostics().Report(diagID);
+  }
+}
+
 void EmitObjAction::ExecuteAction() {
   CompilerInstance &ci = this->instance();
   unsigned DiagID = ci.diagnostics().getCustomDiagID(
       clang::DiagnosticsEngine::Error, "code-generation is not available yet");
   ci.diagnostics().Report(DiagID);
 }

flang/lib/FrontendTool/ExecuteCompilerInvocation.cpp

 //===--- ExecuteCompilerInvocation.cpp ------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file holds ExecuteCompilerInvocation(). It is split into its own file to
 // minimize the impact of pulling in essentially everything else in Flang.
 //
 //===----------------------------------------------------------------------===//
 
 #include "flang/Frontend/CompilerInstance.h"
 #include "flang/Frontend/FrontendActions.h"
 #include "clang/Driver/Options.h"
 #include "llvm/Option/OptTable.h"
 #include "llvm/Option/Option.h"
 #include "llvm/Support/BuryPointer.h"
 #include "llvm/Support/CommandLine.h"
 
 namespace Fortran::frontend {
case DebugDumpSymbols:
     return std::make_unique<DebugDumpSymbolsAction>();
     break;
   case DebugDumpParseTree:
     return std::make_unique<DebugDumpParseTreeAction>();
     break;
   case DebugDumpProvenance:
     return std::make_unique<DebugDumpProvenanceAction>();
     break;
+  case DebugMeasureParseTree:
+    return std::make_unique<DebugMeasureParseTreeAction>();
+    break;
+  case DebugPreFIRTree:
+    return std::make_unique<DebugPreFIRTreeAction>();
+    break;
   default:
     break;
     // TODO:
     // case RunPreprocessor:
     // case ParserSyntaxOnly:
     // case EmitLLVM:
     // case EmitLLVMOnly:
     // case EmitCodeGenOnly:

flang/test/Flang-Driver/debug-measure-parse-tree.f90

+! Ensure argument -fdebug-measure-parse-tree works as expected.
+
+! REQUIRES: new-flang-driver
+
+!--------------------------
+! FLANG DRIVER (flang-new)
+!--------------------------
+! RUN: not %flang-new -fdebug-measure-parse-tree %s  2>&1 | FileCheck %s --check-prefix=FLANG
+
+!----------------------------------------
+! FRONTEND FLANG DRIVER (flang-new -fc1)
+!----------------------------------------
+! RUN: %flang-new -fc1 -fdebug-measure-parse-tree %s  2>&1 | FileCheck %s --check-prefix=FRONTEND
+
+!----------------------------------
+! EXPECTED OUTPUT WITH `flang-new`
+!----------------------------------
+! FLANG:warning: argument unused during compilation: '-fdebug-measure-parse-tree'
+
+!---------------------------------------
+! EXPECTED OUTPUT WITH `flang-new -fc1`
+!---------------------------------------
+! FRONTEND:Parse tree comprises {{[0-9]+}} objects and occupies {{[0-9]+}} total bytes.
+
+program A
+end

flang/test/Flang-Driver/driver-help.f90

 ! HELP-FC1-NEXT: -emit-obj Emit native object files
 ! HELP-FC1-NEXT: -E                     Only run the preprocessor
 ! HELP-FC1-NEXT: -falternative-parameter-statement
 ! HELP-FC1-NEXT: Enable the old style PARAMETER statement
 ! HELP-FC1-NEXT: -fbackslash            Specify that backslash in string introduces an escape character
 ! HELP-FC1-NEXT: -fdebug-dump-parse-tree Dump the parse tree
 ! HELP-FC1-NEXT: -fdebug-dump-provenance Dump provenance
 ! HELP-FC1-NEXT: -fdebug-dump-symbols    Dump symbols after the semantic analysis
+! HELP-FC1-NEXT: -fdebug-measure-parse-tree
+! HELP-FC1-NEXT:                         Measure the parse tree
+! HELP-FC1-NEXT: -fdebug-pre-fir-tree    Dump the pre-FIR tree
 ! HELP-FC1-NEXT: -fdebug-unparse-with-symbols
 ! HELP-FC1-NEXT:                        Unparse and stop.
 ! HELP-FC1-NEXT: -fdebug-unparse        Unparse and stop.
 ! HELP-FC1-NEXT: -ffixed-form           Process source files in fixed form
 ! HELP-FC1-NEXT: -ffixed-line-length=<value>
 ! HELP-FC1-NEXT: Use <value> as character line width in fixed mode
 ! HELP-FC1-NEXT: -ffree-form            Process source files in free form
 ! HELP-FC1-NEXT: -fimplicit-none        No implicit typing allowed unless overridden by IMPLICIT statements

flang/test/Lower/pre-fir-tree01.f90

-! RUN: %f18 -fdebug-pre-fir-tree -fsyntax-only %s | FileCheck %s
+! RUN: %flang_fc1 -fsyntax-only -fdebug-pre-fir-tree %s | FileCheck %s
 
 ! Test structure of the Pre-FIR tree
 
 ! CHECK: Subroutine foo
 subroutine foo()
   ! CHECK: <<DoConstruct>>
   ! CHECK: NonLabelDoStmt
   do i=1,5

flang/test/Lower/pre-fir-tree02.f90

-! RUN: %f18 -fdebug-pre-fir-tree -fsyntax-only %s | FileCheck %s
+! RUN: %flang_fc1 -fsyntax-only -fdebug-pre-fir-tree %s | FileCheck %s
 
 ! Test Pre-FIR Tree captures all the intended nodes from the parse-tree
 ! Coarray and OpenMP related nodes are tested in other files.
 
 ! CHECK: Program test_prog
 program test_prog
   ! Check specification part is not part of the tree.
   interface

flang/test/Lower/pre-fir-tree03.f90

-! RUN: %f18 -fdebug-pre-fir-tree -fsyntax-only -fopenmp %s | FileCheck %s
+! RUN: %flang_fc1 -fsyntax-only -fdebug-pre-fir-tree -fopenmp %s | FileCheck %s
 
 ! Test Pre-FIR Tree captures OpenMP related constructs
 
 ! CHECK: Program test_omp
 program test_omp
   ! CHECK: PrintStmt
   print *, "sequential"
 

flang/test/Lower/pre-fir-tree05.f90

-! RUN: %f18 -fdebug-pre-fir-tree -fsyntax-only -fopenacc %s | FileCheck %s
+! RUN: %flang_fc1 -fsyntax-only -fdebug-pre-fir-tree -fopenacc %s | FileCheck %s
 
 ! Test structure of the Pre-FIR tree with OpenACC construct
 
 ! CHECK: Subroutine foo
 subroutine foo()
   ! CHECK-NEXT: <<OpenACCConstruct>>
   ! CHECK-NEXT: <<OpenACCConstruct>>
   !$acc parallel