[flang][driver] Add debug dump options

[FarisRehman]

Summary

Add the following options:

• -fdebug-dump-symbols
• -fdebug-dump-parse-tree
• -fdebug-dump-provenance

Summary of changes:

• Add 3 new frontend actions: DebugDumpSymbolsAction, DebugDumpParseTreeAction and DebugDumpProvenanceAction
• Add a unique pointer to the Semantics instance created in PrescanAndSemaAction
• Move fatal semantic error reporting to its own method, FrontendActions#reportFatalSemanticErrors
• Port most tests using -fdebug-dump-symbols and -fdebug-dump-parse-tree to the new driver if built, otherwise default to f18

[FarisRehman]

Rebase off main

Rebase off main and update some comments.

[awarzynski]

I think that this patch does a great job demonstrating the use of frontend actions and how _action_ compiler flags map to them. Given how Fortran::semantics::Semantics are defined and created, I think that adding _semantics_ as a member variable to PrescanAndSemaAction makes a lot of sense.

Btw, I also think that it would be worth exploring what Clang does. Ideally we'd have some Sema API that would be a member of CompilerInstance. However, that's outside the scope of this patch :)

Apart from one comment this looks good to me. Thank you for submitting this @FarisRehman !

flang/include/flang/Frontend/FrontendActions.h

56

Do we need to expose it? Shouldn't this be private?

[FarisRehman]

flang/include/flang/Frontend/FrontendActions.h

56

It will not compile if the variable is private as it needs to be referenced in the implementation of ExecuteAction, e.g. DebugDumpParseTreeAction::ExecuteAction()
It will compile if set to protected or public

[FarisRehman]

Make semantics_ private

Make semantics_ in PrescanAndSemaAction private, adding semantics() and setSemantics().

Summary of changes:

• Add semantics() and setSemantics() to PrescanAndSemaAction
• Update tests to use -fsyntax-only but specify it first so that the new driver ignores it as only the last action option is executed. f18 will still run as if -fsyntax-only is present.

flang/include/flang/Frontend/FrontendActions.h

56

Updated to use a public getter and setter

[awarzynski]

This patch doesn't require the tests to be updated, does it? Also: https://reviews.llvm.org/D96870. So it looks these are unrelated changes. Could you remove them before merging?

Otherwise, LGTM. Thanks for working on this!

[FarisRehman]

In D96716#2569102, @awarzynski wrote:

This patch doesn't require the tests to be updated, does it?

They do require updating as -fdebug-dump-symbol and -fdebug-dump-parse-tree are otherwise not tested by the new driver in this patch.

[awarzynski]

In D96716#2569160, @FarisRehman wrote:

They do require updating as -fdebug-dump-symbol and -fdebug-dump-parse-tree are otherwise not tested by the new driver in this patch.

Apologies, I wasn't clear (realized after submitting). We need %f18 to be replaced with %flang_fc1, but we don't need -fsyntax-only, right?

[FarisRehman]

In D96716#2569594, @awarzynski wrote:

We need %f18 to be replaced with %flang_fc1, but we don't need -fsyntax-only, right?

Makes sense thanks, I'll remove before committing.

clang/include/clang/Driver/Options.td

 // FC1 Options
 //===----------------------------------------------------------------------===//
 let Flags = [FC1Option, FlangOnlyOption] in {
 
 def fdebug_unparse : Flag<["-"], "fdebug-unparse">, Group<Action_Group>,
   HelpText<"Unparse and stop.">;
 def fdebug_unparse_with_symbols : Flag<["-"], "fdebug-unparse-with-symbols">, Group<Action_Group>,
   HelpText<"Unparse and stop.">;
+def fdebug_dump_symbols : Flag<["-"], "fdebug-dump-symbols">, Group<Action_Group>,
+  HelpText<"Dump symbols after the semantic analysis">;
+def fdebug_dump_parse_tree : Flag<["-"], "fdebug-dump-parse-tree">, Group<Action_Group>,
+  HelpText<"Dump the parse tree">;
+def fdebug_dump_provenance : Flag<["-"], "fdebug-dump-provenance">, Group<Action_Group>,
+  HelpText<"Dump provenance">;
 
 }
 
 //===----------------------------------------------------------------------===//
 // CC1 Options
 //===----------------------------------------------------------------------===//
 
 let Flags = [CC1Option, NoDriverOption] in {

flang/include/flang/Frontend/FrontendActions.h

 //===- FrontendActions.h -----------------------------------------*- C++-*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_FLANG_FRONTEND_FRONTENDACTIONS_H
 #define LLVM_FLANG_FRONTEND_FRONTENDACTIONS_H
 
 #include "flang/Frontend/FrontendAction.h"
+#include "flang/Semantics/semantics.h"
+#include <memory>
 
 namespace Fortran::frontend {
 
 //===----------------------------------------------------------------------===//
 // Custom Consumer Actions
 //===----------------------------------------------------------------------===//
 
 class InputOutputTestAction : public FrontendAction {
class PrescanAction : public FrontendAction {
   void ExecuteAction() override = 0;
   bool BeginSourceFileAction(CompilerInstance &ci) override;
 };
 
 class PrintPreprocessedAction : public PrescanAction {
   void ExecuteAction() override;
 };
 
+class DebugDumpProvenanceAction : public PrescanAction {
+  void ExecuteAction() override;
+};
+
 //===----------------------------------------------------------------------===//
 // PrescanAndSema Actions
 //===----------------------------------------------------------------------===//
 class PrescanAndSemaAction : public FrontendAction {
+  std::unique_ptr<Fortran::semantics::Semantics> semantics_;
+
   void ExecuteAction() override = 0;
   bool BeginSourceFileAction(CompilerInstance &ci) override;
+
+public:
+  Fortran::semantics::Semantics &semantics() { return *semantics_; }
+  const Fortran::semantics::Semantics &semantics() const { return *semantics_; }
+
+  void setSemantics(std::unique_ptr<Fortran::semantics::Semantics> semantics) {
+    semantics_ = std::move(semantics);
+  }
 };
 
 class DebugUnparseWithSymbolsAction : public PrescanAndSemaAction {
   void ExecuteAction() override;
 };
 
 class DebugUnparseAction : public PrescanAndSemaAction {
   void ExecuteAction() override;
 };
 
+class DebugDumpSymbolsAction : public PrescanAndSemaAction {
+  void ExecuteAction() override;
+};
+
+class DebugDumpParseTreeAction : public PrescanAndSemaAction {
+  void ExecuteAction() override;
+};
+
 class ParseSyntaxOnlyAction : public PrescanAndSemaAction {
   void ExecuteAction() override;
 };
 
 } // namespace Fortran::frontend
 
 #endif // LLVM_FLANG_FRONTEND_FRONTENDACTIONS_H

flang/include/flang/Frontend/FrontendOptions.h

enum ActionKind {
 
   /// Parse, unparse the parse-tree and output a Fortran source file
   DebugUnparse,
 
   /// Parse, resolve the sybmols, unparse the parse-tree and then output a
   /// Fortran source file
   DebugUnparseWithSymbols,
 
+  /// Parse, run semantics and then output symbols from semantics
+  DebugDumpSymbols,
+
+  /// Parse, run semantics and then output the parse tree
+  DebugDumpParseTree,
+
+  /// Dump provenance
+  DebugDumpProvenance
+
   /// TODO: RunPreprocessor, EmitLLVM, EmitLLVMOnly,
   /// EmitCodeGenOnly, EmitAssembly, (...)
 };
 
 /// \param suffix The file extension
 /// \return True if the file extension should be processed as fixed form
 bool isFixedFormSuffix(llvm::StringRef suffix);
 

flang/lib/Frontend/CompilerInvocation.cpp

case clang::driver::options::OPT_emit_obj:
       opts.programAction_ = EmitObj;
       break;
     case clang::driver::options::OPT_fdebug_unparse:
       opts.programAction_ = DebugUnparse;
       break;
     case clang::driver::options::OPT_fdebug_unparse_with_symbols:
       opts.programAction_ = DebugUnparseWithSymbols;
       break;
+    case clang::driver::options::OPT_fdebug_dump_symbols:
+      opts.programAction_ = DebugDumpSymbols;
+      break;
+    case clang::driver::options::OPT_fdebug_dump_parse_tree:
+      opts.programAction_ = DebugDumpParseTree;
+      break;
+    case clang::driver::options::OPT_fdebug_dump_provenance:
+      opts.programAction_ = DebugDumpProvenance;
+      break;
 
       // TODO:
       // case calng::driver::options::OPT_emit_llvm:
       // case clang::driver::options::OPT_emit_llvm_only:
       // case clang::driver::options::OPT_emit_codegen_only:
       // case clang::driver::options::OPT_emit_module:
       // (...)
     }

flang/lib/Frontend/FrontendActions.cpp

 //===--- FrontendActions.cpp ----------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "flang/Frontend/FrontendActions.h"
 #include "flang/Common/default-kinds.h"
 #include "flang/Frontend/CompilerInstance.h"
 #include "flang/Frontend/FrontendOptions.h"
+#include "flang/Parser/dump-parse-tree.h"
 #include "flang/Parser/parsing.h"
 #include "flang/Parser/provenance.h"
 #include "flang/Parser/source.h"
 #include "flang/Parser/unparse.h"
 #include "flang/Semantics/semantics.h"
 #include "flang/Semantics/unparse-with-symbols.h"
+#include "llvm/ADT/StringRef.h"
+#include <clang/Basic/Diagnostic.h>
+#include <memory>
 
 using namespace Fortran::frontend;
 
+void reportFatalSemanticErrors(const Fortran::semantics::Semantics &semantics,
+    clang::DiagnosticsEngine &diags, const llvm::StringRef &bufferName) {
+  if (semantics.AnyFatalError()) {
+    unsigned DiagID = diags.getCustomDiagID(
+        clang::DiagnosticsEngine::Error, "Semantic errors in %0");
+    diags.Report(DiagID) << bufferName;
+  }
+}
+
 bool PrescanAction::BeginSourceFileAction(CompilerInstance &c1) {
   CompilerInstance &ci = this->instance();
 
   std::string currentInputPath{GetCurrentFileOrBufferName()};
 
   Fortran::parser::Options parserOptions = ci.invocation().fortranOpts();
 
   if (ci.invocation().frontendOpts().fortranForm_ == FortranForm::Unknown) {
bool PrescanAndSemaAction::BeginSourceFileAction(CompilerInstance &c1) {
   }
 
   // Report the diagnostics from parsing
   ci.parsing().messages().Emit(llvm::errs(), ci.allCookedSources());
 
   auto &parseTree{*ci.parsing().parseTree()};
 
   // Prepare semantics
-  Fortran::semantics::Semantics semantics{ci.invocation().semanticsContext(),
-      parseTree, ci.parsing().cooked().AsCharBlock()};
+  setSemantics(std::make_unique<Fortran::semantics::Semantics>(
+      ci.invocation().semanticsContext(), parseTree,
+      ci.parsing().cooked().AsCharBlock()));
+  auto &semantics = this->semantics();
 
   // Run semantic checks
   semantics.Perform();
 
   // Report the diagnostics from the semantic checks
   semantics.EmitMessages(ci.semaOutputStream());
-
-  if (semantics.AnyFatalError()) {
-    unsigned DiagID = ci.diagnostics().getCustomDiagID(
-        clang::DiagnosticsEngine::Error, "Semantic errors in %0");
-    ci.diagnostics().Report(DiagID) << GetCurrentFileOrBufferName();
-
-    return false;
-  }
-
   return true;
 }
 
 void InputOutputTestAction::ExecuteAction() {
   CompilerInstance &ci = instance();
 
   // Create a stream for errors
   std::string buf;
if (auto os{ci.CreateDefaultOutputFile(
           /*Binary=*/true, /*InFile=*/GetCurrentFileOrBufferName())}) {
     (*os) << outForPP.str();
   } else {
     llvm::errs() << "Unable to create the output file\n";
     return;
   }
 }
 
-void ParseSyntaxOnlyAction::ExecuteAction() {}
+void DebugDumpProvenanceAction::ExecuteAction() {
+  this->instance().parsing().DumpProvenance(llvm::outs());
+}
+
+void ParseSyntaxOnlyAction::ExecuteAction() {
+  reportFatalSemanticErrors(semantics(), this->instance().diagnostics(),
+      GetCurrentFileOrBufferName());
+}
 
 void DebugUnparseAction::ExecuteAction() {
   auto &parseTree{instance().parsing().parseTree()};
   Fortran::parser::AnalyzedObjectsAsFortran asFortran =
       Fortran::frontend::getBasicAsFortran();
 
   // TODO: Options should come from CompilerInvocation
   Unparse(llvm::outs(), *parseTree,
       /*encoding=*/Fortran::parser::Encoding::UTF_8,
       /*capitalizeKeywords=*/true, /*backslashEscapes=*/false,
       /*preStatement=*/nullptr, &asFortran);
+
+  // Report fatal semantic errors
+  reportFatalSemanticErrors(semantics(), this->instance().diagnostics(),
+      GetCurrentFileOrBufferName());
 }
 
 void DebugUnparseWithSymbolsAction::ExecuteAction() {
   auto &parseTree{*instance().parsing().parseTree()};
 
   Fortran::semantics::UnparseWithSymbols(
       llvm::outs(), parseTree, /*encoding=*/Fortran::parser::Encoding::UTF_8);
+
+  // Report fatal semantic errors
+  reportFatalSemanticErrors(semantics(), this->instance().diagnostics(),
+      GetCurrentFileOrBufferName());
+}
+
+void DebugDumpSymbolsAction::ExecuteAction() {
+  auto &semantics = this->semantics();
+
+  // Dump symbols
+  semantics.DumpSymbols(llvm::outs());
+  // Report fatal semantic errors
+  reportFatalSemanticErrors(
+      semantics, this->instance().diagnostics(), GetCurrentFileOrBufferName());
+}
+
+void DebugDumpParseTreeAction::ExecuteAction() {
+  auto &parseTree{instance().parsing().parseTree()};
+  Fortran::parser::AnalyzedObjectsAsFortran asFortran =
+      Fortran::frontend::getBasicAsFortran();
+
+  // Dump parse tree
+  Fortran::parser::DumpTree(llvm::outs(), parseTree, &asFortran);
+  // Report fatal semantic errors
+  reportFatalSemanticErrors(semantics(), this->instance().diagnostics(),
+      GetCurrentFileOrBufferName());
 }
 
 void EmitObjAction::ExecuteAction() {
   CompilerInstance &ci = this->instance();
   unsigned DiagID = ci.diagnostics().getCustomDiagID(
       clang::DiagnosticsEngine::Error, "code-generation is not available yet");
   ci.diagnostics().Report(DiagID);
 }

flang/lib/FrontendTool/ExecuteCompilerInvocation.cpp

case EmitObj:
     return std::make_unique<EmitObjAction>();
     break;
   case DebugUnparse:
     return std::make_unique<DebugUnparseAction>();
     break;
   case DebugUnparseWithSymbols:
     return std::make_unique<DebugUnparseWithSymbolsAction>();
     break;
+  case DebugDumpSymbols:
+    return std::make_unique<DebugDumpSymbolsAction>();
+    break;
+  case DebugDumpParseTree:
+    return std::make_unique<DebugDumpParseTreeAction>();
+    break;
+  case DebugDumpProvenance:
+    return std::make_unique<DebugDumpProvenanceAction>();
+    break;
   default:
     break;
     // TODO:
     // case RunPreprocessor:
     // case ParserSyntaxOnly:
     // case EmitLLVM:
     // case EmitLLVMOnly:
     // case EmitCodeGenOnly:

flang/test/Flang-Driver/debug-provenance.f90

+! Ensure argument -fdebug-dump-provenance works as expected.
+
+! REQUIRES: new-flang-driver
+
+!--------------------------
+! FLANG DRIVER (flang-new)
+!--------------------------
+! RUN: not %flang-new -fdebug-dump-provenance %s  2>&1 | FileCheck %s --check-prefix=FLANG
+
+!----------------------------------------
+! FRONTEND FLANG DRIVER (flang-new -fc1)
+!----------------------------------------
+! RUN: %flang-new -fc1 -fdebug-dump-provenance %s  2>&1 | FileCheck %s --check-prefix=FRONTEND
+
+!----------------------------------
+! EXPECTED OUTPUT WITH `flang-new`
+!----------------------------------
+! FLANG:warning: argument unused during compilation: '-fdebug-dump-provenance'
+
+!---------------------------------------
+! EXPECTED OUTPUT WITH `flang-new -fc1`
+!---------------------------------------
+! FRONTEND:AllSources:
+! FRONTEND:CookedSource::provenanceMap_:
+program A
+end

flang/test/Flang-Driver/driver-help.f90

 ! HELP-FC1-EMPTY:
 ! HELP-FC1-NEXT:OPTIONS:
 ! HELP-FC1-NEXT: -D <macro>=<value>     Define <macro> to <value> (or 1 if <value> omitted)
 ! HELP-FC1-NEXT: -emit-obj Emit native object files
 ! HELP-FC1-NEXT: -E                     Only run the preprocessor
 ! HELP-FC1-NEXT: -falternative-parameter-statement
 ! HELP-FC1-NEXT: Enable the old style PARAMETER statement
 ! HELP-FC1-NEXT: -fbackslash            Specify that backslash in string introduces an escape character
+! HELP-FC1-NEXT: -fdebug-dump-parse-tree Dump the parse tree
+! HELP-FC1-NEXT: -fdebug-dump-provenance Dump provenance
+! HELP-FC1-NEXT: -fdebug-dump-symbols    Dump symbols after the semantic analysis
 ! HELP-FC1-NEXT: -fdebug-unparse-with-symbols
 ! HELP-FC1-NEXT:                        Unparse and stop.
 ! HELP-FC1-NEXT: -fdebug-unparse        Unparse and stop.
 ! HELP-FC1-NEXT: -ffixed-form           Process source files in fixed form
 ! HELP-FC1-NEXT: -ffixed-line-length=<value>
 ! HELP-FC1-NEXT: Use <value> as character line width in fixed mode
 ! HELP-FC1-NEXT: -ffree-form            Process source files in free form
 ! HELP-FC1-NEXT: -fimplicit-none        No implicit typing allowed unless overridden by IMPLICIT statements

flang/test/Semantics/data05.f90

-!RUN: %f18 -fdebug-dump-symbols %s | FileCheck %s
+!RUN: %flang_fc1 -fdebug-dump-symbols %s | FileCheck %s
 module m
   interface
     integer function ifunc(n)
       integer, intent(in) :: n
     end function
     real function rfunc(x)
       real, intent(in) :: x
     end function

flang/test/Semantics/data08.f90

-! RUN: %f18 -fdebug-dump-symbols %s 2>&1 | FileCheck %s
+! RUN: %flang_fc1 -fdebug-dump-symbols %s 2>&1 | FileCheck %s
 ! CHECK: DATA statement value initializes 'jx' of type 'INTEGER(4)' with CHARACTER
 ! CHECK: DATA statement value initializes 'jy' of type 'INTEGER(4)' with CHARACTER
 ! CHECK: DATA statement value initializes 'jz' of type 'INTEGER(4)' with CHARACTER
 ! CHECK: DATA statement value initializes 'kx' of type 'INTEGER(8)' with CHARACTER
 ! CHECK: jx (InDataStmt) size=4 offset=0: ObjectEntity type: INTEGER(4) init:1684234849_4
 ! CHECK: jy (InDataStmt) size=4 offset=4: ObjectEntity type: INTEGER(4) init:543384161_4
 ! CHECK: jz (InDataStmt) size=4 offset=8: ObjectEntity type: INTEGER(4) init:1684234849_4
 ! CHECK: kx (InDataStmt) size=8 offset=16: ObjectEntity type: INTEGER(8) init:7523094288207667809_8
 
 integer :: jx, jy, jz
 integer(8) :: kx
 data jx/4habcd/
 data jy/3habc/
 data jz/5habcde/
 data kx/'abcdefgh'/
 end

flang/test/Semantics/data09.f90

-! RUN: %f18 -fsyntax-only -fdebug-dump-symbols %s 2>&1 | FileCheck %s
+! RUN: %flang_fc1 -fsyntax-only -fdebug-dump-symbols %s 2>&1 | FileCheck %s
 ! CHECK: init:[INTEGER(4)::1065353216_4,1073741824_4,1077936128_4,1082130432_4]
 ! Verify that the closure of EQUIVALENCE'd symbols with any DATA
 ! initialization produces a combined initializer.
 real :: a(2), b(2), c(2)
 equivalence(a(2),b(1)),(b(2),c(1))
 data a(1)/1./,b(1)/2./,c/3.,4./
 common /block/ a
 end

flang/test/Semantics/offsets01.f90

-!RUN: %f18 -fdebug-dump-symbols %s | FileCheck %s
+!RUN: %flang_fc1 -fdebug-dump-symbols %s | FileCheck %s
 
 ! Size and alignment of intrinsic types
 subroutine s1
   integer(1) :: a_i1  !CHECK: a_i1 size=1 offset=0:
   integer(8) :: b_i8  !CHECK: b_i8 size=8 offset=8:
   real(2)    :: c_r2  !CHECK: c_r2 size=2 offset=16:
   real(2)    :: d_r2  !CHECK: d_r2 size=2 offset=18:
   real(8)    :: e_r8  !CHECK: e_r8 size=8 offset=24:

flang/test/Semantics/offsets02.f90

-!RUN: %f18 -fdebug-dump-symbols %s | FileCheck %s
+!RUN: %flang_fc1 -fdebug-dump-symbols %s | FileCheck %s
 
 ! Size and alignment of derived types
 
 ! Array of derived type with 64-bit alignment
 subroutine s1
   type t1
     real(8) :: a
     real(4) :: b

flang/test/Semantics/offsets03.f90

-!RUN: %f18 -fdebug-dump-symbols %s | FileCheck %s
+!RUN: %flang_fc1 -fdebug-dump-symbols %s | FileCheck %s
 
 ! Size and alignment with EQUIVALENCE and COMMON
 
 ! a1 depends on a2 depends on a3
 module ma
   real :: a1(10), a2(10), a3(10)
   equivalence(a1, a2(3)) !CHECK: a1, PUBLIC size=40 offset=20:
   equivalence(a2, a3(4)) !CHECK: a2, PUBLIC size=40 offset=12:

flang/test/Semantics/resolve100.f90

-!RUN: %f18 -fdebug-dump-symbols %s | FileCheck %s
+!RUN: %flang_fc1 -fdebug-dump-symbols %s | FileCheck %s
 
 program p
   ! CHECK: a size=4 offset=0: ObjectEntity type: LOGICAL(4)
   ! CHECK: b size=4 offset=4: ObjectEntity type: REAL(4)
   logical :: a = .false.
   real :: b = 9.73
   ! CHECK: a: AssocEntity type: REAL(4) expr:b
   ! CHECK: b: AssocEntity type: LOGICAL(4) expr:a
   associate (b => a, a => b)
     print*, a, b
   end associate
   print*, a, b
 end

flang/test/Semantics/rewrite01.f90

-! RUN: %f18 -fdebug-dump-parse-tree %s 2>&1 | FileCheck %s
+! RUN: %flang_fc1 -fdebug-dump-parse-tree %s 2>&1 | FileCheck %s
 ! Ensure that READ(CVAR) [, item-list] is corrected when CVAR is a
 ! character variable so as to be a formatted read from the default
 ! unit, not an unformatted read from an internal unit (which is not
 ! possible in Fortran).
 character :: cvar
 ! CHECK-NOT: IoUnit -> Variable -> Designator -> DataRef -> Name = 'cvar'
 ! CHECK: Format -> Expr = 'cvar'
 read(cvar)
 end