[flang][driver] Add `-fdebug-dump-all`

[awarzynski]

Summary

The new option will run the semantic checks and then dump the parse tree
and all the symbols. This is equivalent to running the driver twice,
once with -fdebug-dump-parse-tree and then with
the -fdebug-dump-symbols action flag.

Currently we wouldn't be able to achieve the same by simply running:

flang-new -fc1 -fdebug-dump-parse-tree -fdebug-dump-symbols <input-file>

That's because the new driver will only run one frontend action per
invocation (both of the flags used here are action flags). Diverging
from this design would lead to costly compromises and it's best avoided.

We may want to consider re-designing our debugging actions (and action
options) in the future so that there's more code re-use. For now, I'm
focusing on making sure that we support all the major cases requested by
our users.

[PeteSteinfeld]

Thanks for doing this. These changes look good, but I can't get them to build successfully. I'm using the GNU 9.3 C++ compiler. My build fails to link the bin/flang-new executable. Here's an excerpt of the error messages:

[4436/4439] Generating ../../../../include/flang/iso_fortran_env.f18.mod
[4437/4439] Linking CXX executable bin/flang-new
FAILED: bin/flang-new 
: && /home/sw/thirdparty/gcc/gcc-9.3.0/linux86-64/bin/g++ -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wno-maybe-uninitialized -Wno-class-memaccess -Wno-redundant-move -Wno-pessimizing-move -Wno-noexcept-type -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment -Wmisleading-indentation -fdiagnostics-color -ffunction-sections -fdata-sections -Werror -Wno-deprecated-copy -fno-strict-aliasing -fno-semantic-interposition -O3 -DNDEBUG -Wl,-rpath,/home/sw/thirdparty/gcc/gcc-9.3.0/linux86-64/lib64:/usr/local/lib:/home/sw/envmod/modules/linux86-64/latest/lib: -Wl,-rpath-link,/local/home/psteinfeld/main/tot/build/./lib  -Wl,-O3 -Wl,--gc-sections tools/flang/tools/flang-driver/CMakeFiles/flang-new.dir/driver.cpp.o tools/flang/tools/flang-driver/CMakeFiles/flang-new.dir/fc1_main.cpp.o -o bin/flang-new -L/local/home/psteinfeld/main/tot/build/./lib -Wl,-rpath,"\$ORIGIN/../lib"  lib/libLLVMX86CodeGen.a  lib/libLLVMX86AsmParser.a  lib/libLLVMX86Desc.a  lib/libLLVMX86Disassembler.a  lib/libLLVMX86Info.a  lib/libLLVMOption.a  lib/libLLVMSupport.a  -lpthread  lib/libflangFrontend.a  lib/libflangFrontendTool.a  lib/libclangDriver.a  lib/libclangBasic.a  lib/libLLVMGlobalISel.a  lib/libLLVMSelectionDAG.a  lib/libLLVMCFGuard.a  lib/libLLVMMCDisassembler.a  lib/libflangFrontend.a  lib/libFortranLower.a  lib/libFortranSemantics.a  lib/libFortranEvaluate.a  lib/libFortranParser.a  lib/libFortranDecimal.a  lib/libFortranCommon.a  lib/libFIROptimizer.a  lib/libLLVMAsmPrinter.a  lib/libLLVMCodeGen.a  lib/libLLVMScalarOpts.a  lib/libLLVMAggressiveInstCombine.a  lib/libLLVMInstCombine.a  lib/libLLVMTarget.a  lib/libLLVMDebugInfoDWARF.a  lib/libLLVMDebugInfoMSF.a  lib/libMLIRAffineTransforms.a  lib/libMLIRAsyncTransforms.a  lib/libMLIRGPU.a  lib/libMLIRAsync.a  lib/libMLIRDLTI.a  lib/libMLIRLLVMToLLVMIRTranslation.a  lib/libMLIRNVVMIR.a  lib/libMLIRROCDLIR.a  lib/libMLIRVectorToLLVM.a  lib/libMLIRArmNeon.a  lib/libMLIRArmSVETransforms.a  lib/libMLIRArmSVE.a  lib/libMLIRAMXTransforms.a  lib/libMLIRAMX.a  lib/libMLIRTargetLLVMIRExport.a  lib/libMLIRLLVMIRTransforms.a  lib/libMLIRTranslation.a  lib/libMLIRMathTransforms.a  lib/libMLIRMemRefTransforms.a  lib/libMLIROpenACC.a  lib/libMLIROpenMP.a  lib/libMLIRSDBM.a  lib/libMLIRShapeOpsTransforms.a  lib/libMLIRShape.a  lib/libMLIRSparseTensorTransforms.a  lib/libMLIRLinalgTransforms.a  lib/libMLIRComplex.a  lib/libMLIRLinalgAnalysis.a  lib/libMLIRLinalgUtils.a  lib/libMLIRSCFTransforms.a  lib/libMLIRVectorToSCF.a  lib/libMLIRSparseTensor.a  lib/libMLIRSPIRVModuleCombiner.a  lib/libMLIRSPIRVTransforms.a  lib/libMLIRSPIRVConversion.a  lib/libMLIRSPIRVUtils.a  lib/libMLIRSPIRV.a  lib/libMLIRStandardOpsTransforms.a  lib/libMLIRTensorTransforms.a  lib/libMLIRTosaTransforms.a  lib/libMLIRX86VectorTransforms.a  lib/libMLIRX86Vector.a  lib/libMLIRStandardToLLVM.a  lib/libMLIRLLVMIR.a  lib/libLLVMBitWriter.a  lib/libLLVMAsmParser.a  lib/libMLIRMath.a  lib/libMLIRTosaTestPasses.a  lib/libMLIRTosa.a  lib/libMLIRQuant.a  lib/libMLIRAffineToStandard.a  lib/libMLIRSCFToStandard.a  lib/libMLIRTransforms.a  lib/libMLIRVector.a  lib/libMLIRAffineUtils.a  lib/libMLIRTransformUtils.a  lib/libMLIRLoopAnalysis.a  lib/libMLIRPresburger.a  lib/libMLIRRewrite.a  lib/libMLIRPDLToPDLInterp.a  lib/libMLIRPDLInterp.a  lib/libMLIRPDL.a  lib/libMLIRPass.a  lib/libMLIRAnalysis.a  lib/libMLIRLinalg.a  lib/libMLIRAffine.a  lib/libMLIRDialectUtils.a  lib/libMLIRParser.a  lib/libMLIRSCF.a  lib/libMLIRMemRef.a  lib/libMLIRMemRefUtils.a  lib/libMLIRDialect.a  lib/libMLIRStandard.a  lib/libMLIRTensor.a  lib/libMLIRViewLikeInterface.a  lib/libMLIRCastInterfaces.a  lib/libMLIRVectorInterfaces.a  lib/libMLIRLoopLikeInterface.a  lib/libMLIRSideEffectInterfaces.a  lib/libMLIRDataLayoutInterfaces.a  lib/libMLIRInferTypeOpInterface.a  lib/libMLIRCallInterfaces.a  lib/libMLIRControlFlowInterfaces.a  lib/libMLIRCopyOpInterface.a  lib/libMLIRIR.a  lib/libMLIRSupport.a  lib/libLLVMFrontendOpenACC.a  lib/libLLVMFrontendOpenMP.a  lib/libLLVMTransformUtils.a  lib/libLLVMAnalysis.a  lib/libLLVMObject.a  lib/libLLVMMCParser.a  lib/libLLVMMC.a  lib/libLLVMDebugInfoCodeView.a  lib/libLLVMBitReader.a  lib/libLLVMTextAPI.a  lib/libclangDriver.a  lib/libLLVMOption.a  lib/libLLVMProfileData.a  lib/libLLVMCore.a  lib/libLLVMRemarks.a  lib/libLLVMBitstreamReader.a  lib/libLLVMBinaryFormat.a  lib/libclangBasic.a  lib/libLLVMSupport.a  -lrt  -ldl  -lpthread  -lm  /usr/lib64/libtinfo.so  lib/libLLVMDemangle.a && :
lib/libflangFrontendTool.a(ExecuteCompilerInvocation.cpp.o): In function `Fortran::frontend::CreateFrontendBaseAction(Fortran::frontend::CompilerInstance&)':
ExecuteCompilerInvocation.cpp:(.text._ZN7Fortran8frontendL24CreateFrontendBaseActionERNS0_16CompilerInstanceE+0x35d): undefined reference to `vtable for Fortran::frontend::DebugDumpAllAction'
collect2: error: ld returned 1 exit status
[4438/4439] Generating ../../../../include/flang/ieee_arithmetic.mod
ninja: build stopped: subcommand failed.

[PeteSteinfeld]

Stay tuned a little, I think I spotted the problem ...

[PeteSteinfeld]

Looks great! Thanks for doing this.

[awarzynski]

Add missing test (forgot to add when making the submission)

[klausler]

The non-shared-library build of the current llvm-project/main tree fails on this new test; it looks as if gfortran is being invoked and passed an option that it does not recognize. The shared library build passes all tests.

[PeteSteinfeld]

In D104305#2822589, @klausler wrote:

The non-shared-library build of the current llvm-project/main tree fails on this new test; it looks as if gfortran is being invoked and passed an option that it does not recognize. The shared library build passes all tests.

Weird. It worked for me. Note that, if you're doing an out-of-tree build, you need to do a "make install" in your main build. Then, in your out-of-tree build, you need to reference the directories from the "install" rather than the "build" directories. Here's the cmake command I used for my out-of-tree build:

cmake \
  -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLVM_EXTERNAL_LIT=/local/home/psteinfeld/main/clean/build/bin/llvm-lit \
  -DLLVM_LIT_ARGS=-v \
  -DFLANG_ENABLE_WERROR=On \
  -DCMAKE_C_COMPILER=/home/sw/thirdparty/gcc/gcc-9.3.0/linux86-64/bin/gcc \
  -DCMAKE_CXX_COMPILER=/home/sw/thirdparty/gcc/gcc-9.3.0/linux86-64/bin/g++ \
  -DCMAKE_CXX_STANDARD=17 \
  -DLLVM_TARGETS_TO_BUILD=host \
  -DLLVM_ENABLE_ZLIB=OFF \
  -DLLVM_ENABLE_ASSERTIONS=ON \
  -DCMAKE_CXX_LINK_FLAGS="-Wl,-rpath,$LD_LIBRARY_PATH" \
  -DLLVM_MAIN_SRC_DIR=/local/home/psteinfeld/main/clean/llvm \
  -DCLANG_DIR=/local/home/psteinfeld/main/clean/install/lib/cmake/clang \
  -DLLVM_DIR=/local/home/psteinfeld/main/clean/install/lib/cmake/llvm \
  -DMLIR_DIR=/local/home/psteinfeld/main/clean/install/lib/cmake/mlir \
  ..

[klausler]

No; I'm hitting this problem with an in-tree static build of llvm-project/main. Weird.

[PeteSteinfeld]

For what it's worth, here's my cmake command that results in a successful static full build:

cmake -G Ninja ../llvm \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLVM_LIT_ARGS=-v \
  -DFLANG_ENABLE_WERROR=On \
  -DCMAKE_C_COMPILER=/home/sw/thirdparty/gcc/gcc-9.3.0/linux86-64/bin/gcc \
  -DCMAKE_CXX_COMPILER=/home/sw/thirdparty/gcc/gcc-9.3.0/linux86-64/bin/g++ \
  -DCMAKE_CXX_STANDARD=17 \
  -DLLVM_TARGETS_TO_BUILD=host \
  -DLLVM_ENABLE_ZLIB=OFF \
  -DLLVM_ENABLE_ASSERTIONS=ON \
  -DCMAKE_CXX_LINK_FLAGS="-Wl,-rpath,$LD_LIBRARY_PATH" \
  -DLLVM_INSTALL_UTILS=On \
  -DLLVM_ENABLE_PROJECTS="clang;mlir;flang" \
  -DCMAKE_INSTALL_PREFIX=$INSTALLDIR

[awarzynski]

Apologies, this is my fault. Should be fixed: https://github.com/llvm/llvm-project/commit/46446e398b5202303280d9e18676c07529f1d0b0. I pushed it without a review as it felt rather straightforward and I wanted to make sure that you are unblocked.

If for whatever reason your issue is still not resolved, please revert and I will fix it once I'm back online.

clang/include/clang/Driver/Options.td

 def fdebug_dump_parse_tree : Flag<["-"], "fdebug-dump-parse-tree">, Group<Action_Group>,
   HelpText<"Dump the parse tree">,
   DocBrief<[{Run the Parser and the semantic checks, and then output the
 parse tree.}]>;
 def fdebug_dump_parse_tree_no_sema : Flag<["-"], "fdebug-dump-parse-tree-no-sema">, Group<Action_Group>,
   HelpText<"Dump the parse tree (skips the semantic checks)">,
   DocBrief<[{Run the Parser and then output the parse tree. Semantic
 checks are disabled.}]>;
+def fdebug_dump_all : Flag<["-"], "fdebug-dump-all">, Group<Action_Group>,
+  HelpText<"Dump symbols and the parse tree after the semantic checks">;
 def fdebug_dump_provenance : Flag<["-"], "fdebug-dump-provenance">, Group<Action_Group>,
   HelpText<"Dump provenance">;
 def fdebug_dump_parsing_log : Flag<["-"], "fdebug-dump-parsing-log">, Group<Action_Group>,
   HelpText<"Run instrumented parse and dump the parsing log">;
 def fdebug_measure_parse_tree : Flag<["-"], "fdebug-measure-parse-tree">, Group<Action_Group>,
   HelpText<"Measure the parse tree">;
 def fdebug_pre_fir_tree : Flag<["-"], "fdebug-pre-fir-tree">, Group<Action_Group>,
   HelpText<"Dump the pre-FIR tree">;

flang/include/flang/Frontend/FrontendActions.h

 class DebugDumpSymbolsAction : public PrescanAndSemaAction {
   void ExecuteAction() override;
 };
 
 class DebugDumpParseTreeAction : public PrescanAndSemaAction {
   void ExecuteAction() override;
 };
 
+class DebugDumpAllAction : public PrescanAndSemaAction {
+  void ExecuteAction() override;
+};
+
 class DebugPreFIRTreeAction : public PrescanAndSemaAction {
   void ExecuteAction() override;
 };
 
 class GetDefinitionAction : public PrescanAndSemaAction {
   void ExecuteAction() override;
 };
 

flang/include/flang/Frontend/FrontendOptions.h

enum ActionKind {
   DebugUnparseWithSymbols,
 
   /// Parse, run semantics and then output symbols from semantics
   DebugDumpSymbols,
 
   /// Parse, run semantics and then output the parse tree
   DebugDumpParseTree,
 
+  /// Parse, run semantics and then output the parse tree and symbols
+  DebugDumpAll,
+
   /// Parse and then output the parse tree, skip the semantic checks
   DebugDumpParseTreeNoSema,
 
   /// Dump provenance
   DebugDumpProvenance,
 
   /// Parse then output the parsing log
   DebugDumpParsingLog,

flang/lib/Frontend/CompilerInvocation.cpp

case clang::driver::options::OPT_fdebug_unparse_with_symbols:
       opts.programAction_ = DebugUnparseWithSymbols;
       break;
     case clang::driver::options::OPT_fdebug_dump_symbols:
       opts.programAction_ = DebugDumpSymbols;
       break;
     case clang::driver::options::OPT_fdebug_dump_parse_tree:
       opts.programAction_ = DebugDumpParseTree;
       break;
+    case clang::driver::options::OPT_fdebug_dump_all:
+      opts.programAction_ = DebugDumpAll;
+      break;
     case clang::driver::options::OPT_fdebug_dump_parse_tree_no_sema:
       opts.programAction_ = DebugDumpParseTreeNoSema;
       break;
     case clang::driver::options::OPT_fdebug_dump_provenance:
       opts.programAction_ = DebugDumpProvenance;
       break;
     case clang::driver::options::OPT_fdebug_dump_parsing_log:
       opts.programAction_ = DebugDumpParsingLog;

flang/lib/Frontend/FrontendActions.cpp

if (!tables.schemata) {
     ci.diagnostics().Report(DiagID);
     llvm::errs() << "\n";
   }
 
   // Dump symbols
   semantics.DumpSymbols(llvm::outs());
 }
 
+void DebugDumpAllAction::ExecuteAction() {
+  CompilerInstance &ci = this->instance();
+
+  // Dump parse tree
+  auto &parseTree{instance().parsing().parseTree()};
+  Fortran::parser::AnalyzedObjectsAsFortran asFortran =
+      Fortran::frontend::getBasicAsFortran();
+  llvm::outs() << "========================";
+  llvm::outs() << " Flang: parse tree dump ";
+  llvm::outs() << "========================\n";
+  Fortran::parser::DumpTree(llvm::outs(), parseTree, &asFortran);
+
+  auto &semantics = this->semantics();
+  auto tables{Fortran::semantics::BuildRuntimeDerivedTypeTables(
+      instance().invocation().semanticsContext())};
+  // The runtime derived type information table builder may find and report
+  // semantic errors. So it is important that we report them _after_
+  // BuildRuntimeDerivedTypeTables is run.
+  reportFatalSemanticErrors(
+      semantics, this->instance().diagnostics(), GetCurrentFileOrBufferName());
+
+  if (!tables.schemata) {
+    unsigned DiagID =
+        ci.diagnostics().getCustomDiagID(clang::DiagnosticsEngine::Error,
+            "could not find module file for __fortran_type_info");
+    ci.diagnostics().Report(DiagID);
+    llvm::errs() << "\n";
+  }
+
+  // Dump symbols
+  llvm::outs() << "=====================";
+  llvm::outs() << " Flang: symbols dump ";
+  llvm::outs() << "=====================\n";
+  semantics.DumpSymbols(llvm::outs());
+}
+
 void DebugDumpParseTreeNoSemaAction::ExecuteAction() {
   auto &parseTree{instance().parsing().parseTree()};
   Fortran::parser::AnalyzedObjectsAsFortran asFortran =
       Fortran::frontend::getBasicAsFortran();
 
   // Dump parse tree
   Fortran::parser::DumpTree(llvm::outs(), parseTree, &asFortran);
 }

flang/lib/FrontendTool/ExecuteCompilerInvocation.cpp

case DebugDumpSymbols:
     return std::make_unique<DebugDumpSymbolsAction>();
     break;
   case DebugDumpParseTree:
     return std::make_unique<DebugDumpParseTreeAction>();
     break;
   case DebugDumpParseTreeNoSema:
     return std::make_unique<DebugDumpParseTreeNoSemaAction>();
     break;
+  case DebugDumpAll:
+    return std::make_unique<DebugDumpAllAction>();
+    break;
   case DebugDumpProvenance:
     return std::make_unique<DebugDumpProvenanceAction>();
     break;
   case DebugDumpParsingLog:
     return std::make_unique<DebugDumpParsingLogAction>();
     break;
   case DebugMeasureParseTree:
     return std::make_unique<DebugMeasureParseTreeAction>();

flang/test/Driver/driver-help.f90

 ! HELP-FC1-NEXT:OPTIONS:
 ! HELP-FC1-NEXT: -cpp                   Enable predefined and command line preprocessor macros
 ! HELP-FC1-NEXT: -D <macro>=<value>     Define <macro> to <value> (or 1 if <value> omitted)
 ! HELP-FC1-NEXT: -emit-obj Emit native object files
 ! HELP-FC1-NEXT: -E                     Only run the preprocessor
 ! HELP-FC1-NEXT: -falternative-parameter-statement
 ! HELP-FC1-NEXT: Enable the old style PARAMETER statement
 ! HELP-FC1-NEXT: -fbackslash            Specify that backslash in string introduces an escape character
+! HELP-FC1-NEXT: -fdebug-dump-all       Dump symbols and the parse tree after the semantic checks
 ! HELP-FC1-NEXT: -fdebug-dump-parse-tree-no-sema
 ! HELP-FC1-NEXT:                        Dump the parse tree (skips the semantic checks)
 ! HELP-FC1-NEXT: -fdebug-dump-parse-tree Dump the parse tree
 ! HELP-FC1-NEXT: -fdebug-dump-parsing-log
 ! HELP-FC1-NEXT:                   Run instrumented parse and dump the parsing log
 ! HELP-FC1-NEXT: -fdebug-dump-provenance Dump provenance
 ! HELP-FC1-NEXT: -fdebug-dump-symbols    Dump symbols after the semantic analysis
 ! HELP-FC1-NEXT: -fdebug-measure-parse-tree

flang/test/Driver/dump-all.f90

+!----------
+! RUN lines
+!----------
+! RUN: %flang_fc1 -fdebug-dump-all %s 2>&1 | FileCheck %s
+
+!----------------
+! EXPECTED OUTPUT
+!----------------
+! CHECK: Flang: parse tree dump
+! CHECK: Flang: symbols dump
+
+!-------
+! INPUT
+!-------
+parameter(i=1)
+integer :: j
+end program