[flang][driver] Make `flang-new` always generate run-time type info

[awarzynski]

Summary

Currently, the driver generates the tables with "run-time type
information for derived types" only when specific actions are run.
However, the corresponding data might be required by the subsequent
compilation stages (e.g. lowering, code-gen) and should be generated
unconditionally. Note that this is only possible once the semantic
checks have been run.

Note that when generating these tables, extra semantic errors might be
generated. The driver will always report these and in most cases such
semantic errors will cause the driver to exit immediately. The only
exception are actions inheriting from PrescanAndSemaDebugAction.
Currently, there's only one such action: DebugDumpAllAction
(corresponds to -fdebug-dump-all command-line flag). I've updated the
comments for this action to clarify this.

This change will mostly affect lowering, which currently is only
available for most basic examples (e.g. empty programs). I wasn't able
to find a working case that would demonstrate the new behaviour. I
hope that this change is straightforward enough and am submitting it
without a test.

[awarzynski]

For a bit more context, after https://reviews.llvm.org/D119555 landed, SNAP no longer builds with flang-new (see the instructions here). This patch fixes that, but we can only be verified with fir-dev. I wasn't sure how to go about fixing this, but IIUC, it's preferred to create initial patches here and then cherry-pick them onto fir-dev. Also, this change is relatively straightforward (I hope!). Let me know what you think!

[jeanPerier]

flang/include/flang/Frontend/FrontendActions.h

138

semantinc -> semantic

flang/lib/Frontend/FrontendAction.cpp

191

@klausler , do can you confirm the runtime type info generation will not generate bogus semantics errors ?

194

I wonder if you should not always complain here if !ci.getRtTyTables().schemata. Currently, it seems DebugDumpSymbolsAction and DebugDumpSymbolsAction would complain. I think at least the regular
compiler workflows should complain if it was not able to generate those. Better than to wait for more cryptic link time errors.

[awarzynski]

flang/lib/Frontend/FrontendAction.cpp

194

Good point - this would also mean better code re-use. Let me update!

[awarzynski]

Move the schemata check to aGenerateRtTypeTables`

Thanks for taking a look @jeanPerier!

[klausler]

flang/lib/Frontend/FrontendAction.cpp

191

Runtime derived type description generation can emit a fatal error in the case that a specification expression for a component bound or type parameter does not resolve to a constant or LEN type parameter.

[jeanPerier]

flang/lib/Frontend/FrontendAction.cpp

203

Mmh. I gave some bad advice... It looks like the builds are failing because this error is reported when compiling the builtin modules before __fortran_type_info.f90 was compiled to create the module (this includes the bootstrapping case of __fortran_type_info.f90). I am not sure how solide it would be to determine if this is a builtin module compilation and that this message should not be emitted.

[awarzynski]

flang/lib/Frontend/FrontendAction.cpp

203

I don't have a good solution. We could either:

1. disable this diagnostic when __fortran_builtins is being compiled (that's the only dependency from __fortran_type_info.f90 that needs to be processed before __fortran_type_info.f90), or
2. implement a dedicated compiler flag to disable this particular diagnostic and use it in CMake for when __fortran_builtins is being compiled.

I'm not sure how to achieve 1. in a reliable way (i.e. how to identify that it's indeed the builtin module is being processed?). As for 2., I really dislike the idea of adding a dedicated compiler flag to work around a very specific corner case. Any other suggestions?

I think that moving this bit back to DebugDumpSymbolsAction::ExecuteAction and DebugDumpAllAction::ExecuteAction would be a good compromise for now. We know that these run-time tables should be generated, but we cannot reliably check whether schemata was indeed set. Not yet.

[awarzynski]

Move back the if (!ci.getRtTyTables().schemata) checks. As discussed inline, this condition is false when compiling __fortran_builtins.f90 and there's no mechanism (yet) to apply this selectively.

[PeteSteinfeld]

All builds and tests correctly and looks good.

flang/include/flang/Frontend/CompilerInstance.h

 //===-- CompilerInstance.h - Flang Compiler Instance ------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 #ifndef LLVM_FLANG_FRONTEND_COMPILERINSTANCE_H
 #define LLVM_FLANG_FRONTEND_COMPILERINSTANCE_H
 
 #include "flang/Frontend/CompilerInvocation.h"
 #include "flang/Frontend/FrontendAction.h"
 #include "flang/Frontend/PreprocessorOptions.h"
 #include "flang/Parser/parsing.h"
 #include "flang/Parser/provenance.h"
+#include "flang/Semantics/runtime-type-info.h"
 #include "flang/Semantics/semantics.h"
 #include "llvm/Support/raw_ostream.h"
 
 namespace Fortran::frontend {
 
 /// Helper class for managing a single instance of the Flang compiler.
 ///
 /// This class serves two purposes:
class CompilerInstance {
   std::shared_ptr<Fortran::parser::AllSources> allSources_;
 
   std::shared_ptr<Fortran::parser::AllCookedSources> allCookedSources_;
 
   std::shared_ptr<Fortran::parser::Parsing> parsing_;
 
   std::unique_ptr<Fortran::semantics::Semantics> semantics_;
 
+  std::unique_ptr<Fortran::semantics::RuntimeDerivedTypeTables> rtTyTables_;
+
   /// The stream for diagnostics from Semantics
   llvm::raw_ostream *semaOutputStream_ = &llvm::errs();
 
   /// The stream for diagnostics from Semantics if owned, otherwise nullptr.
   std::unique_ptr<llvm::raw_ostream> ownedSemaOutputStream_;
 
   /// The diagnostics engine instance.
   llvm::IntrusiveRefCntPtr<clang::DiagnosticsEngine> diagnostics_;
public:
 
   Fortran::semantics::Semantics &semantics() { return *semantics_; }
   const Fortran::semantics::Semantics &semantics() const { return *semantics_; }
 
   void SetSemantics(std::unique_ptr<Fortran::semantics::Semantics> semantics) {
     semantics_ = std::move(semantics);
   }
 
+  void setRtTyTables(
+      std::unique_ptr<Fortran::semantics::RuntimeDerivedTypeTables> tables) {
+    rtTyTables_ = std::move(tables);
+  }
+
+  Fortran::semantics::RuntimeDerivedTypeTables &getRtTyTables() {
+    assert(rtTyTables_ && "Missing runtime derived type tables!");
+    return *rtTyTables_;
+  }
+
   /// }
   /// @name High-Level Operations
   /// {
 
   /// Execute the provided action against the compiler's
   /// CompilerInvocation object.
   /// \param act - The action to execute.
   /// \return - True on success.

flang/include/flang/Frontend/FrontendAction.h

protected:
   // True otherwise.
   bool RunPrescan();
   // Parse the current input file. Return False if fatal errors are reported,
   // True otherwise.
   bool RunParse();
   // Run semantic checks for the current input file. Return False if fatal
   // errors are reported, True otherwise.
   bool RunSemanticChecks();
+  // Generate run-time type information for derived types. This may lead to new
+  // semantic errors. Return False if fatal errors are reported, True
+  // otherwise.
+  bool GenerateRtTypeTables();
 
   // Report fatal semantic errors. Return True if present, false otherwise.
   bool reportFatalSemanticErrors();
 
   // Report fatal scanning errors. Return True if present, false otherwise.
   inline bool reportFatalScanningErrors() {
     return reportFatalErrors("Could not scan %0");
   }

flang/include/flang/Frontend/FrontendActions.h

 class PluginParseTreeAction : public PrescanAndSemaAction {
   void ExecuteAction() override = 0;
 };
 
 //===----------------------------------------------------------------------===//
 // PrescanAndSemaDebug Actions
 //
 // These actions will parse the input, run the semantic checks and execute
-// their actions regardless of whether any semantic errors are found.
+// their actions _regardless of_ whether any semantic errors have been found.
+// This can be useful when adding new languge feature and when you wish to
+// investigate compiler output (e.g. the parse tree) despite any semantic
+// errors.
+//
+// NOTE: Use with care and for development only!
 //===----------------------------------------------------------------------===//
 class PrescanAndSemaDebugAction : public FrontendAction {
 
   void ExecuteAction() override = 0;
   bool BeginSourceFileAction() override;
 };
 
 class DebugDumpAllAction : public PrescanAndSemaDebugAction {

flang/lib/Frontend/FrontendAction.cpp

bool FrontendAction::RunSemanticChecks() {
   }
 
   // Report the diagnostics from the semantic checks
   semantics.EmitMessages(ci.semaOutputStream());
 
   return true;
 }
 
+bool FrontendAction::GenerateRtTypeTables() {
+  instance().setRtTyTables(
+      std::make_unique<Fortran::semantics::RuntimeDerivedTypeTables>(
+          BuildRuntimeDerivedTypeTables(
+              instance().invocation().semanticsContext())));
+
+  // The runtime derived type information table builder may find additional
+  // semantic errors. Report them.
+  if (reportFatalSemanticErrors()) {
+    return false;
+  }
+
+  return true;
+}
+
 template <unsigned N>
 bool FrontendAction::reportFatalErrors(const char (&message)[N]) {
   if (!instance_->parsing().messages().empty() &&
       (instance_->invocation().warnAsErr() ||
           instance_->parsing().messages().AnyFatalError())) {
     const unsigned diagID = instance_->diagnostics().getCustomDiagID(
         clang::DiagnosticsEngine::Error, message);
     instance_->diagnostics().Report(diagID) << GetCurrentFileOrBufferName();
     instance_->parsing().messages().Emit(
         llvm::errs(), instance_->allCookedSources());
     return true;
   }
   return false;
 }

flang/lib/Frontend/FrontendActions.cpp

 //===----------------------------------------------------------------------===//
 bool PrescanAction::BeginSourceFileAction() { return RunPrescan(); }
 
 bool PrescanAndParseAction::BeginSourceFileAction() {
   return RunPrescan() && RunParse();
 }
 
 bool PrescanAndSemaAction::BeginSourceFileAction() {
-  return RunPrescan() && RunParse() && RunSemanticChecks();
+  return RunPrescan() && RunParse() && RunSemanticChecks() &&
+      GenerateRtTypeTables();
 }
 
 bool PrescanAndSemaDebugAction::BeginSourceFileAction() {
-  // Semantic checks are made to succeed unconditionally.
-  return RunPrescan() && RunParse() && (RunSemanticChecks() || true);
+  // This is a "debug" action for development purposes. To facilitate this, the
+  // semantic checks are made to succeed unconditionally to prevent this action
+  // from exiting early (i.e. in the presence of semantic errors). We should
+  // never do this in actions intended for end-users or otherwise regular
+  // compiler workflows!
+  return RunPrescan() && RunParse() && (RunSemanticChecks() || true) &&
+      (GenerateRtTypeTables() || true);
 }
 
 bool CodeGenAction::BeginSourceFileAction() {
   bool res = RunPrescan() && RunParse() && RunSemanticChecks();
   if (!res)
     return res;
 
   CompilerInstance &ci = this->instance();
Fortran::semantics::UnparseWithSymbols(
       llvm::outs(), parseTree, /*encoding=*/Fortran::parser::Encoding::UTF_8);
 
   // Report fatal semantic errors
   reportFatalSemanticErrors();
 }
 
 void DebugDumpSymbolsAction::ExecuteAction() {
   CompilerInstance &ci = this->instance();
-  auto &semantics = ci.semantics();
 
-  auto tables{Fortran::semantics::BuildRuntimeDerivedTypeTables(
-      instance().invocation().semanticsContext())};
-  // The runtime derived type information table builder may find and report
-  // semantic errors. So it is important that we report them _after_
-  // BuildRuntimeDerivedTypeTables is run.
-  reportFatalSemanticErrors();
-
-  if (!tables.schemata) {
+  if (!ci.getRtTyTables().schemata) {
     unsigned DiagID =
         ci.diagnostics().getCustomDiagID(clang::DiagnosticsEngine::Error,
             "could not find module file for __fortran_type_info");
     ci.diagnostics().Report(DiagID);
     llvm::errs() << "\n";
+    return;
   }
 
   // Dump symbols
-  semantics.DumpSymbols(llvm::outs());
+  ci.semantics().DumpSymbols(llvm::outs());
 }
 
 void DebugDumpAllAction::ExecuteAction() {
   CompilerInstance &ci = this->instance();
 
   // Dump parse tree
   auto &parseTree{instance().parsing().parseTree()};
   llvm::outs() << "========================";
   llvm::outs() << " Flang: parse tree dump ";
   llvm::outs() << "========================\n";
   Fortran::parser::DumpTree(
       llvm::outs(), parseTree, &ci.invocation().asFortran());
 
-  auto &semantics = ci.semantics();
-  auto tables{Fortran::semantics::BuildRuntimeDerivedTypeTables(
-      instance().invocation().semanticsContext())};
-  // The runtime derived type information table builder may find and report
-  // semantic errors. So it is important that we report them _after_
-  // BuildRuntimeDerivedTypeTables is run.
-  reportFatalSemanticErrors();
-
-  if (!tables.schemata) {
+  if (!ci.getRtTyTables().schemata) {
     unsigned DiagID =
         ci.diagnostics().getCustomDiagID(clang::DiagnosticsEngine::Error,
             "could not find module file for __fortran_type_info");
     ci.diagnostics().Report(DiagID);
     llvm::errs() << "\n";
+    return;
   }
 
   // Dump symbols
   llvm::outs() << "=====================";
   llvm::outs() << " Flang: symbols dump ";
   llvm::outs() << "=====================\n";
-  semantics.DumpSymbols(llvm::outs());
+  ci.semantics().DumpSymbols(llvm::outs());
 }
 
 void DebugDumpParseTreeNoSemaAction::ExecuteAction() {
   auto &parseTree{instance().parsing().parseTree()};
 
   // Dump parse tree
   Fortran::parser::DumpTree(
       llvm::outs(), parseTree, &this->instance().invocation().asFortran());