[Flang] Rework the overview document

[PeteSteinfeld]

Summary

I brought the overview document up to date and added information for
most compilation phases to dump out the reeults of the phase.

[tschuett]

flang/docs/Overview.md

23

decorated parse tree

[PeteSteinfeld]

flang/docs/Overview.md

23

Good catch! I'll fix it.

[tschuett]

flang/docs/Overview.md

34

described

187–189

clang or flang?

[PeteSteinfeld]

Thanks for the feedback. You're a prince!

flang/docs/Overview.md

34

Another good catch. I'll fix this, too.

187–189

This sentence is not clear. I plan to change it as follows:
"Once the LLVM IR is created the flang driver invokes LLVM's exising infrastructure to generate object code and invoke a linker."

[tschuett]

flang/docs/Overview.md

187–189

My bad. The name of the executable is flang, but it links against the clang driver library.

[PeteSteinfeld]

Responding to @tschuett's excellent feedback.

[sscalpone]

For the information that is being removed, e.g. "do concurrent" or the description of ResolveNames -- is this obsolete or repeated in another document?

[kiranchandramohan]

Looks Good. Thanks for updating this document. A few comments or questions.

flang/docs/Overview.md

58–59

Preprocessing also happens, I guess. Is it worth calling it out?

61–63

Nit: The information will also be used for generating optimization reports, debug information etc.

68

Nit: spelling : -fdebug-dump-provenance

154

We lower to FIR and other MLIR dialects (like Arith, Math, Func, OpenMP, OpenACC). But yes, From a Flang compiler perspective FIR is the backbone. With the HLFIR changes going on, we will lower to that as well.

178

Nit: Not all are optimisations, some are transformations that just create a lower-level representation.

182

Is this a bug that this option does not work with fc1 ?

[kiranchandramohan]

flang/docs/Overview.md

182

I think the -S option is required for this option and the one below to avoid invoking the linker.

[kiranchandramohan]

flang/docs/Overview.md

92–98

Module files also?

138–139

Should the above two be listed as outputs of the Semantics processing?

Also, isn't there a type table? (for derived types)

[tschuett]

Will you use the LLVM remarks facilities for optimisation reports?

[kiranchandramohan]

In D140241#4007759, @tschuett wrote:

Will you use the LLVM remarks facilities for optimisation reports?

We will reuse wherever possible. If you meant the -fsave-optimization-record option, then we will support it. In the beginning, there will be only optimisation info from LLVM, but I believe we can support FIR/MLIR optimisation info as well.

[PeteSteinfeld]

In D140241#4004037, @sscalpone wrote:

For the information that is being removed, e.g. "do concurrent" or the description of ResolveNames -- is this obsolete or repeated in another document?

This information is repeated in other documents. Still, it would be good to reference them from the overview. I'll add such references.

flang/docs/Overview.md

58–59

You're right! I'll add that.

61–63

Good point. I'll add references to those.

68

Good catch. Thanks!

92–98

Yes. Thanks for pointing this out. I'll add that.

138–139

Good points. I'll make some changes.

178

Good point. I'll clarify.

182

I'm not sure if it's a bug or not.

182

Good catch. I'll add "-S".

[PeteSteinfeld]

In D140241#4007759, @tschuett wrote:

Will you use the LLVM remarks facilities for optimisation reports?

I'm not familiar with the LLVM remarks facilities.

[tschuett]

In D140241#4008549, @PeteSteinfeld wrote:

In D140241#4007759, @tschuett wrote:

Will you use the LLVM remarks facilities for optimisation reports?

I'm not familiar with the LLVM remarks facilities.

You can put remarks about hit or miss optimisations into the binary.

[PeteSteinfeld]

Responding to feedback from Steve and Kiran.

flang/docs/Overview.md

 <!--===- docs/Overview.md 
   
    Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
    See https://llvm.org/LICENSE.txt for license information.
    SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
   
 -->
 
-# Intro
-This document goes briefly over compiler phases in Flang. It focuses on the
-internal implementation and as such, it is intended for Flang developers rather
-than end-users.
-
 # Overview of Compiler Phases
 
 ```eval_rst
 .. contents::
    :local:
 ```
+The Flang compiler transforms Fortran source code into an executable file. 
+This transformation proceeds in three high level phases -- analysis, lowering,
+and code generation/linking.
+
+The first high level phase (analysis) transforms Fortran source code into a
+decorated parse tree and a symbol table.  During this phase, all user
+related errors are detected and reported.
+
+The second high level phase (lowering), changes the decorated parse tree and
+symbol table into the Fortran Intermediate Representation (FIR), which is a
+dialect of LLVM's Multi-Level Intermediate Representation or MLIR.  It then
+runs a series of passes on the FIR code which verify its validity, perform a
+series of optimizations, and finally transform it into LLVM's Intermediate
+Representation, or LLVM IR
+
+The third high level phase generates machine code and invokes a linker to
+produce an executable file.
+
+This document describes the first two high level phases.  Each of these is
+described in more detailed phases.
 
-Each phase produces either correct output or fatal errors.
+Each detailed phase is described -- its inputs and outputs along with how to
+produce a readable version of the outputs.
+
+Each detailed phase produces either correct output or fatal errors.
+
+# Analysis
+
+This high level phase validates that the program is correct and creates all of
+the information needed for lowering.
 
 ## Prescan and Preprocess
 
-See: [Preprocessing.md](Preprocessing.md).
+See [Preprocessing.md](Preprocessing.md).
 
 **Input:** Fortran source and header files, command line macro definitions,
   set of enabled compiler directives (to be treated as directives rather than
   comments).
 
 **Output:**
 - A "cooked" character stream: the entire program as a contiguous stream of
   normalized Fortran source.
   Extraneous whitespace and comments are removed (except comments that are
-  compiler directives that are not disabled) and case is normalized.
+  compiler directives that are not disabled) and case is normalized.  Also,
+  directives are processed and macros expanded.
 - Provenance information mapping each character back to the source it came from.
-  This is used in subsequent phases to issue errors messages that refer to source locations.
+  This is used in subsequent phases that need source locations.  This includes
+  error messages, optimization reports, and debugging information.
 
 **Entry point:** `parser::Parsing::Prescan`
 
-**Command:** `flang-new -fc1 -E src.f90` dumps the cooked character stream
+**Commands:** 
+ - `flang-new -fc1 -E src.f90` dumps the cooked character stream
+ - `flang-new -fc1 -fdebug-dump-provenance src.f90` dumps provenance
+   information
+
+## Parsing
+
+**Input:** Cooked character stream
+
+**Output:** A parse tree for each Fortran program unit in the source code
+representing a syntactically correct program, rooted at the program unit.  See:
+[Parsing.md](Parsing.md) and [ParserCombinators.md](ParserCombinators.md).
+
+**Entry point:** `parser::Parsing::Parse`
 
-## Parse
+**Commands:**
+  - `flang-new -fc1 -fdebug-dump-parse-tree-no-sema src.f90` dumps the parse tree
+  - `flang-new -fc1 -fdebug-unparse src.f90` converts the parse tree to normalized Fortran
+  - `flang-new -fc1 -fdebug-dump-parsing-log src.f90` runs an instrumented parse and dumps the log
+  - `flang-new -fc1 -fdebug-measure-parse-tree src.f90` measures the parse tree
 
-**Input:** Cooked character stream.
+## Semantic processing
 
-**Output:** A parse tree representing a syntactically correct program,
-  rooted at a `parser::Program`.
-  See: [Parsing.md](Parsing.md) and [ParserCombinators.md](ParserCombinators.md).
+**Input:** the parse tree, the cooked character stream, and provenance
+information
 
-**Entry point:** `parser::Parsing::Parse`
+**Output:** 
+* a symbol table
+* modified parse tree
+* module files, (see: [ModFiles.md](ModFiles.md))
+* the intrinsic procedure table
+* the target characteristics
+* the runtime derived type derived type tables (see: [RuntimeTypeInfo.md](RuntimeTypeInfo.md))
 
-**Command:**
-  - `flang-new -fc1 -fdebug-dump-parse-tree src.f90` dumps the parse tree
-  - `flang-new -fc1 -fdebug-unparse src.f90` converts the parse tree to normalized Fortran
+**Entry point:** `semantics::Semantics::Perform`
 
-## Validate Labels and Canonicalize Do Statements
+For more detail on semantic analysis, see: [Semantics.md](Semantics.md).
+Semantic processing performs several tasks: 
+* validates labels, see: [LabelResolution.md](LabelResolution.md).
+* canonicalizes DO statements, 
+* canonicalizes OpenACC and OpenMP code
+* resolves names, building a tree of scopes and symbols
+* rewrites the parse tree to correct parsing mistakes (when needed) once semantic information is available to clarify the program's meaning
+* checks the validity of declarations
+* analyzes expressions and statements, emitting error messages where appropriate
+* creates module files if the source code contains modules, 
+  see [ModFiles.md](ModFiles.md).
 
-**Input:** Parse tree.
+In the course of semantic analysis, the compiler:
+* creates the symbol table
+* decorates the parse tree with semantic information (such as pointers into the symbol table)
+* creates the intrinsic procedure table
+* folds constant expressions
 
-**Output:** The parse tree with label constraints and construct names checked,
-  and each `LabelDoStmt` converted to a `NonLabelDoStmt`.
-  See: [LabelResolution.md](LabelResolution.md).
+At the end of semantic processing, all validation of the user's program is complete.  This is the last detailed phase of analysis processing.
 
-**Entry points:** `semantics::ValidateLabels`, `parser::CanonicalizeDo`
+**Commands:**
+  - `flang-new -fc1 -fdebug-dump-parse-tree src.f90` dumps the parse tree after semantic analysis
+  - `flang-new -fc1 -fdebug-dump-symbols src.f90` dumps the symbol table
+  - `flang-new -fc1 -fdebug-dump-all src.f90` dumps both the parse tree and the symbol table
 
-## Resolve Names
+# Lowering
 
-**Input:** Parse tree (without `LabelDoStmt`) and `.mod` files from compilation
-  of USEd modules.
+Lowering takes the parse tree and symbol table produced by analysis and
+produces LLVM IR.
 
-**Output:**
-- Tree of scopes populated with symbols and types
-- Parse tree with some refinements:
-  - each `parser::Name::symbol` field points to one of the symbols
-  - each `parser::TypeSpec::declTypeSpec` field points to one of the types
-  - array element references that were parsed as function references or
-    statement functions are corrected
+## Create the lowering bridge
+
+**Inputs:** 
+  - the parse tree
+  - the symbol table
+  - The default KINDs for intrinsic types (specified by default or command line option)
+  - The intrinsic procedure table (created in semantics processing)
+  - The target characteristics (created during semantics processing)
+  - The cooked character stream
+  - The target triple -- CPU type, vendor, operating system
+  - The mapping between Fortran KIND values to FIR KIND values
+
+The lowering bridge is a container that holds all of the information needed for lowering.
+
+**Output:** A container with all of the information needed for lowering
 
-**Entry points:** `semantics::ResolveNames`, `semantics::RewriteParseTree`
+**Entry point:** lower::LoweringBridge::create
 
-**Command:** `flang-new -fc1 -fdebug-dump-symbols src.f90` dumps the
-  tree of scopes and symbols in each scope
+## Initial lowering
 
-## Check DO CONCURRENT Constraints
+**Input:** the lowering bridge
 
-**Input:** Parse tree with names resolved.
+**Output:** A Fortran IR (FIR) representation of the program.
 
-**Output:** Parse tree with semantically correct DO CONCURRENT loops.
+**Entry point:** `lower::LoweringBridge::lower`
 
-## Write Module Files
+The compiler then takes the information in the lowering bridge and creates a
+pre-FIR tree or PFT.  The PFT is a list of programs and modules.  The programs
+and modules contain lists of function-like units.  The function-like units
+contain a list of evaluations.  All of these contain pointers back into the
+parse tree.  The compiler walks the PFT generating FIR.
 
-**Input:** Parse tree with names resolved.
+**Commands:**
+  - `flang-new -fc1 -fdebug-dump-pft src.f90` dumps the pre-FIR tree
+  - `flang-new -fc1 -emit-mlir src.f90` dumps the FIR to the files src.mlir
 
-**Output:** For each module and submodule, a `.mod` file containing a minimal
-  Fortran representation suitable for compiling program units that depend on it.
-  See [ModFiles.md](ModFiles.md).
+## Transformation passes
 
-## Analyze Expressions and Assignments
+**Input:** initial version of the FIR code
 
-**Input:** Parse tree with names resolved.
+**Output:** An LLVM IR representation of the program
 
-**Output:** Parse tree with `parser::Expr::typedExpr` filled in and semantic
-  checks performed on all expressions and assignment statements.
+**Entry point:** `mlir::PassManager::run`
 
-**Entry points**: `semantics::AnalyzeExpressions`, `semantics::AnalyzeAssignments`
+The compiler then runs a series of passes over the FIR code.  The first is a
+verification pass.  It's followed by a series of transformation passes that
+perform various optimizations and transformations.  The final pass creates an
+LLVM IR representation of the program.
 
-## Produce the Intermediate Representation
+**Commands:**
+  - `flang-new -mmlir --mlir-print-ir-after-all -S src.f90` dumps the FIR code after each pass to standard error
+  - `flang-new -fc1 -emit-llvm src.f90` dumps the LLVM IR to src.ll
 
-**Input:** Parse tree with names and labels resolved.
+# Object code generation and linking
 
-**Output:** An intermediate representation of the executable program.
-  See [FortranIR.md](FortranIR.md).
+After the LLVM IR is created, the flang driver invokes LLVM's existing
+infrastructure to generate object code and invoke a linker to create the
+executable file.