[Flang][driver] Update docs

[awarzynski]

NOTE: We cannot merge this just yet. First, we need add support for code-generation in LLVM Flang. This is currently being developed on fir-dev.

I've started drafting this to clarify and to justify the design. I'm sharing it here so that people have access to it while we are still implementing the basic functionality.

[tschuett]

I always liked the idea that the driver (clang, flang,swift) orchestrates the build. It uses various tools like e.g. compilers, linkers, assemblers, ... to build libraries, executables, ... (artefacts)

For historical reasons, there is only one clang binary. For swift I have swift, swiftc, swift-frontend, and swift-driver. Actually swiftc is a symlink to swift. Maybe give the flang frontend and the flang driver different names and use symlinks.

[richard.barton.arm]

flang/docs/FlangDriver.md

92

external ^tools like assemblers.

[tschuett]

flang/docs/FlangDriver.md

99

It is easier to understand, if talk about the driver and the frontend. frontend-driver is misleading.

[Leporacanthicus]

flang/docs/FlangDriver.md

93

"Nor does it need"?

[awarzynski]

flang/docs/FlangDriver.md

99

"frontend" can mean different things, though. For somebody from e.g. llvm-project/llvm (i.e. people working on the backend), probably both "llvm-project/clang" and "llvm-project/flang" are frontends (i.e. the entire sub-projects, with all the libraries and tools). For an "llvm-project/{flang|clang}" developer, it's probably just the core libraries (e.g. clangSema, clangParse, FortranParser or FortranSemantics).

I think that both perspectives are correct and simply depend on the point of reference. But it means that the term is ambiguous and I try to be careful about the context in which it is used. Also, "frontend" or "frontend tool"? The latter is more accurate (indeed, it's just a tool), but then which "frontend tool" does it refer to? There's already flang-new, flang-new -fc1, tco, bbc.

I really appreciate the suggestion! This is tricky. To me, "frontend driver" makes a lot of sense (i.e. it drives the frontend libraries), but I'm biased. Let's wait and see what others think!

[awarzynski]

Address comments from @richard.barton.arm and @Leporacanthicus

Thank you all for taking a look!

[PeteSteinfeld]

Thanks for working on this, Andrzej.

In addition to the inline comments, there are some things that I don't understand about the driver.

When invoking the compiler driver, I'll want to specify options to the frontend. How do I do that?

The document implies that the frontend options are specifically designed to be of interest to compiler developers and are potentially unstable. Neither one of these seems true to me.

As I hinted in some of my inline comments, my overall understanding of the drivers would be enhanced if I better understood the overall compilation process that they control. I think it would be good to include a brief overview of the compilation processes controlled by the compiler driver and the frontend driver.

flang/docs/FlangDriver.md

22–24

It would help me to understand the difference between the frontend driver and the compiler driver if I knew what the inputs and outputs of the compiler and frontend were. I assume that the inputs of the compiler are source code and module files and that its outputs are executables, libraries, and module files. I assume that the inputs of the frontend are the same as those of the compiler. I'm not sure about its outputs. Where do the tools bbc and tco fit into this picture? Are there other tools in the LLVM compilation tool chain that the compiler driver might invoke to do optimization or analysis?

28

Rather than "e.g.", I would say "other tools such as"

75

Should read "viewd as the driver".

90

There is no scanner in the compiler, so I don't think we should mention scanning.

91

I think that this should read "have to be".

99

In the rest of this document, "frontend driver" is spelled without a hyphen in the middle.

103

This should read "A similar model".

[awarzynski]

Address some of the comments from @PeteSteinfeld

[awarzynski]

Thank you for the review, Pete.

In D111573#3058546, @PeteSteinfeld wrote:

When invoking the compiler driver, I'll want to specify options to the frontend. How do I do that?

You can do this with -Xflang:

flang-new -Xflang -fdebug-dump-parse-tree input.f95

In the above ^^^ invocation, -fdebug-dump-parse-tree is forwarded to flang-new -fc1. Here are a few small experiments that illustrate this:

# Print the input file fist
$ cat input.f95
end program

# 1. Compiler driver: `-fdebug-dump-parse-tree` *is not* supported
$ bin/flang-new -fdebug-dump-parse-tree file.f95
flang-new: warning: argument unused during compilation: '-fdebug-dump-parse-tree'

# 2. Frontend driver: `-fdebug-dump-parse-tree` *is* supported
$ bin/flang-new -fc1 -fdebug-dump-parse-tree file.f95
Program -> ProgramUnit -> MainProgram
| SpecificationPart
| | ImplicitPart ->
| ExecutionPart -> Block
| EndProgramStmt ->

# 3. Compiler driver: `-fdebug-dump-parse-tree` is forwarded to `flang-new -fc1` with `-Xflang`
$ bin/flang-new -Xflang -fdebug-dump-parse-tree file.f95
Program -> ProgramUnit -> MainProgram
| SpecificationPart
| | ImplicitPart ->
| ExecutionPart -> Block
| EndProgramStmt ->

Note that Xflang is similar to the following option flags in Clang:

• Xclang
• Xlinker
• Xassembler

Other, somewhat similar and very popular option: mllvm. GCC has similar options too, see e.g. Xassembler.

The document implies that the frontend options are specifically designed to be of interest to compiler developers and are potentially unstable. Neither one of these seems true to me.

The frontend driver has options like -fdebug-dump-symbols, -fdebug-dump-parse-tree or -fdebug-dump-provenance. These expose the internals of the compiler. Would end-users be interested in these? The very interested ones can use flang-new -fc1. By limiting certain options to flang-new -fc1, we give ourselves a framework for e.g.:

• experimental flags (i.e. features that we are not yet sure about)
• flags to control code-paths that might be unstable or under active development

For example, what should we do with flags that control pass pipelines for generating MLIR or LLVM IR? In the frontend driver we can introduce many experimental flags to fine-tune every pass separately. But should these flags be available in our main user facing interface (i.e. flang-new)? In most cases, we would like users to rely on -O{0|1|2|3|s|z} instead. We've also discussed this briefly with folks from MLIR (https://llvm.discourse.group/t/passpipelineclparser-and-flang-driver/4291/2):

I’d be very cautious about exposing this being a flag that make it very clear that these aren’t stable and meant for development / debugging purpose only: they can change from one Flang release to another.

As for your final point,

I think it would be good to include a brief overview of the compilation processes controlled by the compiler driver and the frontend driver.

We probably need a diagram for this. I will try to produce something.

flang/docs/FlangDriver.md

75

There are many tools that "drive" LLVM (e.g. opt, clang etc). So we probably want "viewed as a driver", right?

90

Would replacing scanning with prescanning & preprocessing make sense?

[PeteSteinfeld]

Thanks for the updates and explanations, Andrzej.

Actually, I do think that end users would find it useful to dump out the symbol table and the parse tree. This is because Fortran's language definition has aspects that make it possible to create programs that are confusing to look at and interpret -- spaces are not syntactically significant, there are no reserved words, and names can be implicitly declared. I personally have spent many hours puzzling over Fortran source code, and I routinely dump out the symbol table and parse tree to help me understand what the compiler thinks the program represents.

I'm not so sure about -fdebug-dump-provenance. I rarely use it.

I agree with you that having experimental flags might be useful. But I don't see why this utility is specific to the frontend. Might it not also be useful to have experimental flags for the compiler driver?

flang/docs/FlangDriver.md

90

I'm not sure what prescanning is. Replacing with preprocessing makes sense.

[awarzynski]

flang/docs/FlangDriver.md

90

I've extracted that from the Flang documentation:

Prescan and preprocess

https://github.com/llvm/llvm-project/blob/main/flang/docs/Overview.md#prescan-and-preprocess

[PeteSteinfeld]

flang/docs/FlangDriver.md

90

I stand corrected! Bizarrely, we prescan, but we don't scan. Go figure.

[awarzynski]

In D111573#3061258, @PeteSteinfeld wrote:

Actually, I do think that end users would find it useful to dump out the symbol table and the parse tree.

We can (and probably should) make them available in flang-new then. As for -fdebug-dump-provenance, it sounds like it should remain in flang-new -fc1.

I agree with you that having experimental flags might be useful. But I don't see why this utility is specific to the frontend. Might it not also be useful to have experimental flags for the compiler driver?

We can have experimental flags in both drivers. I'm not suggesting any strict rules here. But, for every option that we add, we need to make a decision whether it's:

1. only available in flang-new
2. only available in flang-new -fc1
3. available in both flang-new and flang-new -fc1

[awarzynski]

Added a diagram for the frontend and the compiler driver, and also a bit more explanation to better highlight the difference between the two.

[PeteSteinfeld]

Ship it!

[awarzynski]

Add a note that the document is based on a design that we are working towards
rather than the current state of affairs. In particular, at assumes that
code-generation is already available, which is not the case (it's under active
development on fir-dev).

flang/docs/FlangDriver.md

 <!--===- docs/FlangDriver.md
 
    Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
    See https://llvm.org/LICENSE.txt for license information.
    SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 
 -->
 
 # Flang drivers
 
 ```eval_rst
 .. contents::
    :local:
 ```
 
+
+> **_NOTE:_** This document assumes that Flang's drivers can already generate code and
+> produce executables. However, this is still work-in-progress. By making this
+> assumption, we are able to prepare this document ahead-of-time and to provide
+> an overview of the design that we are working towards.
+
 There are two main drivers in Flang:
 * the compiler driver, `flang-new`
 * the frontend driver, `flang-new -fc1`
 
-The compiler driver will allow you to control all compilation phases (i.e.
-preprocessing, frontend code-generation, middlend/backend code-optimisation and
-lowering, linking). For frontend specific tasks, the compiler driver creates a
-Fortran compilation job and delegates it to `flang-new -fc1`, the frontend driver.
-
-The frontend driver glues all of the frontend libraries together and provides
-an easy-to-use and intuitive interface to the frontend. It accepts many
-frontend-specific options not available in `flang-new` and as such it provides a
-finer control over the frontend. Similarly to `-Xclang` in `clang`, you can use
-`-Xflang` to forward the frontend specific flags from the compiler directly to
-the frontend driver.
+> **_NOTE:_** The diagrams in this document refer to `flang` as opposed to
+> `flang-new`. This is because the diagrams reflect the final design that we
+> are still working towards. See the note on [the flang script](https://github.com/llvm/llvm-project/blob/main/flang/docs/FlangDriver.md#the-flang-script)
+> below for more context.
+
+The **compiler driver** will allow you to control all compilation phases (e.g.
+preprocessing, semantic checks, code-generation, code-optimisation, lowering
+and linking). For frontend specific tasks, the compiler driver creates a
+Fortran compilation job and delegates it to `flang-new -fc1`, the frontend
+driver. For linking, it creates a linker job and calls an external linker (e.g.
+LLVM's [`lld`](https://lld.llvm.org/)). It can also call other tools such as
+external assemblers (e.g. [`as`](https://www.gnu.org/software/binutils/)). In
+Clang, the compiler driver can also link the generated binaries with LLVM's
+static analysis/sanitizer libraries (e.g.
+[MemorySanitizer](https://clang.llvm.org/docs/MemorySanitizer.html)). This is
+not yet available in Flang, but will be relatively easy to support once such
+libraries become available. Flang's compiler driver is intended for Flang's
+end-users - its interface needs to remain stable. Otherwise, Flang's users will
+have to adjust their build scripts every time a compiler flag is changed.
+
+| ![Compiler Driver](compiler_driver.png) |
+|:--:|
+| *Flang’s compiler driver and the **tools** that it runs* |
+
+The **frontend driver** glues together and drives all of the Flang's frontend
+libraries. As such, it provides an easy-to-use and intuitive interface to the
+frontend. It uses MLIR and LLVM for code-generation and can be viewed as a
+driver for Flang, LLVM and MLIR libraries. Contrary to the compiler driver, it
+is not capable of calling any external tools (including linkers).  It is aware
+of all the frontend internals that are "hidden" from the compiler driver. It
+accepts many frontend-specific options not available in `flang-new` and as such
+it provides a finer control over the frontend. Note that this tool is mostly
+intended for Flang developers. In particular, there are no guarantees about the
+stability of its interface and compiler developers can use it to experiment
+with new flags.
+
+| ![Frontend Driver](frontend_driver.png) |
+|:-:|
+| *Flang's frontend driver and the **libraries** that it drives* |
+
+Note that similarly to `-Xclang` in `clang`, you can use `-Xflang` to forward a
+frontend specific flag from the _compiler_ directly to the _frontend_ driver,
+e.g.:
+
+```lang=bash
+flang-new -Xflang -fdebug-dump-parse-tree input.f95
+```
+
+In the invocation above, `-fdebug-dump-parse-tree` is forwarded to `flang-new
+-fc1`. Without the forwarding flag, `-Xflang`, you would see the following
+warning:
+
+```lang=bash
+flang-new: warning: argument unused during compilation:
+```
+
+As `-fdebug-dump-parse-tree` is only supported by `flang-new -fc1`, `flang-new`
+will ignore it when used without `Xflang`.
+
+## Why Do We Need Two Drivers?
+As hinted above, `flang-new` and `flang-new -fc1` are two separate tools. The
+fact that these tools are accessed through one binary, `flang-new`, is just an
+implementation detail. Each tool has a separate list of options, albeit defined
+in the same file: `clang/include/clang/Driver/Options.td`.
+
+The separation helps us split various tasks and allows us to implement more
+specialised tools. In particular, `flang-new` is not aware of various
+compilation phases within the frontend (e.g. scanning, parsing or semantic
+checks). It does not have to be. Conversely, the frontend driver, `flang-new
+-fc1`, needs not to be concerned with linkers or other external tools like
+assemblers. Nor does it need to know where to look for various systems
+libraries, which is usually OS and platform specific.
+
+One helpful way of differentiating these tools is to keep in mind that:
+
+* the compiler driver is an end-user tool
+* frontend driver is a compiler developer tool with many additional options,
+
+Also, Since the compiler driver can call external tools, e.g. linkers, it can
+be used to generate **executables**. The frontend driver cannot call external
+tools and hence can only generate **object files**. A similar model is
+implemented in Clang (`clang` vs `clang -cc1` vs `clang -cc1as`), which is
+based on the [architecture of
+GCC](https://en.wikibooks.org/wiki/GNU_C_Compiler_Internals/GNU_C_Compiler_Architecture).
+In fact, Flang needs to adhere to this model in order to be able to re-use
+Clang's driver library. If you are more familiar with the [architecture of
+GFortran](https://gcc.gnu.org/onlinedocs/gcc-4.7.4/gfortran/About-GNU-Fortran.html)
+than Clang, then `flang-new` corresponds to `gfortran` and `flang-new -fc1` to
+`f951`.
 
 ## Compiler Driver
-
 The main entry point for Flang's compiler driver is implemented in
 `flang/tools/flang-driver/driver.cpp`.  Flang's compiler driver is implemented
 in terms of Clang's driver library, `clangDriver`. This approach allows us to:
 * benefit from Clang's support for various targets, platforms and operating systems
 * leverage Clang's ability to drive various backends available in LLVM, as well
   as linkers and assemblers.
 One implication of this dependency on Clang is that all of Flang's compiler
 options are defined alongside Clang's options in

 `clang::driver::Tool`. For Flang we introduced a specialisation of this class:
 `clang::driver::Flang`. This class implements the logic to either translate or
 forward compiler options to the frontend driver, `flang-new -fc1`.
 
 You can read more on the design of `clangDriver` in Clang's [Driver Design &
 Internals](https://clang.llvm.org/docs/DriverInternals.html).
 
 ## Frontend Driver
-Flang's frontend driver is the main interface between end-users and the Flang
-frontend. The high-level design is similar to Clang's frontend driver, `clang
--cc1` and consists of the following classes:
+Flang's frontend driver is the main interface between compiler developers and
+the Flang frontend. The high-level design is similar to Clang's frontend
+driver, `clang -cc1` and consists of the following classes:
 * `CompilerInstance`, which is a helper class that encapsulates and manages
   various objects that are always required by the frontend (e.g. `AllSources`,
   `AllCookedSources, `Parsing`, `CompilerInvocation`, etc.). In most cases
   `CompilerInstance` owns these objects, but it also can share them with its
   clients when required. It also implements utility methods to construct and
   manipulate them.
 * `CompilerInvocation` encapsulates the configuration of the current
   invocation of the compiler as derived from the command-line options and the