[flang] Fix the documentation on how to build flang

[PeteSteinfeld]

Summary

I recently had an email exchange on flang-dev that revealed that the
documentation on how to build flang is incorrect. This update fixes
that.

[sscalpone]

flang/README.md

75

Do you need flang here?

[PeteSteinfeld]

flang/README.md

75

Good catch. I'll add it.

[PeteSteinfeld]

Responding to Steve's comments.

[awarzynski]

Good morning @PeteSteinfeld , thank you for making this documentation so much clearer! I have a couple of small suggestions (these are not blockers for me).

out-of-tree vs standalone
Have you considered switching to "standalone builds" instead of "out-of-tree builds"? That would be more:

• consistent with the actual implementation/naming in the CMake script,
• accurate, i.e. Flang is always in-tree these days (i.e. inside llvm-project).

I guess that "out-of-tree" was accurate/descriptive enough before "f18" was merged into "llvm-project". But these days Flang is always in tree. Also, "standalone" would be more consistent with other sub-projects.

CMake flags
On a different note, would it make sense to trim this set of CMake flags?

cmake \
  -G Ninja \
  ../llvm \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLVM_LIT_ARGS=-v \
  -DFLANG_ENABLE_WERROR=On \
  -DCMAKE_CXX_STANDARD=17 \
  -DLLVM_ENABLE_PROJECTS="clang;mlir;flang;compiler-rt" \
  -DLLVM_BUILD_TOOLS=On \
  -DLLVM_INSTALL_UTILS=On \
  -DLLVM_TARGETS_TO_BUILD=host \
  -DLLVM_ENABLE_ASSERTIONS=ON \
  -DCMAKE_CXX_LINK_FLAGS="-Wl,-rpath,$LD_LIBRARY_PATH" \
  -DCMAKE_INSTALL_PREFIX=$INSTALLDIR

Most developers want to run ninja check-flang and for that the following should be sufficient:

make \
  -G Ninja \
  ../llvm \
  -DCMAKE_BUILD_TYPE=Release \
  -DFLANG_ENABLE_WERROR=On \
  -DLLVM_ENABLE_PROJECTS="clang;mlir;flang" \
  -DLLVM_TARGETS_TO_BUILD=host \
  -DLLVM_ENABLE_ASSERTIONS=ON

Other flags are only really needed for "out-of-tree"/"standalone" builds (in order to install or build dependencies) or to make the LIT output more verbose.

I will also add @Meinersbur to the list of reviewers as he's very familiar with LLVM's build system.

flang/README.md

54

Is compiler-rt really needed? I never include it myself.

59

You could shorten the text by adding this in the bash snippet:

git clone https://github.com/llvm/llvm-project.git
cd llvm-project

59

This will add syntax highlighting. Similar comment for other code snippets in this document.

74

You can skip this, it is set in the CMake build: https://github.com/llvm/llvm-project/blob/main/flang/CMakeLists.txt#L6

110

No compiler-rt here would suggest that it's not required at all.

[PeteSteinfeld]

@awarzynski, thanks for the great feedback.

I like your suggestion of using "standalone". I'll change it.

With respect to invoking CMake and its command line, I started by spending time finding the minimal set of options that would result in a correct, testable build.

I agree that we should minimize the number CMake flags. I'll do it. But I personally always use the verbose output of "lit". It's much more useful, so I think it should be included.

flang/README.md

54

We recently found that it's needed to run executable tests in fir-dev. Including it now will avoid needing to add it later in the instructions.

59

Good suggestion about giving specific instructions for git clone. But I think it's best to separate the cloning from the rest of the build commands.

Thanks for the tip about the bash highlighting. You're a genius!

74

Thanks. Will do.

110

As I mentioned earlier, it's needed for testing flang compiled executables.

[PeteSteinfeld]

Responding to review comments.

[awarzynski]

Thank you for addressing my comments, Pete! I've left a couple more suggestions - what do you think? We could merge this as is and continue the discussion elsewhere - this is already a huge improvement! And I promise not to come back with even more ideas :)

flang/README.md

54

So:

• llvm, mlir and clang are "build-time" dependencies (i.e. Flang won't build without them)
• compiler-rt is a "run-time" dependency (i.e. Flang will build just fine without it, ninja check-flang will also work fine, but e.g. flang-new will fail for some more complicated examples due to missing symbols)

I think that including compiler-rt is fine, but it would be helpful to explain this distinction. In particular, compiler-rt would not be required for Flang developers who would only run ninja check-flang (as far as I know, there are no executable tests in "fir-dev"). Also buildbot maintainers could safely skip it.

By the way, have you considered using LLVM_ENABLE_RUNTIMES CMake flag here? There's been some discussion recently on LLVM_ENABLE_RUNTIMES vs LLVM_ENABLE_PROJECTS for runtime libraries:

• https://lists.llvm.org/pipermail/llvm-dev/2021-October/153238.html
• https://github.com/ClangBuiltLinux/llvm-distributors-conf-2021/blob/main/slides/LLVM%20Runtimes%20Build.pdf and https://www.youtube.com/watch?v=UMDRAmmDBgM

Those discussions focused on Clang (i.e. building a Clang-based toolchain), libc++, libc++abi, libunwind (i.e. C++ runtime libraries) and compiler-rt (relevant here). We could be suggesting -DLLVM_ENABLE_RUNTIMES="compiler-rt" instead of -DLLVM_ENABLE_PROJECTS="compiler-rt" in this README. This would be more consistent with the other sub-projects and would also emphasize that compiler-rt is a slightly different dependency ("build-time" vs "run-time").

Using -DLLVM_ENABLE_RUNTIMES="compiler-rt" would mean that compiler-rt is bootstrapped using Clang that was build earlier in the build process. From the official documentation:

This is the correct way to build libc++ when putting together a toolchain

If I understand correctly, similar logic applies to compiler-rt. Having said all that, both approaches should work fine for now.

123–125

It's a bit confusing that for LLVM it's $base/build/ and otherwise it's $base/install/. If I recall correctly, you need:

• build directory for LLVM, because that's the only location where GTest is available (required for unit tests)
• install directory for Clang, because that's the only location were AddClang.cmake script is available (required for Clang configuration)

Fixing this would be nice, but that's totally out of scope here! Perhaps you could add a note outside this code snippet to highlight this inconsistency,

"Note that for Clang and MLIR you will use the installation directory ($base/install) and for LLVM you will need to build directory ($base/build). You can use the installation directory for all three sub-projects if you don't require the unit tests (which are configured by LLVM and are only available in the build directory)."

Otherwise this inconsistency is confusing and people might think that it's a typo. Or just miss it altogether!

[PeteSteinfeld]

flang/README.md

123–125

You're analysis is correct.

[PeteSteinfeld]

Thanks again, Andrzej for your excellent review comments.

I will add explanations for why compiler-rt is required and to note the peculiar different between the specification of the llvm and the mlir;clang;compiler-rt dependency specifications.

I'm unfamiliar with LLVM_ENABLE_RUNTIMES. I'm reluctant to document something that I don't understand.

[PeteSteinfeld]

Responding to more review comments.

[PeteSteinfeld]

Added a brief explanation for the inclusion of "compiler-rt".

[PeteSteinfeld]

@awarzynski, are there any other changes you need to approve this?

[awarzynski]

LGTM, thank you for this!

flang/README.md

54

[nit] Would you mind adding some line feeds here before merging? People usually tweak their editors for narrower lines.

123–125

Thanks for the update! I believe that with https://reviews.llvm.org/D116731 we should be able to use $base/build for all sub-projects. We can re-visit this document once that's merged.

130

[nit] Would you mind adding some line feeds here before merging? People usually tweak their editors for narrower lines.

[rovka]

flang/README.md

101

I have 2 mostly orthogonal comments here:

1. People might skip directly to this section and then be surprised a few lines down when you mention the in tree build. I think this paragraph should make it clear from the start that you're expected to go through the previous section first. We should probably also mention that if you're building standalone you no longer need to include flang in LLVM_ENABLE_PROJECTS for the in tree build (otherwise people will end up building flang twice when they're really only interested in the standalone one).

2. Do we really need to clone again? It should work to just point cmake to the flang subdirectory inside the first clone.

[PeteSteinfeld]

flang/README.md

101

Thanks for the feedback, Diana.

Good point about needing to tell people that building in tree is a prerequisite for building standalone. I'll add some text.

I originally did not include flang in the list of projects for the in tree build, but an earlier reviewer requested it. In fact, my normal practice is to include it, if only to be able to run the flang tests and verify that the source code is good. I'll add a note to capture this.

You don't need to clone again. But that's my normal practice. This is because I typically create one base, in tree build and then create several standalone builds that all use that base builds. Typically, these stand-along builds have different Git repositories than the base build.

[PeteSteinfeld]

Responding to Diana's comments.

[PeteSteinfeld]

As per Andrzej's request, I've updated the text to fit into 80 columns.

flang/README.md

 If you're interested in contributing to the compiler,
 read the [style guide](docs/C++style.md)
 and
 also review [how flang uses modern C++ features](docs/C++17.md).
 
 If you are interested in writing new documentation, follow 
 [markdown style guide from LLVM](https://github.com/llvm/llvm-project/blob/main/llvm/docs/MarkdownQuickstartTemplate.md).
 
+## Building flang
+There are two ways to build flang.  The first method is to build it at the same
+time that you build all of the projects on which it depends.  This is called
+building in tree.  The second method is to first do an in tree build to create
+all of the projects on which flang depends, then build an install area for
+these projects, and then only build the flang code itself.  This is called
+building standalone.  Building standalone has the advantage of being smaller
+and faster.  Once you create the base build and base install areas, you can
+create multiple standalone builds using them.  
+
+Note that instructions for building LLVM can be found at
+https://llvm.org/docs/GettingStarted.html.
+
+### Building flang in tree
+Building flang in tree means building flang along with all of the projects on
+which it depends.  These projects include mlir, clang, flang, and compiler-rt.  Note that compiler-rt is only needed to access libraries that support 16 bit floating point numbers.  It's not needed to run the automated tests.
+
+Here's a complete set of commands to clone all of the necessary source and do
+the build.  
+
+First clone the source:
+```bash
+git clone https://github.com/llvm/llvm-project.git my-project
+```
+Once the clone is complete, execute the following commands:
+```bash
+cd my-project
+INSTALLDIR=`pwd`/install
+
+rm -rf build
+rm -rf install
+mkdir -p build
+
+cd build
+
+cmake \
+  -G Ninja \
+  ../llvm \
+  -DCMAKE_BUILD_TYPE=Release \
+  -DFLANG_ENABLE_WERROR=On \
+  -DLLVM_ENABLE_ASSERTIONS=ON \
+  -DLLVM_TARGETS_TO_BUILD=host \
+  -DCMAKE_INSTALL_PREFIX=$INSTALLDIR
+  -DLLVM_LIT_ARGS=-v \
+  -DLLVM_ENABLE_PROJECTS="clang;mlir;flang;compiler-rt"
+
+ninja
+```
+
+To run the flang tests on this build, execute the command in the "build"
+directory:
+```bash
+ninja check-flang
+```
+
+If you're happy with the results, the next step is to create the install area.
+While in the `build` directory, run the command:
+```bash
+ninja install
+```
+
+Note that these instructions specify flang as one of the projects to build in the in tree build.  This is not strictly necessary for subsequent standalone builds, but doing so lets you run the flang tests to verify that the source code is in good shape.
+### Building flang standalone
+To do the standalone build, start by building flang in tree as described above.  This build is base build for subsequent standalone builds.  Start each standalone build the same way by cloning the source for llvm-project:
+```bash
+git clone https://github.com/llvm/llvm-project.git standalone
+```
+Once the clone is complete, execute the following commands:
+```bash
+cd standalone
+base=<directory that contains the in tree build>
+
+cd flang
+rm -rf build
+mkdir build
+cd build
+
+cmake \
+  -G Ninja \
+  -DCMAKE_BUILD_TYPE=Release \
+  -DFLANG_ENABLE_WERROR=On \
+  -DLLVM_TARGETS_TO_BUILD=host \
+  -DLLVM_ENABLE_ASSERTIONS=On \
+  -DLLVM_BUILD_MAIN_SRC_DIR=$base/build/lib/cmake/llvm \
+  -DLLVM_LIT_ARGS=-v \
+  -DLLVM_DIR=$base/build/lib/cmake/llvm \
+  -DCLANG_DIR=$base/install/lib/cmake/clang \
+  -DMLIR_DIR=$base/install/lib/cmake/mlir \
+  ..
+
+ninja
+```
+Note that for Clang and MLIR you use the installation directory ($base/install) and for LLVM you use the build directory (`$base/build`).   This is not a typo in the script.  Rather, it is because running the tests requires the GTest infrastructure which is only available in the LLVM build area.  The build also requires the `AddClang.cmake` script from Clang, which is only available in the install area.
+
+To run the flang tests on this build, execute the command in the "flang/build"
+directory:
+```bash
+ninja check-flang
+```
+
 ## Supported C++ compilers
 
 Flang is written in C++17.
 
 The code has been compiled and tested with
 GCC versions from 7.2.0 to 9.3.0.
 
 The code has been compiled and tested with
 clang version 7.0, 8.0, 9.0 and 10.0
 using either GNU's libstdc++ or LLVM's libc++.
 
 The code has been compiled on
 AArch64, x86\_64 and ppc64le servers
 with CentOS7, Ubuntu18.04, Rhel, MacOs, Mojave, XCode and
 Apple Clang version 10.0.1.
 
 The code does not compile with Windows and a compiler that does not have
 support for C++17.
 
-## Building Flang out of tree
-These instructions are for building Flang separately from LLVM; if you are
-building Flang alongside LLVM then follow the standard LLVM build instructions
-and add flang to `LLVM_ENABLE_PROJECTS` instead, as detailed there.
-
-### LLVM dependency
-
-The instructions to build LLVM can be found at
-https://llvm.org/docs/GettingStarted.html. If you are building flang as part
-of LLVM, follow those instructions and add flang to `LLVM_ENABLE_PROJECTS`.
-
-We highly recommend using the same compiler to compile both llvm and flang.
-
-The flang CMakeList.txt file uses
-* `LLVM_DIR` to find the installed LLVM components
-* `MLIR_DIR` to find the installed MLIR components
-* `CLANG_DIR` to find the installed Clang components
-
-To get the correct LLVM, MLIR and Clang libraries included in your flang build,
-define `LLVM_DIR`, `MLIR_DIR` and `CLANG_DIR` on the cmake command line.
-```
-LLVM=<LLVM_BUILD_DIR>/lib/cmake/llvm \
-MLIR=<LLVM_BUILD_DIR>/lib/cmake/mlir \
-CLANG=<LLVM_BUILD_DIR>/lib/cmake/clang \
-cmake -DLLVM_DIR=$LLVM -DMLIR_DIR=$MLIR -DCLANG_DIR=$CLANG ...
-```
-where `LLVM_BUILD_DIR` is
-the top-level directory where LLVM was built.
-
 ### Building flang with GCC
 
 By default,
 cmake will search for g++ on your PATH.
 The g++ version must be one of the supported versions
 in order to build flang.
 
 Or, cmake will use the variable CXX to find the C++ compiler. CXX should include
 the full path to the compiler or a name that will be found on your PATH, e.g.
 g++-8.3, assuming g++-8.3 is on your PATH.
 
-```
+```bash
 export CXX=g++-8.3
 ```
 or
-```
+```bash
 CXX=/opt/gcc-8.3/bin/g++-8.3 cmake ...
 ```
 
 ### Building flang with clang
 
 To build flang with clang,
 cmake needs to know how to find clang++
 and the GCC library and tools that were used to build clang++.
 
 CXX should include the full path to clang++
 or clang++ should be found on your PATH.
-```
+```bash
 export CXX=clang++
 ```
 
 ### Installation Directory
 
 To specify a custom install location,
 add
 `-DCMAKE_INSTALL_PREFIX=<INSTALL_PREFIX>`

 Debug builds execute slowly.
 
 To create a release build,
 add
 `-DCMAKE_BUILD_TYPE=Release`
 to the cmake command.
 Release builds execute quickly.
 
-### Build Flang out of tree
-```
-cd ~/flang/build
-cmake -DLLVM_DIR=$LLVM -DMLIR_DIR=$MLIR -DCLANG_DIR=$CLANG ~/flang/src
-make
-```
-
 # How to Run Tests
 
 Flang supports 2 different categories of tests
 1. Regression tests (https://www.llvm.org/docs/TestingGuide.html#regression-tests)
 2. Unit tests (https://www.llvm.org/docs/TestingGuide.html#unit-tests)
 
-## For out of tree builds
+## For standalone builds
 To run all tests:
-```
+```bash
 cd ~/flang/build
 cmake -DLLVM_DIR=$LLVM -DMLIR_DIR=$MLIR ~/flang/src
-make test check-all
+ninja check-all
 ```
 
 To run individual regression tests llvm-lit needs to know the lit
 configuration for flang. The parameters in charge of this are:
 flang_site_config and flang_config. And they can be set as shown below:
-```
+```bash
 <path-to-llvm-lit>/llvm-lit \
  --param flang_site_config=<path-to-flang-build>/test-lit/lit.site.cfg.py \
  --param flang_config=<path-to-flang-build>/test-lit/lit.cfg.py \
   <path-to-fortran-test>
 
 ```
 
 Unit tests:
 
 If flang was built with `-DFLANG_INCLUDE_TESTS=On` (`ON` by default), it is possible to generate unittests.
-Note: Unit-tests will be skipped for LLVM install for an out-of-tree build as it does not include googletest related headers and libraries.
+Note: Unit-tests will be skipped for LLVM install for an standalone build as it does not include googletest related headers and libraries.
 
 There are various ways to run unit-tests.
 
 ```
 
-1. make check-flang-unit
-2. make check-all or make check-flang
+1. ninja check-flang-unit
+2. ninja check-all or ninja check-flang
 3. <path-to-llvm-lit>/llvm-lit \
         test/Unit
-4. Invoking tests from <out-of-tree flang build>/unittests/<respective unit test folder>
+4. Invoking tests from <standalone flang build>/unittests/<respective unit test folder>
 
 ```
 
 
 ## For in tree builds
 If flang was built with `-DFLANG_INCLUDE_TESTS=On` (`On` by default), it is possible to
 generate unittests.
 
 To run all of the flang unit tests use the `check-flang-unit` target:
-```
-make check-flang-unit
+```bash
+ninja check-flang-unit
 ```
 To run all of the flang regression tests use the `check-flang` target:
-```
-make check-flang
+```bash
+ninja check-flang
 ```
 
 # How to Generate Documentation
 
 ## Generate FIR Documentation
 If flang was built with `-DLINK_WITH_FIR=On` (`On` by default), it is possible to
-generate FIR language documentation by running `make flang-doc`. This will
+generate FIR language documentation by running `ninja flang-doc`. This will
 create `docs/Dialect/FIRLangRef.md` in flang build directory.
 
 ## Generate Doxygen-based Documentation
 To generate doxygen-style documentation from source code
 - Pass `-DLLVM_ENABLE_DOXYGEN=ON -DFLANG_INCLUDE_DOCS=ON` to the cmake command.
 
-```
+```bash
 cd ~/llvm-project/build
 cmake -DLLVM_ENABLE_DOXYGEN=ON -DFLANG_INCLUDE_DOCS=ON ../llvm
-make doxygen-flang
+ninja doxygen-flang
 ```
 
 It will generate html in
 
-```
+```bash
     <build-dir>/tools/flang/docs/doxygen/html # for flang docs
 ```
 ## Generate Sphinx-based Documentation
 <!TODO: Add webpage once we have a website.
 !>
 Flang documentation should preferably be written in `markdown(.md)` syntax (they can be in `reStructuredText(.rst)` format as well but markdown is recommended in first place), it
 is mostly meant to be processed by the Sphinx documentation generation
 system to create HTML pages which would be hosted on the webpage of flang and
 updated periodically.
 
 If you would like to generate and view the HTML locally:
 - Install [Sphinx](http://sphinx-doc.org/), including the [sphinx-markdown-tables](https://pypi.org/project/sphinx-markdown-tables/) extension.
 - Pass `-DLLVM_ENABLE_SPHINX=ON -DSPHINX_WARNINGS_AS_ERRORS=OFF` to the cmake command.
 
-```
+```bash
 cd ~/llvm-project/build
 cmake -DLLVM_ENABLE_SPHINX=ON -DSPHINX_WARNINGS_AS_ERRORS=OFF ../llvm
-make docs-flang-html
+ninja docs-flang-html
 ```
 
 It will generate html in
 
-```
+```bash
    $BROWSER <build-dir>/tools/flang/docs/html/
 ```