[docs] Added solutions to slow build under common problems

[e-leclercq]

Summary

I added a list of options to configure should someone have issues with long build time or running out of memory. This was added under common problems in the getting started section of the documentation.

[lebedev.ri]

I think this also should talk about make vs ninja generator, and using at least ld.gold or even lld.

llvm/docs/GettingStarted.rst

1107

Should mention that this is ninja specific.
And i think similar to the -j option used with make. is missing words "but for link jobs only"

[Meinersbur]

Issues with compile time and memory requirements come up regularly, so we though it would be a good idea to document mitigations. For instance:

https://lists.llvm.org/pipermail/llvm-dev/2020-January/137883.html

https://lists.llvm.org/pipermail/llvm-dev/2019-June/133506.html

http://lists.llvm.org/pipermail/llvm-dev/2019-November/137226.html

https://reviews.llvm.org/D72402

Documenting this has been suggested in those threads several times. I think a separate section just covering this problem would be a good idea. Other parts in the manual (such as https://llvm.org/docs/CMake.html#options-and-variables) are references, and my not be where someone would look when someone tries to compile LLVM the first time.

[abrachet]

Thanks for this! I was thinking about adding this as well because we get failed build emails to the list all the time.

We could recommend using -DLLVM_ENABLE_LLD=ON or gold if they don't have lld. Also -DBUILD_SHARED_LIBS=ON should help with memory a lot too.

Some people try to compile with very little memory, and that's what ultimately kills these builds. I think we should mention that there is a theoretical limit to how small of a memory footprint we can create with these options and sometimes they will be forced to increase their swap space, if their physical memory is just too small.

llvm/docs/GettingStarted.rst

1104

We could add that if you need a Debug build, you could consider using -DLLVM_USE_SPLIT_DWARF=ON which should ease memory pressure on the linker significantly.

1107

We could also mention PARALLEL_COMPILE_JOBS but it's impact is much less significant.

[Meinersbur]

Additional mitigations we should cover:

• Use ninja instead of make
• BUILD_SHARED_LIBS=ON
• Use the gold linker or lld.
• Other components that can be disabled, but are ON by default.
• Separate debug symbols file (-gsplit-dwarf)

[abrachet]

In D75425#1900040, @Meinersbur wrote:

Documenting this has been suggested in those threads several times. I think a separate section just covering this problem would be a good idea. Other parts in the manual (such as https://llvm.org/docs/CMake.html#options-and-variables) are references, and my not be where someone would look when someone tries to compile LLVM the first time.

Perhaps this belongs in another patch but I think you bring up a good point that for someone trying to compile for the first time it might be intimidating to see so many options. Perhaps we could add something like LLVM_LIMIT_RESOURCES which will encompass many of these options?

[erichkeane]

Anything like this without an LLD/Gold reference is likely insufficient. In my experience, switching to Gold or LLD has been the biggest benefit so far.

That said, do we have a version of our build scripts that build LLD and use it from then on out? We might find that despite increasing the build time a bit, it would improve compilation on low-resource machines.

llvm/docs/GettingStarted.rst

1104

While slightly less useful (since you can't as easily debug), a note about release-with-asserts as a good middle ground that compiles roughly as quickly as the normal release build might be valuable.

[Kokan]

Anything like this without an LLD/Gold reference is likely insufficient. In my experience, switching to Gold or LLD has been the biggest benefit so far.

I did some testing comparing -DLLVM_PARALLEL_LINK_JOBS and -DLLVM_USE_LINKER=lld on linux. (for parallel link using different number of values).
Based on that I would also say that if you have lld the parallel linking is insignificant[1].

On the other hand I would limit the list of "tips" here. (As too much option could cause users not to read it; I've been there). Mostly at 1-3 option I would stop, and possible there could be a different section listing all of the other options.
The first in the list should be LLVM_USE_LINKER imho (which I do not write as an llvm veteran which I am not)

[1] https://reviews.llvm.org/D72402 Note: this is no an advertisement of that patch, which is kinda rotten because of me, just to reference some data in the description.

[nhaehnle]

In D75425#1900044, @Meinersbur wrote:

• Use ninja instead of make
• BUILD_SHARED_LIBS=ON

Absolutely, though this should also mention the dynamic library build without BUILD_SHARED_LIBS -- some downstream uses of LLVM have a hard time with BUILD_SHARED_LIBS.

• Separate debug symbols file (-gsplit-dwarf)

There's a CMake option for that -- LLVM_USE_SPLIT_DWARF.

Also worth mentioning is generating the gdb-index, which makes a ridiculous difference for debugging.

llvm/docs/GettingStarted.rst

1107

Personally, I think using PARALLEL_COMPILE_JOBS is an anti-pattern. You should just tell ninja (or make, but really you should be using ninja) to parallelize the build. Ninja uses all hardware threads by default, and you can easily teach this to make as well.

The reason for setting PARALLEL_LINK_JOBS is to reduce the parallelism during linking to avoid out-of-memory conditions.

[dblaikie]

In D75425#1901038, @nhaehnle wrote:

In D75425#1900044, @Meinersbur wrote:

• Use ninja instead of make
• BUILD_SHARED_LIBS=ON

Absolutely, though this should also mention the dynamic library build without BUILD_SHARED_LIBS -- some downstream uses of LLVM have a hard time with BUILD_SHARED_LIBS.

• Separate debug symbols file (-gsplit-dwarf)

There's a CMake option for that -- LLVM_USE_SPLIT_DWARF.

Also worth mentioning is generating the gdb-index, which makes a ridiculous difference for debugging.

FWIW you actually /have/ to use gdb-index when using split DWARF with gdb, so far as I know/last I checked. (gdb will assume there's an index, query a trivial/empty one, and just stop/not go searching for things after the failed lookup). It'd be good if someone could fix the LLVM_USE_SPLIT_DWARF to add the linker flag (-Wl,--gdb-index). I /think/ -gsplit-dwarf when targeting gdb already implies -ggnu-pubnames which are needed to build the index in the linker (unless the linker is going to parse the DWARF manually, which is super slow/not great).

[e-leclercq]

Added ninja as recommended build tool, lld linker option, and gold linker option. Also, clarified similarity to the -j option in make, as requested.

[abrachet]

llvm/docs/GettingStarted.rst

1102

Should we say the CMake flag for using Ninja is -G Ninja?

1104

These are CMake options but these sentence makes it seem like they are ninja. But we should specify that LLVM_PARALLEL_LINK_JOBS is only meaningful when using Ninja

[e-leclercq]

Added the -G Ninja cmake flag. Mentioned that the -DLLVM_PARALLEL_LINK_JOBS option is only for building with ninja.

[e-leclercq]

Added release-with-asserts build type under -DCMAKE_BUILD_TYPE.

[Meinersbur]

llvm/docs/GettingStarted.rst

1108–1112

LLVM_ENABLE_LLD and LLVM_USE_LINKER are mutually exclusive. In particular, LLVM_ENABLE_LLD=ON is equivalent to LLVM_USE_LINKER=lld. I think "choice of linker" should be just one item.

Could you document what LLVM_USE_LINKER would be for using the gold linker?

1114–1119

Actually, LLVM defaults to CMAKE_BUILD_TYPE=Debug in its CMakeLists.txt:

if (NOT CMAKE_BUILD_TYPE AND NOT CMAKE_CONFIGURATION_TYPES)
  message(STATUS "No build type selected, default to Debug")
  set(CMAKE_BUILD_TYPE "Debug" CACHE STRING "Build type (default Debug)" FORCE)
endif()

This is is not generally true for CMake projects, so as a rule of thumb, I personally always define CMAKE_BUILD_TYPE.

For the documentation, we should make aware of CMAKE_BUILD_TYPE=Release (or maybe even MinSizeRel) in contrast to the default Debug. Consider switching LLVM_ENABLE_ASSERTIONS=ON for release builds. It is off by default for non-debug builds.

1122–1124

Could you also mention that link jobs may require much more memory that compile jobs, so limiting the number of parallel link jobs may reduce the peak memory usage?

If memory is limited, I'd also recommend starting with LLVM_PARALLEL_LINK_JOBS=1 and optionally increase it afterwards. Limiting the link jobs does not increase the total build time that much.

1135–1138

A recommendation would be useful for beginners. For instance, LLVM_ENABLE_PROJECTS=clang (instead of LLVM_ENABLE_PROJECT=all) may suffice for many users.

1141

Just "clang static analyzer", no need to captialize.

1144–1146

I have never used this option myself. It would be nice to mention what it does.

See also https://www.productive-cpp.com/improving-cpp-builds-with-split-dwarf/

[Meinersbur]

You should not accept your own patches. Please wait for someone else to greenlight it.

[e-leclercq]

Made the recommended changes: explained what the split dwarf option does, included more information on parallel link jobs, replaced the enable lld option with more information under the use linker option, added more information to the build type including the MinSizeRel setting, added some common projects as examples under the enable projects option, and changed the formatting of clang static analyzer, so that it is now correct.

[Meinersbur]

llvm/docs/GettingStarted.rst

1106

Maybe also mention that ninja is fester than make, especially when there is not a lot to (re-)build.

1151

Could you also mention the file extension (.dwo)?

[e-leclercq]

Added more information to the -G Ninja option and a note about the .dwo extension.

[Meinersbur]

llvm/docs/GettingStarted.rst

1106

To clarify: ninja helps most with incremental builds. That is, only some of the files are changes and only some of the build artifacts need to be recreated, often the case in development cycles. The extreme case is that no file has changed, and make/ninja only need to check every file's modified date. make recursively calls itself in every directory, taking a multiple of the time ninja does for the same task.

The wording "if you are rebuilding or building multiple times." does not say that. Why would someone build the same source multiple times?

[dim]

As far as I'm concerned, most suggestions make sense, and it is good to have this information clearly stated.

[e-leclercq]

This comment was removed by e-leclercq.

[e-leclercq]

Fixed correctness in -G Ninja option

[lebedev.ri]

llvm/docs/GettingStarted.rst

1118

I don't think there actually is a "release-with-asserts" build type,
but "relwithdebinfo"?

1126

Why "of course"?

1127–1128

Since lld is internally parallel, i'd even say this should always be 1 if using lld.

[Meinersbur]

Hi Evan,

I pushed this patch to the repository as 37943e518c5. Thanks and congratulations for your first contribution!

[Meinersbur]

llvm/docs/GettingStarted.rst

1118

What is meant here is -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON.

I don't think relwithdebinfo solves any problem compiling LLVM itself. Files will still be large because they contain debug info.

1127–1128

If this is true, our CMakeLists.txt should set LLVM_PARALLEL_LINK_JOBS whenever LLVM_USE_LINKER is set to lld.

[thakis]

Thanks for doing this!

I spend a lot of time with llvm's build, so here are some more comments.

llvm/docs/GettingStarted.rst

1113

This is misleading. You should always use lld, even when working on lld. Lld is faster at linking lld than gold is. This should recommend lld over gold more strongly, and possibly not mention gold at all -- I'm aware of no reason to use gold (except possibly if you're on a mips box).

1114–1119

Looks like this got lost. I agree that release builds with LLVM_ENABLE_ASSERTIONS=ON is a good build config, so it makes sense to mention both.

1123

Do you have numbers to back this up? At least for release builds, I haven't seen speedup from this (when using lld to link), on systems with as little as 16 gb if memory.

1127–1128

lld doesn't use that much parallelism, and lld jobs don't run that long. I don't think this should default to 1 with lld (if you hand benchmarks that show it's useful that changes things of course, but that'd surprise me).

1137

I believe this is only useful if you're doing debug builds, which you already advise against.

1147

"significantly" oversells this; it reduced full build time of clang by ~5% last I checked.

1150

Maybe mention that this one is Linux only.

[Meinersbur]

@e-leclercq Would you create a new diff to address the new comments?

llvm/docs/GettingStarted.rst

1113

One reason to not use lld is when building LLVM the first time on a system that does not have lld installed (or available as binary in a package repository).

1137

You need a debug build for debugging, when printf-debugging is not enough.

[e-leclercq]

@Meinersbur, should I update the diff here or create a new revision?

[Meinersbur]

Create a new revision.

llvm/docs/GettingStarted.rst

 
 Common Problems
 ===============
 
 If you are having problems building or using LLVM, or if you have any other
 general questions about LLVM, please consult the `Frequently Asked
 Questions <FAQ.html>`_ page.
 
+If you are having problems with limited memory and build time, please try 
+building with ninja instead of make. Please consider configuring the 
+following options with cmake:
+ 
+ * -G Ninja
+   Setting this option will allow you to build with ninja instead of make. This 
+   will significantly improve build time and memory usage, especially if you are
+   rebuilding or building multiple times.
+
+ * -DLLVM_USE_LINKER
+   Setting this option to either gold or lld will significantly improve 
+   performance. In the case that you are compiling lld, you may wish to use the 
+   gold linker as a faster alternative.
+
+ * -DCMAKE_BUILD_TYPE  
+   This option defaults to Debug; however, this may consume more memory during 
+   the linking phase. So, you may wish to use the build type Release. Another 
+   build type you may wish to consider is release-with-asserts which compiles at 
+   nearly the same rate as the Release build; however, it may not be as easy
+   to debug. MinSizeRel is another build type you may wish to consider, if you 
+   are still having problems with slow build time.
+
+ * -DLLVM_PARALLEL_LINK_JOBS  
+   Set this equal to number of jobs you wish to run simultaneously. This is 
+   similar to the -j option used with make, but only for link jobs. This option 
+   is of course only meaningful if you plan to build with ninja. You may wish to 
+   use a very low number of jobs, as this will greatly reduce the amount memory 
+   used during the build process. If you have limited memory, you may wish to 
+   set this to 1.
+ 
+ * -DLLVM_TARGETS_TO_BUILD  
+   Set this equal to the target you wish to build. You may wish to set this to 
+   X86; however, you will find a full list of targets within the 
+   llvm-project/llvm/lib/Target directory. 
+
+ * -DLLVM_OPTIMIZED_TABLEGEN  
+   Set this to ON to generate a fully optimized tablegen during build. This will 
+   significantly improve your build time.
+
+ * -DLLVM_ENABLE_PROJECTS  
+   Set this equal to the projects you wish to compile (e.g. clang, lld, etc.) If 
+   compiling more than one project, deliniate the list with a semicolon. Should 
+   you run into issues with the semicolon, try surrounding it with single quotes.  
+ 
+ * -DCLANG_ENABLE_STATIC_ANALYZER  
+   Set this option to OFF if you do not require the clang static analyzer. This 
+   should improve your build time significantly.
+ 
+ * -DLLVM_USE_SPLIT_DWARF
+   Consider setting this to ON if you require a debug build, as this will ease 
+   memory pressure on the linker. This will make linking much faster, as the 
+   binaries will not contain any of the debug information; however, this will 
+   generate the debug information in the form of a DWARF object file (with the 
+   extension .dwo).
+
 .. _links:
 
 Links
 =====
 
 This document is just an **introduction** on how to use LLVM to do some simple
 things... there are many more interesting and complicated things that you can do
 that aren't documented here (but we'll gladly accept a patch if you want to
 write something up!).  For more information about LLVM, check out:
 
 * `LLVM Homepage <http://llvm.org/>`_
 * `LLVM Doxygen Tree <http://llvm.org/doxygen/>`_
 * `Starting a Project that Uses LLVM <http://llvm.org/docs/Projects.html>`_
 
 .. _installing arcanist: https://secure.phabricator.com/book/phabricator/article/arcanist_quick_start/