[gn build] Build with Fission on non-mac non-win when using lld

[thakis]

Summary

In release+sym builds (-O2 -g), reduces time to link clang
from 2.3s to 1.3s (-42%).

In debug builds (-g), reduces time to link clang
from 5.4s to 4.5s (-17.4%).

See the phab review for full ministat numbers.

In the CMake build this is opt-in via LLVM_USE_SPLIT_DWARF.
Since the GN build is targeted at developers, enabling it by default
seems like a better default setting here. (If it turns out to cause
problems, we can add an opt-out.)

Time to load the binary into gdb and to set a breakpoint is unchanged.
Time from run to hitting a breakpoint in main feel a bit faster
(~4s -> ~2s), but I dind't do a careful statistical anlysis for this.

[thakis]

Full numbers:

is_debug = true

$ ministat dbg_before.txt dbg_after.txt
x dbg_before.txt
+ dbg_after.txt
+----------------------------------------------------------------------------------------------------------------------------+
|+       +         +  +                            +                             x   xx               x                     x|
| |________________M_A__________________|                                     |_______M_________A________________|           |
+----------------------------------------------------------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5     5.2263865     5.7603414     5.2823768     5.4064289    0.22182468
+   5     4.2161133     4.8450658     4.4456208     4.4625463    0.23867625
Difference at 95.0% confidence
-0.943883 +/- 0.336032
-17.4585% +/- 6.21541%
(Student's t, pooled s = 0.230405)

release/sym

thakis@thakis:~/src/llvm-project/out/gn$ ministat rel_before.txt rel_after.txt
x rel_before.txt
+ rel_after.txt
+----------------------------------------------------------------------------------------------------------------------------+
|+  +  +   + +                                                                                               xx    x     x  x|
| |____A____|                                                                                                |_____MA_____|  |
+----------------------------------------------------------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5      2.204227      2.335144     2.2594814     2.2635095   0.057581912
+   5     1.2554414     1.3601334     1.3091564     1.3093105   0.042406506
Difference at 95.0% confidence
-0.954199 +/- 0.0737487
-42.1557% +/- 3.25816%
(Student's t, pooled s = 0.0505667)

For fun, I also tried with gold ld and bfd ld, but only in a release+sym build since it was fairly slow there already.

gold used -Wl,--gdb-index, except in the _noindex runs. bfd ld doesn't have --gdb-index, so I think fission doesn't work well with it (?). (I also haven't tried actually running the binary under gdb in these builds). (after vs after2 is doing the same 12 hours later, so there's some long-term noise here apparently. I trust the after2 numbers less since they claim gold is faster if it has to compute an index in those numbers, and that seems like nonsense).

I think the summary here is that fission helps a _lot_ more for gold and bfd ld (but even with fission, they're still way slower than lld). The very fastest version here is 1/3rd the speed of lld, and the slowest takes over 45x (!) as long as lld. gold without fission takes way over 10x as long as lld without fission.

$ ~/src/ministat/ministat gcc*
x gcc_bfd_after2.txt
+ gcc_bfd_after.txt
* gcc_bfd_before.txt
% gcc_gold_after2_noindex.txt
# gcc_gold_after2.txt
@ gcc_gold_after.txt
O gcc_gold_before_noindex.txt
~ gcc_gold_before.txt
+----------------------------------------------------------------------------------------------------------------------------+
|#   %       x  @@                                                        O               ~                                * |
|##  % %     x x@@@ ++                                                  OOO             ~~~~                    *   *     ** |
|A|  MA      MA |A|A_|                                                   |A             |_A|                      |____A__M_||
+----------------------------------------------------------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5     10.555829     11.590927     10.691926     10.836577    0.42719901
+   5     12.727631     15.063868     13.113123     13.631338     1.0142038
Difference at 95.0% confidence
2.79476 +/- 1.13492
25.7901% +/- 10.4731%
(Student's t, pooled s = 0.778174)
*   5     64.289209     70.342028     69.866205     68.286443     2.6778043
Difference at 95.0% confidence
57.4499 +/- 2.79647
530.148% +/- 25.8059%
(Student's t, pooled s = 1.91744)
%   5     6.3529992     7.0316789     6.3846138     6.5051929    0.29492724
Difference at 95.0% confidence
-4.33138 +/- 0.535351
-39.97% +/- 4.94022%
(Student's t, pooled s = 0.36707)
#   5     4.0398211     4.4484537     4.1234727     4.1840817    0.16065823
Difference at 95.0% confidence
-6.6525 +/- 0.470684
-61.3893% +/- 4.34347%
(Student's t, pooled s = 0.322731)
@   5     12.177341     13.516762     12.589737     12.712317    0.50764556
Difference at 95.0% confidence
1.87574 +/- 0.684228
17.3093% +/- 6.31406%
(Student's t, pooled s = 0.46915)
O   5     42.854285     43.832344     43.634122     43.500009    0.38393929
Difference at 95.0% confidence
32.6634 +/- 0.59234
301.418% +/- 5.46612%
(Student's t, pooled s = 0.406146)
~   5      51.27234      53.16056     52.245277     52.270382    0.69728535
Difference at 95.0% confidence
41.4338 +/- 0.843319
382.351% +/- 7.78215%
(Student's t, pooled s = 0.578233)

[hans]

lgtm nice!

[dblaikie]

Cool!

llvm/utils/gn/build/BUILD.gn

85

Yep, gdb relies on the index when using Split DWARF - without it you can expect a fair bit of uncertainty/variability in gdb about name lookup failures.

109–110

Actually there can be some benefit to -g1/-gmlt even with Split DWARF - especially in optimized builds.

-g1 includes some debug info beyond the actual line table, needed to describe inlining - with split-dwarf that information can still be split out into the .dwo file and save some space/time for linking.

llvm/utils/gn/build/BUILD.gn

if (symbol_level == 2) {
       # gdb symbol reading time 1500% faster (lld links in 4.4 instead of 3.9s,
       # and gdb loads and runs it in 2s instead of in 30s).  It's likely that
       # people doing symbol_level=2 want to run a debugger (since
       # symbol_level=2 isn't the default). So this seems like the right
       # tradeoff.
       if (host_os != "mac" && use_lld) {
         cflags += [ "-ggnu-pubnames" ]  # PR34820
         ldflags += [ "-Wl,--gdb-index" ]
+
+        # Use debug fission. In this mode, detailed debug information is
+        # written to a .dwo file next to each .o file instead of into the .o
+        # file directly. The linker then only links the .o files, which contain
+        # a pointer to each .dwo file. The debugger then reads debug info out
+        # of all the .dwo files instead of from the binary.
+        #
+        # (The dwp tool can link all the debug info together into a single
+        # "debug info binary", but that's not done as part of the build.)
+        #
+        # This requires `-Wl,--gdb-index` (above) to work well.
+        #
+        # With lld, this reduces link time:
+        # - in release + symbol_level=2 builds: From 2.3s to 1.3s
+        # - in debug builds: From 5.2s to 4.6s
+        #
+        # Time needed for gdb startup and setting a breakpoint is comparable,
+        # the time from from `r` to hititng a breakpoint on main goes from 4s
+        # to 2s.
+        #
+        # (macOS's linker always keeps debug info out of its output executables
+        # and debuggers there also know to load debug info from the .o files.
+        # macOS also has a debug info linker like dwp, it's called dsymutil.
+        # This happens by default, so there's no need to pass a flag there.)
+        cflags += [ "-gsplit-dwarf" ]
+        ldflags += [ "-gsplit-dwarf" ]  # Needed for ThinLTO builds.
       }
     } else if (symbol_level == 1) {
       cflags += [ "-g1" ]
       # For linetable-only -g1 builds, --gdb-index makes links ~8% slower, but
       # links are 4x faster than -g builds so it's a fairly small absolute cost.
       # On the other hand, gdb startup is well below 1s with and without the
       # index, and people using -g1 likely don't use a debugger. So don't use
       # the flag here.
+      # Linetables always go in the .o file, even with -gsplit-dwarf, so there's
+      # no point in passing -gsplit-dwarf here.
     }
     if (is_optimized) {
       cflags += [ "-O3" ]
     }
     cflags += [ "-fdiagnostics-color" ]
     if (use_lld) {
       ldflags += [ "-Wl,--color-diagnostics" ]
     }