[gn build] Link with -Wl,--gdb-index

[thakis]

Summary

For full-debug-info (is_debug=true / symbol_level=2 builds), this makes
linking 15% slower, but gdb startup 1500% faster (for lld: link time
3.9s->4.4s, gdb load time >30s->2s).

For link time, I ran

bench.py -o {noindex,index}.txt \
    sh -c 'rm out/gn/bin/lld && ninja -C out/gn lld'

and then ministat noindex.txt index.txt:

x noindex.txt
+ index.txt
    N           Min           Max        Median           Avg        Stddev
x   5      3.784461     4.0200169     3.8452811     3.8754988   0.089902595
+   5       4.32496     4.6058481     4.3361208     4.4141198    0.12288267
Difference at 95.0% confidence
        0.538621 +/- 0.15702
        13.8981% +/- 4.05161%
        (Student's t, pooled s = 0.107663)

For gdb load time I loaded the crash in PR48392 with

gdb -ex r --args ../out/gn/bin/ld64.lld.darwinnew @response.txt

and just stopped the time until the crash got displayed with a stopwatch
a few times. So the speedup there is less precise, but it's so
pronounced that that's ok (loads ~instantly with the patch, takes a very
long time without it).

Only doing this for LLD because I haven't tried it with other linkers.

---

Not sure if anyone cares about this. Holler if you think we shouldn't do this and/or put it behind a flag. See comment in the build file for the reasoning why I think this is a good default.

[thakis]

Hm D58628 suggests that maybe bfd ld doesn't support this. So maybe we should only do it if lld? (There's no "use_gold" var and I'd rather not add one.)

[thakis]

lld only

[aeubanks]

I'm confused on the term "release build". I typically use is_optimized = true and is_debug = true to get reasonably fast binaries but with backtraces for crashes (I work with a lot of crashes). Which = false does a "release build" refer to?

15% slower links seems noticeable. I'd prefer putting this behind a flag (can default to is_debug if you really want). I think putting all optional compiler/linker command line flags behind gn args makes sense.

[thakis]

In D92844#2441259, @aeubanks wrote:

I'm confused on the term "release build". I typically use is_optimized = true and is_debug = true to get reasonably fast binaries but with backtraces for crashes (I work with a lot of crashes). Which = false does a "release build" refer to?

Will replace "release" with "no-debug-information".

15% slower links seems noticeable. I'd prefer putting this behind a flag (can default to is_debug if you really want). I think putting all optional compiler/linker command line flags behind gn args makes sense.

You're setting is_debug=true but you don't run the binary under gdb? If so, wy do you set is_debug=true? For stacks with line numbers?

[aeubanks]

15% slower links seems noticeable. I'd prefer putting this behind a flag (can default to is_debug if you really want). I think putting all optional compiler/linker command line flags behind gn args makes sense.

You're setting is_debug=true but you don't run the binary under gdb? If so, wy do you set is_debug=true? For stacks with line numbers?

Yup

[MaskRay]

If -gsplit-dwarf is set, -Wl,--gdb-index makes sense.

[thakis]

In D92844#2442915, @aeubanks wrote:

15% slower links seems noticeable. I'd prefer putting this behind a flag (can default to is_debug if you really want). I think putting all optional compiler/linker command line flags behind gn args makes sense.

You're setting is_debug=true but you don't run the binary under gdb? If so, wy do you set is_debug=true? For stacks with line numbers?

Yup

Do you think we should have some mode to get -g1 for that use case? I'd expect that to make your use case link faster than it currently does. (At least while we don't use fission, but even then it'll reduce .o file size which likely means faster build time with distributed compiles.) And then we could do the index only on -g not -g1 builds.

In D92844#2442982, @MaskRay wrote:

If -gsplit-dwarf is set, -Wl,--gdb-index makes sense.

The GN build currently never sets gsplit-dwarf and -Wl,--gdb-index still seems to speed up things quite a bit. Are you saying we should _also_ use -gsplit-dwarf (probably a good idea), or are you saying something else?

[aeubanks]

In D92844#2443293, @thakis wrote:

In D92844#2442915, @aeubanks wrote:

15% slower links seems noticeable. I'd prefer putting this behind a flag (can default to is_debug if you really want). I think putting all optional compiler/linker command line flags behind gn args makes sense.

You're setting is_debug=true but you don't run the binary under gdb? If so, wy do you set is_debug=true? For stacks with line numbers?

Yup

Do you think we should have some mode to get -g1 for that use case? I'd expect that to make your use case link faster than it currently does. (At least while we don't use fission, but even then it'll reduce .o file size which likely means faster build time with distributed compiles.) And then we could do the index only on -g not -g1 builds.

Oh yes that would be ideal.

[dblaikie]

In D92844#2443293, @thakis wrote:

In D92844#2442915, @aeubanks wrote:

15% slower links seems noticeable. I'd prefer putting this behind a flag (can default to is_debug if you really want). I think putting all optional compiler/linker command line flags behind gn args makes sense.

You're setting is_debug=true but you don't run the binary under gdb? If so, wy do you set is_debug=true? For stacks with line numbers?

Yup

Do you think we should have some mode to get -g1 for that use case? I'd expect that to make your use case link faster than it currently does. (At least while we don't use fission, but even then it'll reduce .o file size which likely means faster build time with distributed compiles.) And then we could do the index only on -g not -g1 builds.

In D92844#2442982, @MaskRay wrote:

If -gsplit-dwarf is set, -Wl,--gdb-index makes sense.

The GN build currently never sets gsplit-dwarf and -Wl,--gdb-index still seems to speed up things quite a bit.

Yep - gdb-index should still be a significant speed-up even without split-dwarf. (I think there was some discussion elsewhere (on a bug or the like), but you are enabling gnu pubnames (-ggnu-pubnames) on your compiles along with gdb-index, yeah?)

Are you saying we should _also_ use -gsplit-dwarf (probably a good idea), or are you saying something else?

gdb-index doesn't need split-dwarf, but split-dwarf (at least with GDB) does need gdb-index, FWIW

If split-dwarf works for the build system (in terms of having the .dwo files available - if there's an distributed element, then that distributed system needs to be aware of the dwo files and be able to move them back - and the pathnames in the .o files need to be created in such a way that the debugger can find the dwo files after they've been moved around, etc) then, yeah, there's some benefits to be had there - smaller linker input/outputs and so some speedups.

[thakis]

Do you think we should have some mode to get -g1 for that use case? I'd expect that to make your use case link faster than it currently does. (At least while we don't use fission, but even then it'll reduce .o file size which likely means faster build time with distributed compiles.) And then we could do the index only on -g not -g1 builds.

Oh yes that would be ideal.

I rebased this across your CL (not yet uploaded) and measured --gdb-index impact for -g1:

    N           Min           Max        Median           Avg        Stddev
x   5    0.91551208     1.0278211    0.99482608    0.98919582    0.04413594
+   5     1.0646391      1.082478     1.0822251      1.076741   0.008073845
Difference at 95.0% confidence
	0.0875452 +/- 0.0462716
	8.85014% +/- 4.6777%
	(Student's t, pooled s = 0.0317267)

So with -g1 it still makes likes ~9% slower, but the absolute time difference for lld is < 0.1s, instead of 0.5s for -g2. (Load time in gdb in both cases is < 1s, with and without index. With index feels maybe a bit faster, but it's fast enough anyways that it isn't a dramatic benefit.) Do you have an opinion on if we should pass this for g1 too? Benefit is a slightly more regular build configuration, drawback is very slightly slower links. Let me know what you prefer :)

[thakis]

rebase, and only for full debug info

[aeubanks]

<0.1s seems pretty unnoticeable.
But does anybody use debuggers with -g1? If not then I don't see any reason to pass --gdb-index.

llvm/utils/gn/build/BUILD.gn

if (current_os == "mac") {
     ldflags += [ "-mmacosx-version-min=10.10" ]
   }
 
   assert(symbol_level == 0 || symbol_level == 1 || symbol_level == 2,
          "Unexpected symbol_level")
   if (host_os != "win") {
     if (symbol_level == 2) {
       cflags += [ "-g" ]
+
+      # For full debug-info -g builds, --gdb-index makes links ~15% slower, and
+      # gdb symbol reading time 1500% faster (lld links in 4.4 instead of 3.9s,
+      # and gdb loads and runs it in 2s instead of in 30s).  It's likely that
+      # people doing symbol_level=2 want to run a debugger (since
+      # symbol_level=2 isn't the default). So this seems like the right
+      # tradeoff.
+      if (host_os != "mac" && use_lld) {
+        cflags += [ "-ggnu-pubnames" ]  # PR34820
+        ldflags += [ "-Wl,--gdb-index" ]
+      }
     } else if (symbol_level == 1) {
       cflags += [ "-g1" ]
+
+      # For linetable-only -g1 builds, --gdb-index makes links ~8% slower, but
+      # links are 4x faster than -g builds so it's a fairly small absolute cost.
+      # On the other hand, gdb startup is well below 1s with and without the
+      # index, and people using -g1 likely don't use a debugger. So don't use
+      # the flag here.
     }
     if (is_optimized) {
       cflags += [ "-O3" ]
     }
     cflags += [ "-fdiagnostics-color" ]
     if (use_lld) {
       ldflags += [ "-Wl,--color-diagnostics" ]
     }