[LTO] Parse deplibs sections for possible libcalls.

[mysterymath]

Summary

[LTO] Parse deplibs sections for possible libcalls.

LTO codegen can add symbol references resolved by object files with
.deplibs sections. These object files will be added to the link by
symbol resolution, but parseFile() will never be called on the object
file, so the .deplibs will never be resolved. This can lead to undefined
references.

This change scans the deplibs sections of such object files before LTO
occurs and summarily adds these libraries to the link. Shared libraries
are added as if --as-needed were specified, since they may not end up
actually being required. Similarly, shared libraries will only be
extracted if something actually ends up referencing the newly added
symbols.

Since the problem was first observed on AArch64-specific libcalls, this
also adds an ELF lld target interface for obtaining target-specific
libcalls (those corresponding to setLibcallName() calls in target
lowering). These are filled in for AArch64.

[mysterymath]

Add test case.

[mysterymath]

Fix requires.

[mysterymath]

Add comment.

[mysterymath]

Move parse loop earlier.

[mysterymath]

Update description.

[mysterymath]

Expand the summary.

[MaskRay]

This breaks the symbol resolution model of LTO. The symbol resolution intends to know all symbols and no new symbol can be added after LTO compiling.

[mysterymath]

In D127885#3587455, @MaskRay wrote:

This breaks the symbol resolution model of LTO. The symbol resolution intends to know all symbols and no new symbol can be added after LTO compiling.

Does that apply to symbols like __aarch64_ldadd4_relax? AFAICT, these are only discovered after LTO completes, which is the origin of the issue.
There's a bit of logic to handleLibcalls() before the link, but only if the libcalls are supplied via bitcode themselves, but not if they're provided via object files, as in the test case.

[mysterymath]

I've created https://github.com/llvm/llvm-project/issues/56070 to track the general issue and provide more background on the issue's diagnosis.

[MaskRay]

In D127885#3589557, @mysterymath wrote:

I've created https://github.com/llvm/llvm-project/issues/56070 to track the general issue and provide more background on the issue's diagnosis.

Thanks and I tagged folks who may want to participate in the discussion. Sending the patch without tagging appropriate folks motivated me to click "Request Changes".

[mysterymath]

Extract deplibs before LTO, as discussed in #56070.

This substantially reworks the change to partially parse deplibs sections from
object files that provide libcalls. A target interface is also added to supply
target-specific libcall names; this is filled out for AArch64.

Any found shared libraries are added as-needed, since the libcalls that might
require them may never be emitted. Static libraries are added lazily, as usual.

[mysterymath]

Since there wasn't a clear resolution around the ideal semantics, I'll abandon this change.

lld/ELF/Driver.cpp

const bool skipLinkedOutput = config->thinLTOIndexOnly || config->emitLLVM ||
                                 config->ltoEmitAsm ||
                                 !config->thinLTOModulesToCompile.empty();
 
   // Do link-time optimization if given files are LLVM bitcode files.
   // This compiles bitcode files into real object files.
   //
   // With this the symbol table should be complete. After this, no new names
   // except a few linker-synthesized ones will be added to the symbol table.
+  const size_t numFilesBeforeLTO = files.size();
   const size_t numObjsBeforeLTO = objectFiles.size();
   invokeELFT(compileBitcodeFiles, skipLinkedOutput);
 
+  // Adding the LTO object file to the link may have added additional input
+  // files to the link, e.g. if a builtin libcall was resolved using an object
+  // file with deplibs.
+  {
+    llvm::TimeTraceScope timeScope("Parse input files");
+    for (size_t i = numFilesBeforeLTO; i < files.size(); ++i) {
+      llvm::TimeTraceScope timeScope("Parse input files", files[i]->getName());
+      parseFile(files[i]);
+    }
+  }
+
   // Symbol resolution finished. Report backward reference problems,
   // --print-archive-stats=, and --why-extract=.
   reportBackrefs();
   writeArchiveStats();
   writeWhyExtract();
   if (errorCount())
     return;
 

lld/test/ELF/deplibs-lto.s

+# REQUIRES: aarch64
+
+# RUN: rm -rf %t.inputs %t.dir
+# RUN: mkdir -p %t.inputs %t.dir
+# RUN: split-file %s %t.inputs
+# RUN: llvm-mc -filetype=obj -triple=aarch64 %t.inputs/deplibs.s -o %tdeplibs.o
+# RUN: llvm-mc -filetype=obj -triple=aarch64 %t.inputs/foo.s -o %tfoo.o
+# RUN: llvm-as %t.inputs/lto.ll -o %t.o
+# RUN: llvm-ar rc %t.dir/libdeplibs.a %tdeplibs.o
+# RUN: llvm-ar rc %t.dir/libfoo.a %tfoo.o
+
+# LTO can emit libcalls that are resolved using libraries that contain .deplibs.
+# Such libraries must be handled as if they were input files, and any referenced
+# deplibs must be added to the link. Otherwise, symbols in the library may go
+# undefined.
+
+# RUN: ld.lld %t.o -u a -o /dev/null -L %t.dir -ldeplibs --trace 2>&1 | \
+# RUN:   FileCheck %s -DOBJ=%t.o -DDIR=%t.dir --check-prefix=LIBA-DIR
+
+# LIBA-DIR:      [[OBJ]]
+# LIBA-DIR-NEXT: [[DIR]]{{[\\/]}}libdeplibs.a
+# LIBA-DIR-NEXT: [[DIR]]{{[\\/]}}libfoo.a
+
+#--- foo.s
+.global foo
+foo:
+#--- deplibs.s
+.global __aarch64_ldadd4_relax
+__aarch64_ldadd4_relax:
+    b foo
+.section ".deplibs","MS",@llvm_dependent_libraries,1
+    .asciz "foo"
+#--- lto.ll
+target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
+target triple = "aarch64"
+
+define void @a(i32* nocapture %0) #0 {
+  %2 = atomicrmw add i32* %0, i32 1 monotonic, align 4
+  ret void
+}
+
+attributes #0 = { "target-features"="+outline-atomics" }