[LTO] Parse deplibs sections for possible libcalls.

[mysterymath]

Summary

[LTO] Parse deplibs sections for possible libcalls.

LTO codegen can add symbol references resolved by object files with
.deplibs sections. These object files will be added to the link by
symbol resolution, but parseFile() will never be called on the object
file, so the .deplibs will never be resolved. This can lead to undefined
references.

This change scans the deplibs sections of such object files before LTO
occurs and summarily adds these libraries to the link. Shared libraries
are added as if --as-needed were specified, since they may not end up
actually being required. Similarly, shared libraries will only be
extracted if something actually ends up referencing the newly added
symbols.

Since the problem was first observed on AArch64-specific libcalls, this
also adds an ELF lld target interface for obtaining target-specific
libcalls (those corresponding to setLibcallName() calls in target
lowering). These are filled in for AArch64.

[mysterymath]

Add test case.

[mysterymath]

Fix requires.

[mysterymath]

Add comment.

[mysterymath]

Move parse loop earlier.

[mysterymath]

Update description.

[mysterymath]

Expand the summary.

[MaskRay]

This breaks the symbol resolution model of LTO. The symbol resolution intends to know all symbols and no new symbol can be added after LTO compiling.

[mysterymath]

In D127885#3587455, @MaskRay wrote:

This breaks the symbol resolution model of LTO. The symbol resolution intends to know all symbols and no new symbol can be added after LTO compiling.

Does that apply to symbols like __aarch64_ldadd4_relax? AFAICT, these are only discovered after LTO completes, which is the origin of the issue.
There's a bit of logic to handleLibcalls() before the link, but only if the libcalls are supplied via bitcode themselves, but not if they're provided via object files, as in the test case.

[mysterymath]

I've created https://github.com/llvm/llvm-project/issues/56070 to track the general issue and provide more background on the issue's diagnosis.

[MaskRay]

In D127885#3589557, @mysterymath wrote:

I've created https://github.com/llvm/llvm-project/issues/56070 to track the general issue and provide more background on the issue's diagnosis.

Thanks and I tagged folks who may want to participate in the discussion. Sending the patch without tagging appropriate folks motivated me to click "Request Changes".

[mysterymath]

Extract deplibs before LTO, as discussed in #56070.

This substantially reworks the change to partially parse deplibs sections from
object files that provide libcalls. A target interface is also added to supply
target-specific libcall names; this is filled out for AArch64.

Any found shared libraries are added as-needed, since the libcalls that might
require them may never be emitted. Static libraries are added lazily, as usual.

[mysterymath]

Since there wasn't a clear resolution around the ideal semantics, I'll abandon this change.

lld/ELF/Driver.cpp

void LinkerDriver::link(opt::InputArgList &args) {
   {
     llvm::TimeTraceScope timeScope("Parse input files");
     for (size_t i = 0; i < files.size(); ++i) {
       llvm::TimeTraceScope timeScope("Parse input files", files[i]->getName());
       parseFile(files[i]);
     }
   }
 
-  // Now that we have every file, we can decide if we will need a
-  // dynamic symbol table.
-  // We need one if we were asked to export dynamic symbols or if we are
-  // producing a shared library.
-  // We also need one if any shared libraries are used and for pie executables
-  // (probably because the dynamic linker needs it).
-  config->hasDynSymTab =
-      !sharedFiles.empty() || config->isPic || config->exportDynamic;
-
   // Some symbols (such as __ehdr_start) are defined lazily only when there
   // are undefined symbols for them, so we add these to trigger that logic.
   for (StringRef name : script->referencedSymbols) {
     Symbol *sym = addUnusedUndefined(name);
     sym->isUsedInRegularObj = true;
     sym->referenced = true;
   }
 
const bool skipLinkedOutput = config->thinLTOIndexOnly || config->emitLLVM ||
                                 config->ltoEmitAsm ||
                                 !config->thinLTOModulesToCompile.empty();
 
   // Do link-time optimization if given files are LLVM bitcode files.
   // This compiles bitcode files into real object files.
   //
   // With this the symbol table should be complete. After this, no new names
   // except a few linker-synthesized ones will be added to the symbol table.
+  const size_t numFilesBeforeLTO = files.size();
   const size_t numObjsBeforeLTO = objectFiles.size();
   invokeELFT(compileBitcodeFiles, skipLinkedOutput);
 
   // Symbol resolution finished. Report backward reference problems,
   // --print-archive-stats=, and --why-extract=.
   reportBackrefs();
   writeArchiveStats();
   writeWhyExtract();
   if (errorCount())
     return;
 
   // Bail out if normal linked output is skipped due to LTO.
   if (skipLinkedOutput)
     return;
 
+  {
+    llvm::TimeTraceScope timeScope("Parse input files");
+    for (size_t i = numFilesBeforeLTO; i < files.size(); ++i) {
+      llvm::TimeTraceScope timeScope("Parse input files", files[i]->getName());
+      parseFile(files[i]);
+    }
+  }
+
+  // Now that we have every file, we can decide if we will need a
+  // dynamic symbol table.
+  // We need one if we were asked to export dynamic symbols or if we are
+  // producing a shared library.
+  // We also need one if any shared libraries are used and for pie executables
+  // (probably because the dynamic linker needs it).
+  config->hasDynSymTab =
+      !sharedFiles.empty() || config->isPic || config->exportDynamic;
+
   // compileBitcodeFiles may have produced lto.tmp object files. After this, no
   // more file will be added.
   auto newObjectFiles = makeArrayRef(objectFiles).slice(numObjsBeforeLTO);
   parallelForEach(newObjectFiles, initializeLocalSymbols);
   parallelForEach(newObjectFiles, postParseObjectFile);
   for (const DuplicateSymbol &d : ctx->duplicates)
     reportDuplicate(*d.sym, d.file, d.section, d.value);