[ELF2] Add --[no-]whole-archive command line switches

[ikudrin]

Summary

If --whole-archive is used, all symbols from the following archives are added to the output. --no-whole-archive restores default behavior. These switches can be used multiple times.

NB. We have to keep an ArchiveFile instance within SymbolTable even if --whole-archive mode is active since it can be a thin archive which contains just names of external files. In that case actual memory buffers for the archive members will be stored within the File member of ArchiveFile class.

[grimar]

ELF/Driver.cpp

171

There is no default section. Not sure is is necessarily to use though since I guess warning about that is turned off in most cases.

ELF/InputFiles.cpp

223

Looks like that assert should be at first line. Also that would be consistent with ArchiveFile::getMembers()

[ruiu]

ELF/InputFiles.cpp

236–237

We don't have this assertion at other places in this file, so we shouldn't have this one as well. Also, nullptr != something looks inconsistent.

ELF/SymbolTable.cpp

30–31

You want to avoid calling parse() on an archive file if Config->WholeArchive to keep the existing behavior of parse().

[ikudrin]

• Updated to the top.
• Addressed review comments.

[ruiu]

ELF/InputFiles.cpp

184–191

The point is that we never want to call both parse() and getMembers() on the same ArchiveFile object. If you call getMembers() on an ArchiveFile, that's the end of that archive file, and we won't call any other member function after that.

Restore the original code here so that it sets this class' File pointer only when parse() is called. We don't have to set that pointer in getMembers().

187–191

I'd be easy to read if you write this way:

File = open(MB);

assuming you modify the function as

std::unique_ptr<Archive> open(MemoryBufferRef MB);

233–235

This guard should be removed. You can call either getMembers() or pares() but not both.

ELF/SymbolTable.cpp

37–38

Do not push archive file objects to the vector if WholeArchive is true. If --whole-archive is set, archive files are treated as if they did not exist.

39–40

Early return after this for-loop.

[ikudrin]

ELF/SymbolTable.cpp

37–38

If I understand it right, Archive::Child::getBuffer() can, under certain circumstances, create a new memory buffer and return its reference. This memory buffer is stored in the ThinBuffers member of the instance of Archive. That's why we need to hold Archive through holding ArchiveFile.

[ikudrin]

• Rebase to the top
• Fix indentation
• Sort switches
• Slightly rework ArchiveFile && add explanations why we have to keep it

[ruiu]

ELF/InputFiles.cpp

219

Remove this comment.

ELF/SymbolTable.cpp

33–36

Is that true? MemoryBuffers are not freed until the end of Driver, so it should be safe if thin archives have references to MemoryBuffers.

39–40

Please return here.

[ikudrin]

ELF/SymbolTable.cpp

33–36

A thin archive holds just names of the external files. These files are opened and their memory buffers are stored within the instance of Archive class, not in Driver.

[ruiu]

ELF/SymbolTable.cpp

33–36

I see. Please note that in the commit message and remove this comment from here.

[ikudrin]

• Addressed review comments

[ruiu]

LGTM with a nit.

ELF/Options.td

39–40

"_" is smaller than "i" in ASCIIbetical order. Move this before noinhibit_exec.

ELF/Config.h

struct Configuration {
   bool AllowMultipleDefinition;
   bool DiscardAll;
   bool DiscardLocals;
   bool DiscardNone;
   bool ExportDynamic;
   bool NoInhibitExec;
   bool Shared;
   bool Static = false;
+  bool WholeArchive = false;
 };
 
 extern Configuration *Config;
 
 } // namespace elf2
 } // namespace lld
 
 #endif

ELF/Driver.cpp

case OPT_INPUT:
       addFile(Arg->getValue());
       break;
     case OPT_Bstatic:
       Config->Static = true;
       break;
     case OPT_Bdynamic:
       Config->Static = false;
       break;
+    case OPT_whole_archive:
+      Config->WholeArchive = true;
+      break;
+    case OPT_no_whole_archive:
+      Config->WholeArchive = false;
+      break;
     default:
       break;
     }
   }
 
   if (Symtab.getObjectFiles().empty())
     error("no input files.");
 
   // Write the result.
   const ELFFileBase *FirstObj = Symtab.getFirstELF();
   switch (FirstObj->getELFKind()) {
   case ELF32LEKind:
     writeResult<object::ELF32LE>(&Symtab);
     return;
   case ELF32BEKind:
     writeResult<object::ELF32BE>(&Symtab);
     return;
   case ELF64LEKind:
     writeResult<object::ELF64LE>(&Symtab);
     return;
   case ELF64BEKind:
     writeResult<object::ELF64BE>(&Symtab);
     return;
   }
 }

ELF/InputFiles.h

public:
   void parse() override;
 
   // Returns a memory buffer for a given symbol. An empty memory buffer
   // is returned if we have already returned the same memory buffer.
   // (So that we don't instantiate same members more than once.)
   MemoryBufferRef getMember(const Archive::Symbol *Sym);
 
   llvm::MutableArrayRef<Lazy> getLazySymbols() { return LazySymbols; }
+  std::vector<MemoryBufferRef> getMembers();
 
 private:
   std::unique_ptr<Archive> File;
   std::vector<Lazy> LazySymbols;
   llvm::DenseSet<uint64_t> Seen;
 };
 
 // .so file.

ELF/InputFiles.cpp

SymbolBody *elf2::ObjectFile<ELFT>::createSymbolBody(StringRef StringTable,
   default:
     error("unexpected binding");
   case STB_GLOBAL:
   case STB_WEAK:
     return new (Alloc) DefinedRegular<ELFT>(Name, *Sym, *Sections[SecIndex]);
   }
 }
 
-void ArchiveFile::parse() {
+static std::unique_ptr<Archive> openArchive(MemoryBufferRef MB) {
   ErrorOr<std::unique_ptr<Archive>> ArchiveOrErr = Archive::create(MB);
   error(ArchiveOrErr, "Failed to parse archive");
-  File = std::move(*ArchiveOrErr);
+  return std::move(*ArchiveOrErr);
+}
+
+void ArchiveFile::parse() {
+  File = openArchive(MB);
 
   // Allocate a buffer for Lazy objects.
   size_t NumSyms = File->getNumberOfSymbols();
   LazySymbols.reserve(NumSyms);
 
   // Read the symbol table to construct Lazy objects.
   for (const Archive::Symbol &Sym : File->symbols())
     LazySymbols.emplace_back(this, Sym);
if (!Seen.insert(It->getChildOffset()).second)
     return MemoryBufferRef();
 
   ErrorOr<MemoryBufferRef> Ret = It->getMemoryBufferRef();
   error(Ret, Twine("Could not get the buffer for the member defining symbol ") +
                  Sym->getName());
   return *Ret;
 }
 
+std::vector<MemoryBufferRef> ArchiveFile::getMembers() {
+  // We have to hold it in case of a thin archive.
+  File = openArchive(MB);
+
+  std::vector<MemoryBufferRef> Result;
+  for (const Archive::Child &Child : File->children()) {
+    ErrorOr<MemoryBufferRef> MbOrErr = Child.getMemoryBufferRef();
+    error(MbOrErr,
+          Twine("Could not get the buffer for a child of the archive ") +
+              File->getFileName());
+    Result.push_back(MbOrErr.get());
+  }
+  return Result;
+}
+
 template <class ELFT>
 SharedFile<ELFT>::SharedFile(MemoryBufferRef M)
     : SharedFileBase(getStaticELFKind<ELFT>(), M), ELFData<ELFT>(M) {}
 
 template <class ELFT> void SharedFile<ELFT>::parse() {
   typedef typename ELFFile<ELFT>::Elf_Dyn Elf_Dyn;
   typedef typename ELFFile<ELFT>::uintX_t uintX_t;
   const Elf_Shdr *DynamicSec = nullptr;
 
   const ELFFile<ELFT> Obj = this->ELFObj;
   for (const Elf_Shdr &Sec : Obj.sections()) {
     uint32_t Type = Sec.sh_type;
     if (Type == SHT_DYNSYM)

ELF/Options.td

def export_dynamic : Flag<["--"], "export-dynamic">,
   HelpText<"Put symbols in the dynamic symbol table">;
 
 def l : Joined<["-"], "l">, MetaVarName<"<libName>">,
   HelpText<"Root name of library to use">;
 
 def noinhibit_exec : Flag<["--"], "noinhibit-exec">,
   HelpText<"Retain the executable output file whenever it is still usable">;
 
+def no_whole_archive : Flag<["--"], "no-whole-archive">,
+  HelpText<"Restores the default behavior of loading archive members">;
+
 def output : Separate<["-"], "o">, MetaVarName<"<path>">,
   HelpText<"Path to file to write output">;
 
 def rpath : Separate<["-"], "rpath">,
   HelpText<"Add a DT_RUNPATH to the output">;
 
 def shared : Flag<["-"], "shared">,
   HelpText<"Build a shared object">;
 
 def sysroot : Joined<["--"], "sysroot=">,
   HelpText<"Set the system root">;
 
+def whole_archive : Flag<["--"], "whole-archive">,
+  HelpText<"Force load of all members in a static library">;
+
 // Aliases
 def alias_Bdynamic_call_shared: Flag<["-"], "call_shared">, Alias<Bdynamic>;
 def alias_Bdynamic_dy: Flag<["-"], "dy">, Alias<Bdynamic>;
 def alias_Bstatic_dn: Flag<["-"], "dn">, Alias<Bstatic>;
 def alias_Bstatic_non_shared: Flag<["-"], "non_shared">, Alias<Bstatic>;
 def alias_Bstatic_static: Flag<["-"], "static">, Alias<Bstatic>;
 def alias_L : Joined<["--"], "library-path=">, Alias<L>;
 def alias_discard_all: Flag<["-"], "x">, Alias<discard_all>;

ELF/SymbolTable.cpp

 
 SymbolTable::SymbolTable() {}
 
 bool SymbolTable::shouldUseRela() const {
   ELFKind K = getFirstELF()->getELFKind();
   return K == ELF64LEKind || K == ELF64BEKind;
 }
 
 void SymbolTable::addFile(std::unique_ptr<InputFile> File) {
-  File->parse();
-  InputFile *FileP = File.release();
-  if (auto *AF = dyn_cast<ArchiveFile>(FileP)) {
+  if (auto *AF = dyn_cast<ArchiveFile>(File.get())) {
+    File.release();
+    // ArchiveFile should be kept in both cases:
+    // * WholeArchive: it might be a thin archive which holds actual
+    //   memory buffers of its members.
+    // * !WholeArchive: it holds lazy symbols.
     ArchiveFiles.emplace_back(AF);
+    if (Config->WholeArchive) {
+      for (MemoryBufferRef &MBRef : AF->getMembers())
+        addFile(createELFFile<ObjectFile>(MBRef));
+    } else {
+      AF->parse();
       for (Lazy &Sym : AF->getLazySymbols())
         addLazy(&Sym);
+    }
     return;
   }
-  addELFFile(cast<ELFFileBase>(FileP));
+  File->parse();
+  addELFFile(cast<ELFFileBase>(File.release()));
 }
 
 static TargetInfo *createTarget(uint16_t EMachine) {
   switch (EMachine) {
   case EM_386:
     return new X86TargetInfo();
   case EM_AARCH64:
     return new AArch64TargetInfo();

test/elf2/Inputs/whole-archive.s

+.globl _bar;
+_bar:

test/elf2/whole-archive.s

+// REQUIRES: x86
+
+// RUN: llvm-mc -filetype=obj -triple=x86_64-unknown-linux %s -o %t.o
+// RUN: llvm-mc -filetype=obj -triple=x86_64-unknown-linux \
+// RUN:   %p/Inputs/whole-archive.s -o %ta.o
+// RUN: rm -f %t.a
+// RUN: llvm-ar rcs %t.a %ta.o
+
+// Should not add symbols from the archive by default as they are not required
+// RUN: lld -flavor gnu2 -o %t3 %t.o %t.a
+// RUN: llvm-readobj --symbols %t3 | FileCheck --check-prefix=NOTADDED %s
+// NOTADDED: Symbols [
+// NOTADDED-NOT: Name: _bar
+// NOTADDED: ]
+
+// Should add symbols from the archive if --whole-archive is used
+// RUN: lld -flavor gnu2 -o %t3 %t.o --whole-archive %t.a
+// RUN: llvm-readobj --symbols %t3 | FileCheck --check-prefix=ADDED %s
+// ADDED: Symbols [
+// ADDED: Name: _bar
+// ADDED: ]
+
+// --no-whole-archive should restore default behaviour
+// RUN: lld -flavor gnu2 -o %t3 %t.o --whole-archive --no-whole-archive %t.a
+// RUN: llvm-readobj --symbols %t3 | FileCheck --check-prefix=NOTADDED %s
+
+// --whole-archive and --no-whole-archive should affect only archives which follow them
+// RUN: lld -flavor gnu2 -o %t3 %t.o %t.a --whole-archive --no-whole-archive
+// RUN: llvm-readobj --symbols %t3 | FileCheck --check-prefix=NOTADDED %s
+// RUN: lld -flavor gnu2 -o %t3 %t.o --whole-archive %t.a --no-whole-archive
+// RUN: llvm-readobj --symbols %t3 | FileCheck --check-prefix=ADDED %s
+
+.globl _start;
+_start: