[ELF2] Add --[no-]whole-archive command line switches

[ikudrin]

Summary

If --whole-archive is used, all symbols from the following archives are added to the output. --no-whole-archive restores default behavior. These switches can be used multiple times.

NB. We have to keep an ArchiveFile instance within SymbolTable even if --whole-archive mode is active since it can be a thin archive which contains just names of external files. In that case actual memory buffers for the archive members will be stored within the File member of ArchiveFile class.

[grimar]

ELF/Driver.cpp

147

There is no default section. Not sure is is necessarily to use though since I guess warning about that is turned off in most cases.

ELF/InputFiles.cpp

210

Looks like that assert should be at first line. Also that would be consistent with ArchiveFile::getMembers()

[ruiu]

ELF/InputFiles.cpp

223–224

We don't have this assertion at other places in this file, so we shouldn't have this one as well. Also, nullptr != something looks inconsistent.

ELF/SymbolTable.cpp

31

You want to avoid calling parse() on an archive file if Config->WholeArchive to keep the existing behavior of parse().

[ikudrin]

• Updated to the top.
• Addressed review comments.

[ruiu]

ELF/InputFiles.cpp

183–186

The point is that we never want to call both parse() and getMembers() on the same ArchiveFile object. If you call getMembers() on an ArchiveFile, that's the end of that archive file, and we won't call any other member function after that.

Restore the original code here so that it sets this class' File pointer only when parse() is called. We don't have to set that pointer in getMembers().

186

I'd be easy to read if you write this way:

File = open(MB);

assuming you modify the function as

std::unique_ptr<Archive> open(MemoryBufferRef MB);

220–222

This guard should be removed. You can call either getMembers() or pares() but not both.

ELF/SymbolTable.cpp

34–35

Do not push archive file objects to the vector if WholeArchive is true. If --whole-archive is set, archive files are treated as if they did not exist.

36–37

Early return after this for-loop.

[ikudrin]

ELF/SymbolTable.cpp

34–35

If I understand it right, Archive::Child::getBuffer() can, under certain circumstances, create a new memory buffer and return its reference. This memory buffer is stored in the ThinBuffers member of the instance of Archive. That's why we need to hold Archive through holding ArchiveFile.

[ikudrin]

• Rebase to the top
• Fix indentation
• Sort switches
• Slightly rework ArchiveFile && add explanations why we have to keep it

[ruiu]

ELF/InputFiles.cpp

206

Remove this comment.

ELF/SymbolTable.cpp

35–38

Is that true? MemoryBuffers are not freed until the end of Driver, so it should be safe if thin archives have references to MemoryBuffers.

36–37

Please return here.

[ikudrin]

ELF/SymbolTable.cpp

35–38

A thin archive holds just names of the external files. These files are opened and their memory buffers are stored within the instance of Archive class, not in Driver.

[ruiu]

ELF/SymbolTable.cpp

35–38

I see. Please note that in the commit message and remove this comment from here.

[ikudrin]

• Addressed review comments

[ruiu]

LGTM with a nit.

ELF/Options.td

47–48

"_" is smaller than "i" in ASCIIbetical order. Move this before noinhibit_exec.

ELF/Config.h

struct Configuration {
   std::vector<llvm::StringRef> InputSearchPaths;
   bool AllowMultipleDefinition = false;
   bool DiscardAll = false;
   bool DiscardLocals = false;
   bool DiscardNone = false;
   bool ExportDynamic = false;
   bool NoInhibitExec = false;
   bool Shared = false;
+  bool WholeArchive = false;
 };
 
 extern Configuration *Config;
 
 } // namespace elf2
 } // namespace lld
 
 #endif

ELF/Driver.cpp

void LinkerDriver::link(ArrayRef<const char *> ArgsArr) {
   Config->AllowMultipleDefinition = Args.hasArg(OPT_allow_multiple_definition);
   Config->DiscardAll = Args.hasArg(OPT_discard_all);
   Config->DiscardLocals = Args.hasArg(OPT_discard_locals);
   Config->DiscardNone = Args.hasArg(OPT_discard_none);
   Config->ExportDynamic = Args.hasArg(OPT_export_dynamic);
   Config->NoInhibitExec = Args.hasArg(OPT_noinhibit_exec);
   Config->Shared = Args.hasArg(OPT_shared);
 
-  // Create a list of input files.
-  std::vector<MemoryBufferRef> Inputs;
+  // Create a symbol table.
+  SymbolTable Symtab;
 
-  for (auto *Arg : Args.filtered(OPT_l, OPT_INPUT)) {
-    StringRef Path = Arg->getValue();
-    if (Arg->getOption().getID() == OPT_l) {
-      Inputs.push_back(openFile(searchLibrary(Path)));
-      continue;
+  // Parse all input files considering corresponding command line options.
+  // Put all symbols to the symbol table which takes care of name resolution.
+  for (auto *Arg : Args) {
+    switch (Arg->getOption().getID()) {
+    case OPT_l:
+      Symtab.addFile(createFile(openFile(searchLibrary(Arg->getValue()))));
+      break;
+    case OPT_INPUT:
+      Symtab.addFile(createFile(openFile(Arg->getValue())));
+      break;
+    case OPT_whole_archive:
+      Config->WholeArchive = true;
+      break;
+    case OPT_no_whole_archive:
+      Config->WholeArchive = false;
+      break;
     }
-    Inputs.push_back(openFile(Path));
   }
 
-  if (Inputs.empty())
+  const ELFFileBase *FirstObj = Symtab.getFirstELF();
+  if (!FirstObj)
     error("no input files.");
 
-  // Create a symbol table.
-  SymbolTable Symtab;
-
-  // Parse all input files and put all symbols to the symbol table.
-  // The symbol table will take care of name resolution.
-  for (MemoryBufferRef MB : Inputs) {
-    std::unique_ptr<InputFile> File = createFile(MB);
-    Symtab.addFile(std::move(File));
-  }
-
   // Write the result.
-  const ELFFileBase *FirstObj = Symtab.getFirstELF();
   switch (FirstObj->getELFKind()) {
   case ELF32LEKind:
     writeResult<object::ELF32LE>(&Symtab);
     return;
   case ELF32BEKind:
     writeResult<object::ELF32BE>(&Symtab);
     return;
   case ELF64LEKind:
     writeResult<object::ELF64LE>(&Symtab);
     return;
   case ELF64BEKind:
     writeResult<object::ELF64BE>(&Symtab);
     return;
   }
 }

ELF/InputFiles.h

public:
   static bool classof(const InputFile *F) { return F->kind() == ArchiveKind; }
   void parse() override;
 
   // Returns a memory buffer for a given symbol. An empty memory buffer
   // is returned if we have already returned the same memory buffer.
   // (So that we don't instantiate same members more than once.)
   MemoryBufferRef getMember(const Archive::Symbol *Sym);
 
-  llvm::MutableArrayRef<Lazy> getLazySymbols() { return LazySymbols; }
+  llvm::MutableArrayRef<Lazy> getLazySymbols();
+  std::vector<MemoryBufferRef> getMembers();
 
 private:
   std::unique_ptr<Archive> File;
   std::vector<Lazy> LazySymbols;
   llvm::DenseSet<uint64_t> Seen;
 };
 
 // .so file.

ELF/InputFiles.cpp

SymbolBody *elf2::ObjectFile<ELFT>::createSymbolBody(StringRef StringTable,
   default:
     error("unexpected binding");
   case STB_GLOBAL:
   case STB_WEAK:
     return new (Alloc) DefinedRegular<ELFT>(Name, *Sym, *Sections[SecIndex]);
   }
 }
 
 void ArchiveFile::parse() {
   ErrorOr<std::unique_ptr<Archive>> ArchiveOrErr = Archive::create(MB);
   error(ArchiveOrErr, "Failed to parse archive");
   File = std::move(*ArchiveOrErr);
-
-  // Allocate a buffer for Lazy objects.
-  size_t NumSyms = File->getNumberOfSymbols();
-  LazySymbols.reserve(NumSyms);
-
-  // Read the symbol table to construct Lazy objects.
-  for (const Archive::Symbol &Sym : File->symbols())
-    LazySymbols.emplace_back(this, Sym);
 }
 
 // Returns a buffer pointing to a member file containing a given symbol.
 MemoryBufferRef ArchiveFile::getMember(const Archive::Symbol *Sym) {
   ErrorOr<Archive::child_iterator> ItOrErr = Sym->getMember();
   error(ItOrErr,
         Twine("Could not get the member for symbol ") + Sym->getName());
   Archive::child_iterator It = *ItOrErr;
 
   if (!Seen.insert(It->getChildOffset()).second)
     return MemoryBufferRef();
 
   ErrorOr<MemoryBufferRef> Ret = It->getMemoryBufferRef();
   error(Ret, Twine("Could not get the buffer for the member defining symbol ") +
                  Sym->getName());
   return *Ret;
 }
 
+llvm::MutableArrayRef<Lazy> ArchiveFile::getLazySymbols() {
+  if (!LazySymbols.empty())
+    return LazySymbols;
+
+  assert(nullptr != File && "File isn't opened, call parse() first");
+
+  // Allocate a buffer for Lazy objects.
+  size_t NumSyms = File->getNumberOfSymbols();
+  LazySymbols.reserve(NumSyms);
+
+  // Read the symbol table to construct Lazy objects.
+  for (const Archive::Symbol &Sym : File->symbols())
+    LazySymbols.emplace_back(this, Sym);
+
+  return LazySymbols;
+}
+
+std::vector<MemoryBufferRef> ArchiveFile::getMembers() {
+  assert(nullptr != File && "File isn't opened, call parse() first");
+
+  std::vector<MemoryBufferRef> Result;
+  for (const Archive::Child &Child : File->children()) {
+    ErrorOr<MemoryBufferRef> MbOrErr = Child.getMemoryBufferRef();
+    error(MbOrErr,
+          Twine("Could not get the buffer for a child of the archive ") +
+              File->getFileName());
+    Result.push_back(MbOrErr.get());
+  }
+  return Result;
+}
+
 template <class ELFT>
 SharedFile<ELFT>::SharedFile(MemoryBufferRef M)
     : SharedFileBase(getStaticELFKind<ELFT>(), M), ELFData<ELFT>(M) {}
 
 template <class ELFT> void SharedFile<ELFT>::parse() {
   for (const Elf_Shdr &Sec : this->ELFObj.sections()) {
     if (Sec.sh_type == SHT_DYNSYM) {
       this->Symtab = &Sec;

ELF/Options.td

def export_dynamic : Flag<["--"], "export-dynamic">,
      HelpText<"Put symbols in the dynamic symbol table">;
 
 def allow_multiple_definition: Flag<["--"], "allow-multiple-definition">,
     HelpText<"Allow multiple definitions">;
 
 def L : Joined<["-"], "L">, MetaVarName<"<dir>">,
      HelpText<"Directory to search for libraries">;
 
 def alias_L : Joined<["--"], "library-path=">,
      Alias<L>;
 
 def l : Joined<["-"], "l">, MetaVarName<"<libName>">,
      HelpText<"Root name of library to use">;
 
 def alias_l : Joined<["--"], "library=">,
      Alias<l>;
 
 def sysroot : Joined<["--"], "sysroot=">,
      HelpText<"Set the system root">;
 
+def whole_archive : Flag<["--"], "whole-archive">,
+     HelpText<"Force load of all members in a static library">;
+
+def no_whole_archive : Flag<["--"], "no-whole-archive">,
+     HelpText<"Restores the default behavior of loading archive members">;
+
 def entry : Separate<["--", "-"], "entry">, MetaVarName<"<entry>">,
      HelpText<"Name of entry point symbol">;
 
 def alias_entry : Separate<["-"], "e">,
      Alias<entry>;

ELF/SymbolTable.cpp

 SymbolTable::SymbolTable() {}
 
 bool SymbolTable::shouldUseRela() const {
   ELFKind K = getFirstELF()->getELFKind();
   return K == ELF64LEKind || K == ELF64BEKind;
 }
 
 void SymbolTable::addFile(std::unique_ptr<InputFile> File) {
   File->parse();
   InputFile *FileP = File.release();
   if (auto *AF = dyn_cast<ArchiveFile>(FileP)) {
     ArchiveFiles.emplace_back(AF);
+    if (Config->WholeArchive) {
+      for (MemoryBufferRef &MBRef : AF->getMembers())
+        addFile(createELFFile<ObjectFile>(MBRef));
+    } else {
       for (Lazy &Sym : AF->getLazySymbols())
         addLazy(&Sym);
+    }
     return;
   }
   addELFFile(cast<ELFFileBase>(FileP));
 }
 
 static TargetInfo *createTarget(uint16_t EMachine) {
   switch (EMachine) {
   case EM_386:

test/elf2/Inputs/whole-archive.s

+.globl _bar;
+_bar:

test/elf2/whole-archive.s

+// REQUIRES: x86
+
+// RUN: llvm-mc -filetype=obj -triple=x86_64-unknown-linux %s -o %t.o
+// RUN: llvm-mc -filetype=obj -triple=x86_64-unknown-linux \
+// RUN:     %p/Inputs/whole-archive.s -o %ta.o
+// RUN: rm -f %t.a
+// RUN: llvm-ar rcs %t.a %ta.o
+
+// Should not add symbols from the archive by default as they are not required
+// RUN: lld -flavor gnu2 -o %t3 %t.o %t.a
+// RUN: llvm-readobj --symbols %t3 | FileCheck --check-prefix=NOTADDED %s
+// NOTADDED: Symbols [
+// NOTADDED-NOT: Name: _bar
+// NOTADDED: ]
+
+// Should add symbols from the archive if --whole-archive is used
+// RUN: lld -flavor gnu2 -o %t3 %t.o --whole-archive %t.a
+// RUN: llvm-readobj --symbols %t3 | FileCheck --check-prefix=ADDED %s
+// ADDED: Symbols [
+// ADDED: Name: _bar
+// ADDED: ]
+
+// --no-whole-archive should restore default behaviour
+// RUN: lld -flavor gnu2 -o %t3 %t.o --whole-archive --no-whole-archive %t.a
+// RUN: llvm-readobj --symbols %t3 | FileCheck --check-prefix=NOTADDED %s
+
+// --whole-archive and --no-whole-archive should affect only archives which follow them
+// RUN: lld -flavor gnu2 -o %t3 %t.o %t.a --whole-archive --no-whole-archive
+// RUN: llvm-readobj --symbols %t3 | FileCheck --check-prefix=NOTADDED %s
+// RUN: lld -flavor gnu2 -o %t3 %t.o --whole-archive %t.a --no-whole-archive
+// RUN: llvm-readobj --symbols %t3 | FileCheck --check-prefix=ADDED %s
+
+.globl _start;
+_start: