[ELF2] - Implemented --exclude-libs flag

[grimar]

Summary

Specifies a list of archive libraries from which symbols should not be automatically exported. The library names may be delimited by commas. Specifying "--exclude-libs ALL" excludes symbols in all archive libraries from automatic export. For ELF targeted ports, symbols affected by this option will be treated as hidden.

Notes:

• Original flag description contains "the library names may be delimited by commas or colons". This implemention handles only commas. Not sure do I need to add colons handling ? Never see them anywhere in use for any command line I think.

[grimar]

ELF/Driver.cpp

9

Redundant declaration of Libs. Temporary code, will remove.

[ruiu]

I'm still reviewing this change, but I guess my question I have now is how important this feature is. Do you need this now, or is this high priority? New ELF LLD is still very early, and all code is likely to be refactored or modified, so I wouldn't want to add that much amount of code unless a feature is implemented pretty easily or important.

ELF/Config.h

2

Just use std::vector.

ELF/Driver.cpp

8–10

SmallVector<StringRef, 5> V;
Arg->getValue().split(V, ",", -1, false);
Config->ExcludeLibs.append(Config->ExcludeLibs.end(), V.begin(), V.end());

ELF/InputFiles.h

7

SymbolsHidden = false;

7–8

Do not define accessors if both are visible. They don't provide additional value. Just make the field public.

8

bool HideSymbols = true;

[grimar]

Review comments addressed.

[grimar]

In D13392#258851, @ruiu wrote:

I'm still reviewing this change, but I guess my question I have now is how important this feature is. Do you need this now, or is this high priority? New ELF LLD is still very early, and all code is likely to be refactored or modified, so I wouldn't want to add that much amount of code unless a feature is implemented pretty easily or important.

Not sure how this feature is important and I definitely dont need it right now.
I just thought it could be useful. At least from design side, before this patch there was no possibility to determine to which archive currently parced file belongs to.
Further modifications and refactorings at least should take care of that. All other changes performed seems minor and/or local for that flag to me.
Just in case I updated the diff to address all comments and be in touch with latest revision.
But If you think its too early fot that - please let me know and I will abandon the diff.

[bero]

This patch would be very useful for us: The Android build system relies on --exclude-libs ALL in several places (e.g. Bionic). We currently have an ongoing project to get AOSP to build entirely with lld (currently we're at a point where we use lld for binaries and keep using gold for shared libraries because of the --exclude-libs issue and a few problems we still have to hunt down).

[ruiu]

If Android needs the flag, I'm happy to re-implement this feature, but can you give me a pointer where the flag is used in Android? I want to understand how the flag is being used before implementing it.

[bero]

Thanks! This is where it's currently being used (sometimes with a comment explaining the use):
https://android.googlesource.com/platform/bionic/+/master/libdl/Android.bp#18
https://android.googlesource.com/platform/bionic/+/master/libdl/Android.bp#37
https://android.googlesource.com/platform/bionic/+/master/libdl/Android.bp#43
https://android.googlesource.com/platform/bionic/+/master/libdl/Android.bp#86
https://android.googlesource.com/platform/bionic/+/master/libdl/Android.bp#91
https://android.googlesource.com/platform/bionic/+/master/libdl/Android.bp#94
https://android.googlesource.com/platform/bionic/+/master/linker/Android.bp#80
https://android.googlesource.com/platform/external/compiler-rt/+/master/Android.bp#376
https://android.googlesource.com/platform/external/compiler-rt/+/master/Android.bp#380
https://android.googlesource.com/platform/external/compiler-rt/+/master/Android.bp#384
https://android.googlesource.com/platform/external/compiler-rt/+/master/Android.bp#388
https://android.googlesource.com/platform/external/compiler-rt/+/master/Android.bp#392
https://android.googlesource.com/platform/external/compiler-rt/+/master/Android.bp#397
https://android.googlesource.com/platform/external/cblas/+/master/Android.bp#181
https://android.googlesource.com/platform/external/libcxx/+/master/Android.bp#89
https://android.googlesource.com/platform/frameworks/native/+/master/opengl/libs/Android.bp#110
https://android.googlesource.com/platform/frameworks/rs/+/master/support.bp#167
https://android.googlesource.com/platform/frameworks/rs/+/master/cpp/Android.bp#85
https://android.googlesource.com/platform/frameworks/rs/+/master/support/jni/Android.bp#33
https://android.googlesource.com/platform/frameworks/rs/+/master/support/jni/Android.bp#71

[ruiu]

I re-implemented the feature in https://reviews.llvm.org/D34422. The new implementation uses a different strategy than this patch, and by utilizing the relationship of archive files and object files, the new implementation is much simpler and less intrusive than this one.

[ruiu]

Bernhard,

I landed https://reviews.llvm.org/rL305920 to add the -exclude-libs flag.

ELF/Config.h

 //===- Config.h -------------------------------------------------*- C++ -*-===//
 //
 //                             The LLVM Linker
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLD_ELF_CONFIG_H
 #define LLD_ELF_CONFIG_H
 
 #include "llvm/ADT/StringRef.h"
 
 #include <vector>
 
 namespace lld {
 namespace elf2 {
 
 struct Configuration {
   llvm::StringRef DynamicLinker;
   llvm::StringRef Entry;
   llvm::StringRef OutputFile = "a.out";
   llvm::StringRef SoName;
   llvm::StringRef Sysroot;
   std::string RPath;
+  std::vector<llvm::StringRef> ExcludeLibs;
   std::vector<llvm::StringRef> InputSearchPaths;
   bool AllowMultipleDefinition;
   bool DiscardAll;
   bool DiscardLocals;
   bool DiscardNone;
   bool ExportDynamic;
   bool NoInhibitExec;
   bool NoUndefined;

ELF/Driver.cpp

 //===- Driver.cpp ---------------------------------------------------------===//
 //
 //                             The LLVM Linker
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "Driver.h"
 #include "Config.h"
 #include "Error.h"
 #include "InputFiles.h"
 #include "SymbolTable.h"
 #include "Writer.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/Support/FileSystem.h"
for (auto *Arg : Args.filtered(OPT_L))
     Config->InputSearchPaths.push_back(Arg->getValue());
 
   if (auto *Arg = Args.getLastArg(OPT_entry))
     Config->Entry = Arg->getValue();
 
   if (auto *Arg = Args.getLastArg(OPT_soname))
     Config->SoName = Arg->getValue();
 
+  if (auto *Arg = Args.getLastArg(OPT_exclude_libs)) {
+    SmallVector<StringRef, 5> V;
+    ((StringRef)Arg->getValue()).split(V, ",", -1, false);
+    Config->ExcludeLibs.insert(Config->ExcludeLibs.end(), V.begin(), V.end());
+  }
+
   Config->AllowMultipleDefinition = Args.hasArg(OPT_allow_multiple_definition);
   Config->DiscardAll = Args.hasArg(OPT_discard_all);
   Config->DiscardLocals = Args.hasArg(OPT_discard_locals);
   Config->DiscardNone = Args.hasArg(OPT_discard_none);
   Config->ExportDynamic = Args.hasArg(OPT_export_dynamic);
   Config->NoInhibitExec = Args.hasArg(OPT_noinhibit_exec);
   Config->NoUndefined = Args.hasArg(OPT_no_undefined);
   Config->Shared = Args.hasArg(OPT_shared);

ELF/InputFiles.h

 //===- InputFiles.h ---------------------------------------------*- C++ -*-===//
 //
 //                             The LLVM Linker
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLD_ELF_INPUT_FILES_H
 #define LLD_ELF_INPUT_FILES_H
 
 #include "InputSection.h"
 #include "Error.h"
 #include "Symbols.h"
 
public:
   static bool classof(const InputFile *F) {
     return F->kind() == ObjectKind &&
            cast<ELFFileBase>(F)->getELFKind() == getStaticELFKind<ELFT>();
   }
 
   explicit ObjectFile(MemoryBufferRef M);
   void parse() override;
 
+  void hideAllSymbols() { HideSymbols = true; }
+
   ArrayRef<InputSection<ELFT> *> getSections() const { return Sections; }
 
   SymbolBody *getSymbolBody(uint32_t SymbolIndex) const {
     uint32_t FirstNonLocal = this->Symtab->sh_info;
     if (SymbolIndex < FirstNonLocal)
       return nullptr;
     return SymbolBodies[SymbolIndex - FirstNonLocal]->getReplacement();
   }
 
   Elf_Sym_Range getLocalSymbols();
 
   const Elf_Shdr *getSymbolTable() const { return this->Symtab; };
   ArrayRef<Elf_Word> getSymbolTableShndx() const { return SymtabSHNDX; };
 
 private:
   void initializeSections();
   void initializeSymbols();
 
   SymbolBody *createSymbolBody(StringRef StringTable, const Elf_Sym *Sym);
 
   // List of all sections defined by this file.
   std::vector<InputSection<ELFT> *> Sections;
 
   ArrayRef<Elf_Word> SymtabSHNDX;
+
+  bool HideSymbols = false;
 };
 
 class ArchiveFile : public InputFile {
 public:
   explicit ArchiveFile(MemoryBufferRef M) : InputFile(ArchiveKind, M) {}
   static bool classof(const InputFile *F) { return F->kind() == ArchiveKind; }
   void parse() override;
 
   // Returns a memory buffer for a given symbol. An empty memory buffer
   // is returned if we have already returned the same memory buffer.
   // (So that we don't instantiate same members more than once.)
   MemoryBufferRef getMember(const Archive::Symbol *Sym);
 
   llvm::MutableArrayRef<Lazy> getLazySymbols() { return LazySymbols; }
   std::vector<MemoryBufferRef> getMembers();
 
+  bool SymbolsHidden = false;
+
 private:
   std::unique_ptr<Archive> File;
   std::vector<Lazy> LazySymbols;
   llvm::DenseSet<uint64_t> Seen;
 };
 
 // .so file.
 class SharedFileBase : public ELFFileBase {

ELF/InputFiles.cpp

template <class ELFT> void elf2::ObjectFile<ELFT>::initializeSections() {
   }
 }
 
 template <class ELFT> void elf2::ObjectFile<ELFT>::initializeSymbols() {
   this->initStringTable();
   Elf_Sym_Range Syms = this->getNonLocalSymbols();
   uint32_t NumSymbols = std::distance(Syms.begin(), Syms.end());
   SymbolBodies.reserve(NumSymbols);
-  for (const Elf_Sym &Sym : Syms)
-    SymbolBodies.push_back(createSymbolBody(this->StringTable, &Sym));
+  for (const Elf_Sym &Sym : Syms) {
+    SymbolBody *Body = createSymbolBody(this->StringTable, &Sym);
+    if (HideSymbols) {
+      uint8_t Visibility = Body->getMostConstrainingVisibility();
+      // the attributes, ordered from least to most constraining: STV_PROTECTED,
+      // STV_HIDDEN, STV_INTERNAL.
+      if (Visibility == STV_DEFAULT || Visibility == STV_PROTECTED)
+        Visibility = STV_HIDDEN;
+      Body->setMostConstrainingVisibility(Visibility);
+    }
+    SymbolBodies.push_back(Body);
+  }
 }
 
 template <class ELFT>
 SymbolBody *elf2::ObjectFile<ELFT>::createSymbolBody(StringRef StringTable,
                                                      const Elf_Sym *Sym) {
   ErrorOr<StringRef> NameOrErr = Sym->getName(StringTable);
   error(NameOrErr.getError());
   StringRef Name = *NameOrErr;
void ArchiveFile::parse() {
   File = openArchive(MB);
 
   // Allocate a buffer for Lazy objects.
   size_t NumSyms = File->getNumberOfSymbols();
   LazySymbols.reserve(NumSyms);
 
   // Read the symbol table to construct Lazy objects.
   for (const Archive::Symbol &Sym : File->symbols())
-    LazySymbols.emplace_back(this, Sym);
+    LazySymbols.emplace_back(this, Sym, SymbolsHidden ? llvm::ELF::STV_HIDDEN
+                                                      : llvm::ELF::STV_DEFAULT);
 }
 
 // Returns a buffer pointing to a member file containing a given symbol.
 MemoryBufferRef ArchiveFile::getMember(const Archive::Symbol *Sym) {
   ErrorOr<Archive::child_iterator> ItOrErr = Sym->getMember();
   error(ItOrErr,
         Twine("Could not get the member for symbol ") + Sym->getName());
   Archive::child_iterator It = *ItOrErr;

ELF/Options.td

def discard_none : Flag<["-"], "discard-none">,
   HelpText<"Keep all symbols in the symbol table">;
 
 def dynamic_linker : Separate<["--", "-"], "dynamic-linker">,
   HelpText<"Which dynamic linker to use">;
 
 def entry : Separate<["--", "-"], "entry">, MetaVarName<"<entry>">,
   HelpText<"Name of entry point symbol">;
 
+def exclude_libs : Separate<["--"], "exclude-libs">, MetaVarName<"<exclude_libs>">,
+  HelpText<"Specifies a list of archives from which symbols should not be automatically exported">;
+
 def export_dynamic : Flag<["--", "-"], "export-dynamic">,
   HelpText<"Put symbols in the dynamic symbol table">;
 
 def l : Joined<["-"], "l">, MetaVarName<"<libName>">,
   HelpText<"Root name of library to use">;
 
 def no_allow_shlib_undefined : Flag<["--"], "no-allow-shlib-undefined">;
 

ELF/SymbolTable.h

 // conflicts. For example, obviously, a defined symbol is better than
 // an undefined symbol. Or, if there's a conflict between a lazy and a
 // undefined, it'll read an archive member to read a real definition
 // to replace the lazy symbol. The logic is implemented in resolve().
 class SymbolTable {
 public:
   SymbolTable();
 
-  void addFile(std::unique_ptr<InputFile> File);
+  void addFile(std::unique_ptr<InputFile> File,
+               ArchiveFile *HeadArchive = nullptr);
 
   const ELFFileBase *getFirstELF() const {
     if (!ObjectFiles.empty())
       return ObjectFiles[0].get();
     if (!SharedFiles.empty())
       return SharedFiles[0].get();
     return nullptr;
   }
void addSyntheticSym(StringRef Name, OutputSection<ELFT> &Section,
                        typename llvm::object::ELFFile<ELFT>::uintX_t Value);
 
   template <class ELFT> void addIgnoredSym(StringRef Name);
 
 private:
   Symbol *insert(SymbolBody *New);
   template <class ELFT> void addELFFile(ELFFileBase *File);
   void addELFFile(ELFFileBase *File);
-  void addLazy(Lazy *New);
+  void addLazy(Lazy *New, ArchiveFile *HeadArchive);
   void addMemberFile(Lazy *Body);
+  void addArchiveMemberFile(Lazy *Body, ArchiveFile *HeadArchive);
 
   template <class ELFT> void init(uint16_t EMachine);
   template <class ELFT> void resolve(SymbolBody *Body);
   template <class ELFT>
   void dupError(const SymbolBody &Old, const SymbolBody &New);
 
   std::vector<std::unique_ptr<ArchiveFile>> ArchiveFiles;
 

ELF/SymbolTable.cpp

 //===- SymbolTable.cpp ----------------------------------------------------===//
 //
 //                             The LLVM Linker
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "SymbolTable.h"
 #include "Config.h"
 #include "Error.h"
 #include "Symbols.h"
 #include "Target.h"
+#include "llvm\Support\Path.h"
 
 using namespace llvm;
 using namespace llvm::object;
 using namespace llvm::ELF;
 
 using namespace lld;
 using namespace lld::elf2;
 
 SymbolTable::SymbolTable() {}
 
 bool SymbolTable::shouldUseRela() const {
   ELFKind K = getFirstELF()->getELFKind();
   return K == ELF64LEKind || K == ELF64BEKind;
 }
 
-void SymbolTable::addFile(std::unique_ptr<InputFile> File) {
+namespace {
+bool shouldExcludeLibs(InputFile *FileP) {
+  if (Config->ExcludeLibs.empty())
+    return false;
+  if (Config->ExcludeLibs.size() == 1 && Config->ExcludeLibs[0] == "ALL")
+    return true;
+
+  StringRef FileName = llvm::sys::path::filename(FileP->getName());
+  auto Found =
+      std::find_if(Config->ExcludeLibs.begin(), Config->ExcludeLibs.end(),
+                   [&FileName](const StringRef &R) { return R == FileName; });
+  return (Found != Config->ExcludeLibs.end());
+}
+
+template <class ELFT> void setSymbolsHidden(InputFile *File) {
+  if (ObjectFile<ELFT> *O = dyn_cast<ObjectFile<ELFT>>(File)) {
+    O->hideAllSymbols();
+    return;
+  }
+  assert(!"Unexpected content of archive.");
+}
+}
+
+void SymbolTable::addFile(std::unique_ptr<InputFile> File,
+                          ArchiveFile *HeadArchive) {
   if (auto *AF = dyn_cast<ArchiveFile>(File.get())) {
     File.release();
     ArchiveFiles.emplace_back(AF);
     if (Config->WholeArchive) {
       for (MemoryBufferRef &MBRef : AF->getMembers())
-        addFile(createELFFile<ObjectFile>(MBRef));
+        addFile(createELFFile<ObjectFile>(MBRef), HeadArchive);
       return;
     }
+    AF->SymbolsHidden = shouldExcludeLibs(HeadArchive ? HeadArchive : AF);
     AF->parse();
+    ArchiveFile *Container = HeadArchive ? HeadArchive : AF;
     for (Lazy &Sym : AF->getLazySymbols())
-      addLazy(&Sym);
+      addLazy(&Sym, Container);
     return;
   }
   if (auto *S = dyn_cast<SharedFileBase>(File.get())) {
     S->parseSoName();
     if (!IncludedSoNames.insert(S->getSoName()).second)
       return;
   }
   File->parse();
if (!Sym) {
     Sym = new (Alloc) Symbol(New);
     New->setBackref(Sym);
     return Sym;
   }
   New->setBackref(Sym);
   return Sym;
 }
 
-void SymbolTable::addLazy(Lazy *New) {
+void SymbolTable::addLazy(Lazy *New, ArchiveFile *HeadArchive) {
   Symbol *Sym = insert(New);
   if (Sym->Body == New)
     return;
   SymbolBody *Existing = Sym->Body;
   if (Existing->isDefined() || Existing->isLazy() || Existing->isWeak())
     return;
   Sym->Body = New;
   assert(Existing->isUndefined() && "Unexpected symbol kind.");
-  addMemberFile(New);
+  addArchiveMemberFile(New, HeadArchive);
+}
+
+void SymbolTable::addArchiveMemberFile(Lazy *Body, ArchiveFile *HeadArchive) {
+  std::unique_ptr<InputFile> File = Body->getMember();
+  // getMember returns nullptr if the member was already read from the library.
+  if (!File)
+    return;
+
+  if (HeadArchive->SymbolsHidden) {
+    InputFile *FileP = File.get();
+    switch (cast<ELFFileBase>(FileP)->getELFKind()) {
+    case ELF32LEKind:
+      setSymbolsHidden<object::ELF32LE>(FileP);
+      break;
+    case ELF32BEKind:
+      setSymbolsHidden<object::ELF32BE>(FileP);
+      break;
+    case ELF64LEKind:
+      setSymbolsHidden<object::ELF64LE>(FileP);
+      break;
+    case ELF64BEKind:
+      setSymbolsHidden<object::ELF64BE>(FileP);
+      break;
+    default:
+      assert("Unsupported ELF kind.");
+      break;
+    }
+  }
+
+  addFile(std::move(File), HeadArchive);
 }
 
 void SymbolTable::addMemberFile(Lazy *Body) {
   std::unique_ptr<InputFile> File = Body->getMember();
 
   // getMember returns nullptr if the member was already read from the library.
   if (!File)
     return;

ELF/Symbols.h

public:
 
   // Returns the symbol name.
   StringRef getName() const { return Name; }
 
   uint8_t getMostConstrainingVisibility() const {
     return MostConstrainingVisibility;
   }
 
+  void setMostConstrainingVisibility(uint8_t V) {
+    MostConstrainingVisibility = V;
+  }
+
   unsigned getDynamicSymbolTableIndex() const {
     return DynamicSymbolTableIndex;
   }
   void setDynamicSymbolTableIndex(unsigned V) { DynamicSymbolTableIndex = V; }
 
   unsigned getGotIndex() const { return GotIndex; }
   bool isInGot() const { return GotIndex != -1U; }
   void setGotIndex(unsigned I) { GotIndex = I; }

 
 // This class represents a symbol defined in an archive file. It is
 // created from an archive file header, and it knows how to load an
 // object file from an archive to replace itself with a defined
 // symbol. If the resolver finds both Undefined and Lazy for
 // the same name, it will ask the Lazy to load a file.
 class Lazy : public SymbolBody {
 public:
-  Lazy(ArchiveFile *F, const llvm::object::Archive::Symbol S)
-      : SymbolBody(LazyKind, S.getName(), false, llvm::ELF::STV_DEFAULT),
-        File(F), Sym(S) {}
+  Lazy(ArchiveFile *F, const llvm::object::Archive::Symbol S,
+       uint8_t Visibility = llvm::ELF::STV_DEFAULT)
+      : SymbolBody(LazyKind, S.getName(), false, Visibility), File(F), Sym(S) {}
 
   static bool classof(const SymbolBody *S) { return S->kind() == LazyKind; }
 
   // Returns an object file for this symbol, or a nullptr if the file
   // was already returned.
   std::unique_ptr<InputFile> getMember();
 
 private:
   ArchiveFile *File;
   const llvm::object::Archive::Symbol Sym;
 };
 
 } // namespace elf2
 } // namespace lld
 
 #endif

test/elf2/Inputs/exclude-libs.s

+.globl _usedFunction
+.type _usedFunction, @function
+  _usedFunction:

test/elf2/exclude-libs.s

+// REQUIRES: x86
+// RUN: llvm-mc -filetype=obj -triple=x86_64-unknown-linux %s -o %tmain.o
+// RUN: llvm-mc -filetype=obj -triple=x86_64-unknown-linux \
+// RUN:   %p/Inputs/exclude-libs.s -o %ta.o
+// RUN: rm -f %t.a
+// RUN: llvm-ar rcs %t.a %ta.o
+
+// without --exclude-libs flag symbol from archive should appear
+// both in .symtab and .dynsym.
+// RUN: lld -shared -flavor gnu2 %tmain.o %t.a -o %t.out
+// RUN: llvm-readobj -symbols -dyn-symbols %t.out | FileCheck --check-prefix=YSYM %s
+
+// --exclude-libs hides symbols from all archives. Here check is performed for one existent.
+// RUN: lld --exclude-libs ALL -shared -flavor gnu2 %tmain.o %t.a -o %t.out
+// RUN: llvm-readobj -symbols -dyn-symbols %t.out | FileCheck --check-prefix=NSYM %s
+
+// after excluding fake archives symbols should not be hidden.
+// RUN: lld --exclude-libs libFake1.a,libFake2.a,libFake3.a -shared -flavor gnu2 %tmain.o %t.a -o %t.out
+// RUN: llvm-readobj -symbols -dyn-symbols %t.out | FileCheck --check-prefix=YSYM %s
+
+// YSYM: Symbols [
+// YSYM: Name: _usedFunction
+// YSYM: ]
+// YSYM: DynamicSymbols [
+// YSYM: Name: _usedFunction
+// YSYM: ]
+
+// NSYM: Symbols [
+// NSYM-NOT: Name: _usedFunction
+// NSYM: ]
+// NSYM: DynamicSymbols [
+// NSYM-NOT: Name: _usedFunction
+// NSYM: ]
+
+.globl _start
+_start:
+  call _usedFunction