[ELF2] Add --undefined option

[denis-protivensky]

Summary

Add symbol specified with -u as undefined which may cause additional object files from archives to be linked into the resulting binary.

[ruiu]

ELF/SymbolTable.cpp

82–83 ↗

(On Diff #36236)

nit: If it gets two lines, I'd write like this

resolve<ELFT>(
    new (Alloc) ...);

83 ↗

(On Diff #36236)

Why optional? All symbols specified with -u must be resolved, no?

ELF/Symbols.h

250–263 ↗

(On Diff #36236)

This seems overkill because I don't think we are going to allocate a lot of Synthetic Undefined symbols. Can you simply make these static fields non-static? It's going to allocate a Elf_Sym per Undefined, but that wouldn't really affect anything to performance.

[denis-protivensky]

ELF/SymbolTable.cpp

83 ↗

(On Diff #36236)

As I understand, the doc says it shouldn't make linking process degrade, I mean, it shouldn't cause link errors if some of these undefined symbols are not resolved. The only purpose of this is to trigger adding of object files from archives if they contain symbols specified.
These symbols technically don't have any logical impact on binary, so with these left undefined there won't be any changes in runtime behavior since these undefines are not used by any code.

ELF/Symbols.h

250–263 ↗

(On Diff #36236)

Good.

[ruiu]

ELF/SymbolTable.cpp

83 ↗

(On Diff #36236)

Yeah, that's right.

Is there any reason we can't merge canKeepUndefined with isWeak?

[• rafael]

Change the last test to create a shared library and check the the 'unknown' symbol shows up only on the regular symbol table, not the dynamic one.

[denis-protivensky]

In D13345#257809, @rafael wrote:

Change the last test to create a shared library and check the the 'unknown' symbol shows up only on the regular symbol table, not the dynamic one.

Good.

ELF/SymbolTable.cpp

83 ↗

(On Diff #36236)

We can't use weak undefined symbol to trigger resolution and addition of object files from archives.
This is explicitly stated in the ELF reference:
"The link editor does not extract archive members to resolve undefined weak symbols."

ELF/Symbols.h

250–263 ↗

(On Diff #36236)

Looks like I won't be able to rework this fully. I need a flag to indicate that the symbol is not "regular" undefined, but a special one added with -u. This flag is the SyntheticOptional symbol itself, and I check its address with what current Undefined object holds in canKeepUndefined. Otherwise I'll need to hold std::unique_ptr with synthetic symbol generated dynamically, for example, and check that it's not empty, but I won't be able to distinguish between entry point and --undefined added symbol (they now have two separate static symbols SyntheticRequired and SyntheticOptional).
I'll move static initializer to a better place though.

[denis-protivensky]

Updated:

• simplify initialization of static synthetic symbol
• add test to check that undefined symbol is not in dynamic table

[• rafael]

LGTM

[ruiu]

LGTM

ELF/SymbolTable.cpp

91 ↗

(On Diff #36356)

This is not guaranteed to be thread-safe since if this function is called simultaneously, two assignments could run at the same time, and although that looks benign, that's a race condition.

It is OK for now. We don't do multi-threading soon. We probably want to have a init() function for each file which initialize static members. We can do that later.

[• rafael]

It is OK for now. We don't do multi-threading soon. We probably want to have a init() function for each file which initialize static members. We can do that later.

It should be possible to statically initialize this somehow, but need
to look deeper a bit on why just

template <class ELFT>
typename DefinedAbsolute<ELFT>::Elf_Sym
DefinedAbsolute<ELFT>::IgnoreUndef = {0, 0, ..., 2, 0... };

Doesn't compile.

Cheers,
Rafael

lld/trunk/ELF/Driver.cpp

for (auto *Arg : Args) {
     default:
       break;
     }
   }
 
   if (Symtab.getObjectFiles().empty())
     error("no input files.");
 
+  for (auto *Arg : Args.filtered(OPT_undefined))
+    Symtab.addUndefinedSym(Arg->getValue());
+
   // Write the result.
   const ELFFileBase *FirstObj = Symtab.getFirstELF();
   switch (FirstObj->getELFKind()) {
   case ELF32LEKind:
     writeResult<object::ELF32LE>(&Symtab);
     return;
   case ELF32BEKind:
     writeResult<object::ELF32BE>(&Symtab);

lld/trunk/ELF/Options.td

def shared : Flag<["-"], "shared">,
   HelpText<"Build a shared object">;
 
 def soname : Joined<["-"], "soname=">,
   HelpText<"Set DT_SONAME">;
 
 def sysroot : Joined<["--"], "sysroot=">,
   HelpText<"Set the system root">;
 
+def undefined : Joined<["--"], "undefined=">,
+  HelpText<"Force undefined symbol during linking">;
+
 def whole_archive : Flag<["--"], "whole-archive">,
   HelpText<"Force load of all members in a static library">;
 
 // Aliases
 def alias_Bdynamic_call_shared: Flag<["-"], "call_shared">, Alias<Bdynamic>;
 def alias_Bdynamic_dy: Flag<["-"], "dy">, Alias<Bdynamic>;
 def alias_Bstatic_dn: Flag<["-"], "dn">, Alias<Bstatic>;
 def alias_Bstatic_non_shared: Flag<["-"], "non_shared">, Alias<Bstatic>;
 def alias_Bstatic_static: Flag<["-"], "static">, Alias<Bstatic>;
 def alias_L__library_path : Joined<["--"], "library-path=">, Alias<L>;
 def alias_discard_all_x: Flag<["-"], "x">, Alias<discard_all>;
 def alias_discard_locals_X: Flag<["-"], "X">, Alias<discard_locals>;
 def alias_entry_e : Separate<["-"], "e">, Alias<entry>;
 def alias_l__library : Joined<["--"], "library=">, Alias<l>;
 def alias_rpath_rpath : Joined<["-"], "rpath=">, Alias<rpath>;
 def alias_soname_h : Separate<["-"], "h">, Alias<soname>;
 def alias_soname_soname : Separate<["-"], "soname">, Alias<soname>;
+def alias_undefined_u : Separate<["-"], "u">, Alias<undefined>;
 
 // Options listed below are silently ignored now.
 def O3 : Flag<["-"], "O3">;
 def as_needed : Flag<["--"], "as-needed">;
 def build_id : Flag<["--"], "build-id">;
 def eh_frame_hdr : Flag<["--"], "eh-frame-hdr">;
 def enable_new_dtags : Flag<["--"], "enable-new-dtags">;
 def end_group : Flag<["--"], "end-group">;

lld/trunk/ELF/SymbolTable.h

public:
   }
 
   SymbolBody *getEntrySym() const {
     if (!EntrySym)
       return nullptr;
     return EntrySym->getReplacement();
   }
 
+  void addUndefinedSym(StringRef Name);
+
   template <class ELFT>
   void addSyntheticSym(StringRef Name, OutputSection<ELFT> &Section,
                        typename llvm::object::ELFFile<ELFT>::uintX_t Value);
 
   template <class ELFT> void addIgnoredSym(StringRef Name);
 
 private:
   Symbol *insert(SymbolBody *New);
   template <class ELFT> void addELFFile(ELFFileBase *File);
   void addELFFile(ELFFileBase *File);
   void addLazy(Lazy *New);
   void addMemberFile(Lazy *Body);
+  template <class ELFT> void addUndefinedSym(StringRef Name);
 
   template <class ELFT> void init(uint16_t EMachine);
   template <class ELFT> void resolve(SymbolBody *Body);
   template <class ELFT>
   void dupError(const SymbolBody &Old, const SymbolBody &New);
 
   std::vector<std::unique_ptr<ArchiveFile>> ArchiveFiles;
 

lld/trunk/ELF/SymbolTable.cpp

static TargetInfo *createTarget(uint16_t EMachine) {
   case EM_PPC64:
     return new PPC64TargetInfo();
   case EM_X86_64:
     return new X86_64TargetInfo();
   }
   error("Unknown target machine");
 }
 
+void SymbolTable::addUndefinedSym(StringRef Name) {
+  switch (getFirstELF()->getELFKind()) {
+  case ELF32LEKind:
+    addUndefinedSym<ELF32LE>(Name);
+    break;
+  case ELF32BEKind:
+    addUndefinedSym<ELF32BE>(Name);
+    break;
+  case ELF64LEKind:
+    addUndefinedSym<ELF64LE>(Name);
+    break;
+  case ELF64BEKind:
+    addUndefinedSym<ELF64BE>(Name);
+    break;
+  }
+}
+
+template <class ELFT> void SymbolTable::addUndefinedSym(StringRef Name) {
+  Undefined<ELFT>::SyntheticOptional.setVisibility(STV_HIDDEN);
+  resolve<ELFT>(new (Alloc)
+                    Undefined<ELFT>(Name, Undefined<ELFT>::SyntheticOptional));
+}
+
 template <class ELFT>
 void SymbolTable::addSyntheticSym(StringRef Name, OutputSection<ELFT> &Section,
                                   typename ELFFile<ELFT>::uintX_t Value) {
   typedef typename DefinedSynthetic<ELFT>::Elf_Sym Elf_Sym;
   auto ESym = new (Alloc) Elf_Sym;
   memset(ESym, 0, sizeof(Elf_Sym));
   ESym->st_value = Value;
   auto Sym = new (Alloc) DefinedSynthetic<ELFT>(Name, *ESym, Section);
   resolve<ELFT>(Sym);
 }
 
 template <class ELFT> void SymbolTable::addIgnoredSym(StringRef Name) {
   DefinedAbsolute<ELFT>::IgnoreUndef.setVisibility(STV_HIDDEN);
   auto Sym = new (Alloc)
       DefinedAbsolute<ELFT>(Name, DefinedAbsolute<ELFT>::IgnoreUndef);
   resolve<ELFT>(Sym);
 }
 
 template <class ELFT> void SymbolTable::init(uint16_t EMachine) {
   Target.reset(createTarget(EMachine));
   if (Config->Shared)
     return;
   EntrySym = new (Alloc) Undefined<ELFT>(
       Config->Entry.empty() ? Target->getDefaultEntry() : Config->Entry,
-      Undefined<ELFT>::Synthetic);
+      Undefined<ELFT>::SyntheticRequired);
   resolve<ELFT>(EntrySym);
 
   // In the assembly for 32 bit x86 the _GLOBAL_OFFSET_TABLE_ symbol is magical
   // and is used to produce a R_386_GOTPC relocation.
   // The R_386_GOTPC relocation value doesn't actually depend on the
   // symbol value, so it could use an index of STN_UNDEF which, according to the
   // spec, means the symbol value is 0.
   // Unfortunately both gas and MC keep the _GLOBAL_OFFSET_TABLE_ symbol in

lld/trunk/ELF/Symbols.h

enum Kind {
     LazyKind
   };
 
   Kind kind() const { return static_cast<Kind>(SymbolKind); }
 
   bool isWeak() const { return IsWeak; }
   bool isUndefined() const { return SymbolKind == UndefinedKind; }
   bool isDefined() const { return SymbolKind <= DefinedLast; }
-  bool isStrongUndefined() const { return !IsWeak && isUndefined(); }
   bool isCommon() const { return SymbolKind == DefinedCommonKind; }
   bool isLazy() const { return SymbolKind == LazyKind; }
   bool isShared() const { return SymbolKind == SharedKind; }
   bool isUsedInRegularObj() const { return IsUsedInRegularObj; }
   bool isUsedInDynamicReloc() const { return IsUsedInDynamicReloc; }
   void setUsedInDynamicReloc() { IsUsedInDynamicReloc = true; }
 
   // Returns the symbol name.

 };
 
 // Undefined symbol.
 template <class ELFT> class Undefined : public ELFSymbolBody<ELFT> {
   typedef ELFSymbolBody<ELFT> Base;
   typedef typename Base::Elf_Sym Elf_Sym;
 
 public:
-  static Elf_Sym Synthetic;
+  static Elf_Sym SyntheticRequired;
+  static Elf_Sym SyntheticOptional;
 
   Undefined(StringRef N, const Elf_Sym &Sym)
       : ELFSymbolBody<ELFT>(Base::UndefinedKind, N, Sym) {}
 
   static bool classof(const SymbolBody *S) {
     return S->kind() == Base::UndefinedKind;
   }
+
+  bool canKeepUndefined() const { return &this->Sym == &SyntheticOptional; }
 };
 
 template <class ELFT>
-typename Undefined<ELFT>::Elf_Sym Undefined<ELFT>::Synthetic;
+typename Undefined<ELFT>::Elf_Sym Undefined<ELFT>::SyntheticRequired;
+template <class ELFT>
+typename Undefined<ELFT>::Elf_Sym Undefined<ELFT>::SyntheticOptional;
 
 template <class ELFT> class SharedSymbol : public Defined<ELFT> {
   typedef Defined<ELFT> Base;
   typedef typename Base::Elf_Sym Elf_Sym;
 
 public:
   static bool classof(const SymbolBody *S) {
     return S->kind() == Base::SharedKind;

lld/trunk/ELF/Writer.cpp

if (OutputSection<ELFT> *OS = DynamicSec.InitArraySec) {
     Symtab.addIgnoredSym<ELFT>("__init_array_end");
   }
 
   // FIXME: Try to avoid the extra walk over all global symbols.
   std::vector<DefinedCommon<ELFT> *> CommonSymbols;
   for (auto &P : Symtab.getSymbols()) {
     StringRef Name = P.first;
     SymbolBody *Body = P.second->Body;
-    if (Body->isStrongUndefined())
+    if (auto *U = dyn_cast<Undefined<ELFT>>(Body)) {
+      if (!U->isWeak() && !U->canKeepUndefined())
         reportUndefined<ELFT>(Symtab, *Body);
+    }
 
     if (auto *C = dyn_cast<DefinedCommon<ELFT>>(Body))
       CommonSymbols.push_back(C);
     if (!includeInSymtab(*Body))
       continue;
     SymTabSec.addSymbol(Name);
 
     if (needsDynamicSections() && includeInDynamicSymtab(*Body))

lld/trunk/test/elf2/undefined-opt.s

+# RUN: llvm-mc -filetype=obj -triple=x86_64-unknown-linux %s -o %t.o
+# RUN: llvm-mc -filetype=obj -triple=x86_64-unknown-linux \
+# RUN:     %p/Inputs/abs.s -o %tabs.o
+# RUN: llvm-mc -filetype=obj -triple=x86_64-unknown-linux \
+# RUN:     %p/Inputs/shared.s -o %tshared.o
+# RUN: rm -f %tar.a
+# RUN: llvm-ar rcs %tar.a %tabs.o %tshared.o
+# REQUIRES: x86
+
+# Symbols from the archive are not in if not needed
+# RUN: lld -flavor gnu2 -o %t1 %t.o %tar.a
+# RUN: llvm-readobj --symbols %t1 | FileCheck --check-prefix=NO-UNDEFINED %s
+# NO-UNDEFINED: Symbols [
+# NO-UNDEFINED-NOT: Name: abs
+# NO-UNDEFINED-NOT: Name: big
+# NO-UNDEFINED-NOT: Name: bar
+# NO-UNDEFINED-NOT: Name: zed
+# NO-UNDEFINED: ]
+
+# Symbols from the archive are in if needed, but only from the
+# containing object file
+# RUN: lld -flavor gnu2 -o %t2 %t.o %tar.a -u bar
+# RUN: llvm-readobj --symbols %t2 | FileCheck --check-prefix=ONE-UNDEFINED %s
+# ONE-UNDEFINED: Symbols [
+# ONE-UNDEFINED-NOT: Name: abs
+# ONE-UNDEFINED-NOT: Name: big
+# ONE-UNDEFINED: Name: bar
+# ONE-UNDEFINED: Name: zed
+# ONE-UNDEFINED: ]
+
+# Use the option couple of times, both short and long forms
+# RUN: lld -flavor gnu2 -o %t3 %t.o %tar.a -u bar --undefined=abs
+# RUN: llvm-readobj --symbols %t3 | FileCheck --check-prefix=TWO-UNDEFINED %s
+# TWO-UNDEFINED: Symbols [
+# TWO-UNDEFINED: Name: abs
+# TWO-UNDEFINED: Name: big
+# TWO-UNDEFINED: Name: bar
+# TWO-UNDEFINED: Name: zed
+# TWO-UNDEFINED: ]
+
+# Added undefined symbol may be left undefined without error, but
+# shouldn't show up in the dynamic table.
+# RUN: lld -flavor gnu2 -shared -o %t4 %t.o %tar.a -u unknown
+# RUN: llvm-readobj --dyn-symbols %t4 | \
+# RUN:     FileCheck --check-prefix=UNK-UNDEFINED-SO %s
+# UNK-UNDEFINED-SO: DynamicSymbols [
+# UNK-UNDEFINED-SO-NOT:     Name: unknown
+# UNK-UNDEFINED-SO: ]
+
+.globl _start;
+_start: