[lld] [ELF] Implement --export-dynamic/-E

[rafaelauler]

Summary

When creating a dynamic executable and receiving the -E flag, the linker should
export all globally visible symbols in its dynamic symbol table.

[shankarke]

include/lld/ReaderWriter/ELFLinkingContext.h

162

should be applicable even for buiding shared libraries (ET_DYN) as well.

lib/Driver/GnuLdOptions.td

83–88

Can you move this under dynamic library and executable options ?

lib/ReaderWriter/ELF/OutputELFWriter.h

187

should we change the dynamicExport for all atoms, when dynamic export is set, to be dynamicExportAlways ? What do you think ?

[rafaelauler]

Hi Shankar,

Thanks for putting in time to review this patch. Comments below.

include/lld/ReaderWriter/ELFLinkingContext.h

162

I'm somewhat confused because I based this patch on what GNU ld manual says:

"When creating a dynamically linked *executable*, add all symbols to the dynamic symbol table. The dynamic symbol table is the set of symbols which are visible from dynamic objects at run time. If you do not use this option, the dynamic symbol table will normally contain only those symbols which are referenced by some dynamic object mentioned in the link. If you use dlopen to load a dynamic object which needs to refer back to the symbols defined by the program, rather than some other dynamic object, then you will probably need to use this option when linking the program itself."

Can you confirm that this should work for shared libraries as well? If so, what is the effect, since shared libraries already export all symbols by default?

lib/ReaderWriter/ELF/OutputELFWriter.h

187

Indeed I think It would be a cleaner solution, I'll look into this.

[shankarke]

include/lld/ReaderWriter/ELFLinkingContext.h

162

--dynamic-list would alter this behavior, which is applicable for building shared libraries and dynamic executables.

[rafaelauler]

I updated this patch according to suggestions. While this patch doesn't implement the effect of --export-dynamic when building shared libraries (currently, DynamicLibraryWriter already exports all symbols), it doesn't restrict the flag only to executables. I also moved the logic that exports symbols in the dynamic symbol table from OutputELFWriter to the ExecutableWriter class. I think it is not correct to leave this at OutputELFWriter because DynamicLibraryWriter, another subclass of OutputELFWriter, already exports all symbols, meaning we can potentially end up with duplicated symbols in the dynamic symbol table when creating shared libs.

Another aspect worth commenting is implementing --export-dynamic by making all ELF symbols to have the dynamicExportAlways propriety. I made a patch for this, but it required changing too many interfaces just to allow the "--export-dynamic" information to reach the ELFFile class, since it is loosely coupled via the Registry. The advantages would be to have exposed during the entire linking process that this symbol will be exported. After checking whether the dynamicExportAlways propriety is actually used during linking, I found out that it is not important, except when actually building the symbol table.

Therefore, I concluded that it was not worth the change and that we end up with simpler code if we just stick with checks for --export-dynamic when building the symbol table. If you think that we should change dynamicExportAlways, though, I can submit the other patch.

[rafaelauler]

Committed r219334.

include/lld/ReaderWriter/ELFLinkingContext.h

lld::elf::TargetHandler<ELFT> &getTargetHandler() const {
     return static_cast<lld::elf::TargetHandler<ELFT> &>(*_targetHandler.get());
   }
 
   TargetHandlerBase *targetHandler() const { return _targetHandler.get(); }
   void addPasses(PassManager &pm) override;
 
   void setTriple(llvm::Triple trip) { _triple = trip; }
   void setNoInhibitExec(bool v) { _noInhibitExec = v; }
+  void setExportDynamic(bool v) { _exportDynamic = v; }
   void setIsStaticExecutable(bool v) { _isStaticExecutable = v; }
   void setMergeCommonStrings(bool v) { _mergeCommonStrings = v; }
   void setUseShlibUndefines(bool use) { _useShlibUndefines = use; }
   void setOutputELFType(uint32_t type) { _outputELFType = type; }
 
+  bool shouldExportDynamic() const { return _exportDynamic; }
+
   void createInternalFiles(std::vector<std::unique_ptr<File>> &) const override;
 
   /// \brief Set the dynamic linker path
   void setInterpreter(StringRef dynamicLinker) {
     _dynamicLinkerArg = true;
     _dynamicLinkerPath = dynamicLinker;
   }
 
protected:
   std::unique_ptr<File> createUndefinedSymbolFile() const override;
 
   uint16_t _outputELFType; // e.g ET_EXEC
   llvm::Triple _triple;
   std::unique_ptr<TargetHandlerBase> _targetHandler;
   uint64_t _baseAddress;
   bool _isStaticExecutable;
   bool _noInhibitExec;
+  bool _exportDynamic;
   bool _mergeCommonStrings;
   bool _runLayoutPass;
   bool _useShlibUndefines;
   bool _dynamicLinkerArg;
   bool _noAllowDynamicLibraries;
   bool _mergeRODataToTextSegment;
   OutputMagic _outputMagic;
   StringRefVector _inputSearchPaths;

lib/Driver/GnuLdDriver.cpp

case OPT_output:
       _outputOptionSet = true;
       ctx->setOutputPath(inputArg->getValue());
       break;
 
     case OPT_noinhibit_exec:
       ctx->setAllowRemainingUndefines(true);
       break;
 
+    case OPT_export_dynamic:
+      ctx->setExportDynamic(true);
+      break;
+
     case OPT_merge_strings:
       ctx->setMergeCommonStrings(true);
       break;
 
     case OPT_t:
       ctx->setLogInputFiles(true);
       break;
 

lib/Driver/GnuLdOptions.td

def L : Joined<["-"], "L">, MetaVarName<"<dir>">,
      Group<grp_main>;
 def l : Joined<["-"], "l">, MetaVarName<"<libName>">,
      HelpText<"Root name of library to use">,
      Group<grp_main>;
 def noinhibit_exec : Flag<["--"], "noinhibit-exec">,
      HelpText<"Retain the executable output file whenever"
               " it is still usable">,
      Group<grp_main>;
 defm e : smDash<"e", "entry",
       "Name of entry point symbol">,
       Group<grp_main>;
 defm init: dashEq<"init", "init",
       "Specify an initializer function">,
       Group<grp_main>;
 defm fini: dashEq<"fini", "fini",
       "Specify a finalizer function">,
       Group<grp_main>;
 def whole_archive: Flag<["--"], "whole-archive">,
      HelpText<"Force load of all members in a static library">,
      Group<grp_main>;
 def no_whole_archive: Flag<["--"], "no-whole-archive">,
      HelpText<"Restores the default behavior of loading archive members">,

 def dynamic_linker_alias : Separate<["-"], "dynamic-linker">,
     Alias<dynamic_linker>;
 defm rpath : dashEq<"rpath", "rpath",
      "Add a directory to the runtime library search path">,
      Group<grp_dynlibexec>;
 def rpath_link : Separate<["-"], "rpath-link">,
      HelpText<"Specifies the first set of directories to search">,
      Group<grp_dynlibexec>;
+def export_dynamic : Flag<["--"], "export-dynamic">,
+     HelpText<"Add all symbols to the dynamic symbol table"
+              " when creating executables">,
+     Group<grp_main>;
+def alias_export_dynamic: Flag<["-"], "E">,
+      Alias<export_dynamic>;
 
 //===----------------------------------------------------------------------===//
 /// Dynamic Library Options
 //===----------------------------------------------------------------------===//
 def grp_dynlib : OptionGroup<"opts">,
      HelpText<"DYNAMIC LIBRARY OPTIONS">;
 def soname : Joined<["-", "--"], "soname=">,
      HelpText<"Set the internal DT_SONAME field to the specified name">,

lib/ReaderWriter/ELF/ExecutableWriter.h

 template<class ELFT>
 class ExecutableWriter : public OutputELFWriter<ELFT> {
 public:
   ExecutableWriter(const ELFLinkingContext &context, TargetLayout<ELFT> &layout)
       : OutputELFWriter<ELFT>(context, layout),
         _runtimeFile(new CRuntimeFile<ELFT>(context)) {}
 
 protected:
+  virtual void buildDynamicSymbolTable(const File &file);
   virtual void addDefaultAtoms();
   virtual bool createImplicitFiles(std::vector<std::unique_ptr<File> > &);
   virtual void finalizeDefaultAtomValues();
   virtual void createDefaultSections();
   LLD_UNIQUE_BUMP_PTR(InterpSection<ELFT>) _interpSection;
   std::unique_ptr<CRuntimeFile<ELFT> > _runtimeFile;
 };
 
 //===----------------------------------------------------------------------===//
 //  ExecutableWriter
 //===----------------------------------------------------------------------===//
+template<class ELFT>
+void ExecutableWriter<ELFT>::buildDynamicSymbolTable(const File &file) {
+  for (auto sec : this->_layout.sections())
+    if (auto section = dyn_cast<AtomSection<ELFT>>(sec))
+      for (const auto &atom : section->atoms()) {
+        const DefinedAtom *da = dyn_cast<const DefinedAtom>(atom->_atom);
+        if (!da)
+          continue;
+        if (da->dynamicExport() != DefinedAtom::dynamicExportAlways &&
+            !this->_context.isDynamicallyExportedSymbol(da->name()) &&
+            !(this->_context.shouldExportDynamic() &&
+              da->scope() == Atom::Scope::scopeGlobal))
+          continue;
+        this->_dynamicSymbolTable->addSymbol(atom->_atom, section->ordinal(),
+                                             atom->_virtualAddr, atom);
+      }
+
+  OutputELFWriter<ELFT>::buildDynamicSymbolTable(file);
+}
 
 /// \brief Add absolute symbols by default. These are linker added
 /// absolute symbols
 template<class ELFT>
 void ExecutableWriter<ELFT>::addDefaultAtoms() {
   _runtimeFile->addUndefinedAtom(this->_context.entrySymbolName());
   _runtimeFile->addAbsoluteAtom("__bss_start");
   _runtimeFile->addAbsoluteAtom("__bss_end");

lib/ReaderWriter/ELF/OutputELFWriter.h

for (auto &atom : _layout.absoluteAtoms())
     _symtab->addSymbol(atom->_atom, ELF::SHN_ABS, atom->_virtualAddr);
   for (const UndefinedAtom *a : file.undefined())
     _symtab->addSymbol(a, ELF::SHN_UNDEF);
 }
 
 template <class ELFT>
 void OutputELFWriter<ELFT>::buildDynamicSymbolTable(const File &file) {
   ScopedTask task(getDefaultDomain(), "buildDynamicSymbolTable");
-  for (auto sec : this->_layout.sections())
-    if (auto section = dyn_cast<AtomSection<ELFT>>(sec))
-      for (const auto &atom : section->atoms()) {
-        const DefinedAtom *da = dyn_cast<const DefinedAtom>(atom->_atom);
-        if (da && (da->dynamicExport() == DefinedAtom::dynamicExportAlways ||
-                   _context.isDynamicallyExportedSymbol(da->name())))
-          _dynamicSymbolTable->addSymbol(atom->_atom, section->ordinal(),
-                                         atom->_virtualAddr, atom);
-      }
   for (const auto &sla : file.sharedLibrary()) {
     if (isDynSymEntryRequired(sla))
       _dynamicSymbolTable->addSymbol(sla, ELF::SHN_UNDEF);
     if (isNeededTagRequired(sla))
       _soNeeded.insert(sla->loadName());
   }
   // Never mark the dynamic linker as DT_NEEDED
   _soNeeded.erase(sys::path::filename(_context.getInterpreter()));
   for (const auto &loadName : _soNeeded) {
     Elf_Dyn dyn;
     dyn.d_tag = DT_NEEDED;
     dyn.d_un.d_val = _dynamicStringTable->addString(loadName.getKey());
     _dynamicTable->addEntry(dyn);
   }

test/elf/export-dynamic.test

+# Tests the --export-dynamic (-E) flag. When creating a dynamic executable and
+# receiving this flag, the linker should export all globally visible symbols in
+# its dynamic symbol table.
+
+#RUN: yaml2obj -format=elf %s -o=%t.o
+#RUN: lld -flavor gnu  -target x86_64 -E %t.o -e=main -o %t1
+#RUN: llvm-readobj -dt %t1 | FileCheck -check-prefix CHECKSYMS %s
+
+#CHECKSYMS: myfunc1@
+#CHECKSYMS: main@
+#CHECKSYMS: myvar1@
+
+# The object file below was generated with the following code:
+#
+# (command line clang -c prog.c -o prog.o)
+#
+# int myvar1 = 22;
+#
+# static int mysecretvar = 11;
+#
+# int myfunc1() {
+#   return 23;
+# }
+#
+# static int mysecretfunc() {
+#   return 42;
+# }
+#
+# int main() {
+#   return mysecretfunc() + mysecretvar;
+# }
+---
+FileHeader:      
+  Class:           ELFCLASS64
+  Data:            ELFDATA2LSB
+  OSABI:           ELFOSABI_GNU
+  Type:            ET_REL
+  Machine:         EM_X86_64
+Sections:        
+  - Name:            .text
+    Type:            SHT_PROGBITS
+    Flags:           [ SHF_ALLOC, SHF_EXECINSTR ]
+    AddressAlign:    0x0000000000000010
+    Content:         554889E5B8170000005DC30F1F440000554889E54883EC10C745FC00000000E81C000000030425000000004883C4105DC36666666666662E0F1F840000000000554889E5B82A0000005DC3
+  - Name:            .rela.text
+    Type:            SHT_RELA
+    Link:            .symtab
+    AddressAlign:    0x0000000000000008
+    Info:            .text
+    Relocations:     
+      - Offset:          0x0000000000000027
+        Symbol:          .data
+        Type:            R_X86_64_32S
+        Addend:          4
+  - Name:            .data
+    Type:            SHT_PROGBITS
+    Flags:           [ SHF_WRITE, SHF_ALLOC ]
+    AddressAlign:    0x0000000000000004
+    Content:         160000000B000000
+  - Name:            .bss
+    Type:            SHT_NOBITS
+    Flags:           [ SHF_WRITE, SHF_ALLOC ]
+    AddressAlign:    0x0000000000000004
+    Content:         ''
+Symbols:         
+  Local:           
+    - Name:            mysecretfunc
+      Type:            STT_FUNC
+      Section:         .text
+      Value:           0x0000000000000040
+      Size:            0x000000000000000B
+    - Name:            mysecretvar
+      Type:            STT_OBJECT
+      Section:         .data
+      Value:           0x0000000000000004
+      Size:            0x0000000000000004
+    - Name:            .text
+      Type:            STT_SECTION
+      Section:         .text
+    - Name:            .data
+      Type:            STT_SECTION
+      Section:         .data
+    - Name:            .bss
+      Type:            STT_SECTION
+      Section:         .bss
+  Global:          
+    - Name:            main
+      Type:            STT_FUNC
+      Section:         .text
+      Value:           0x0000000000000010
+      Size:            0x0000000000000021
+    - Name:            myfunc1
+      Type:            STT_FUNC
+      Section:         .text
+      Size:            0x000000000000000B
+    - Name:            myvar1
+      Type:            STT_OBJECT
+      Section:         .data
+      Size:            0x0000000000000004