[LLD] Support for --unresolved-symbols option in llvm lld for ELF file format

[• joshishr]

Summary

Adding support for --unresolved-symbols option for ELF file format in llvm linker lld.
Details of this option can be found in gnu ld(1) man page.

Implemented all four values of this option: report-all, ignore-all, ignore-in-object-files and ignore-in-shared-libs
Implemented behavior of all the values as standalone as well as in combination with options --allow-shlib-undefined, --no-allow-shlib-undefined, -shared, -static

Bug# 24524
https://llvm.org/bugs/show_bug.cgi?id=24524

Patch by Shridhar Joshi

[• joshishr]

Added Test case.

[ruiu]

lib/Driver/GnuLdDriver.cpp

439–442

It's better to define enums for these strings and use them in Resolver.cpp instead of strings.

458

I'm wondering if this is correct. Does --unresolved-symbols really negate --allow-shlib-undefines?

[atanasyan]

test/elf/unresolved-symbols.test

9

Is it possible to escape using Clang in the test? Usually build-bots do not build Clang with LLD.

[• joshishr]

It's better to define enums for these strings and use them in Resolver.cpp

instead of strings.
Yes, enum will be good

I'm wondering if this is correct. Does --unresolved-symbols really negate

--allow-shlib-undefines?
Not all values of --unresolved-symbols option negate
--allow-shlib-undefines.
But --allow-shlib-undefines option is only for shared libraries
whereas --unresolved-symbols is for relocatable object files, shared
libraries as well as executable.
I have tested behavior of all values of --unresolved-symbols in
combination with --allow-shlib-undefines and --no-allow-shlib-undefines
with gnu ld and gold.

Implementation of --unresolved-symbols option submitted in lld is in
compliance with gnu ld and gold.

@simon Atanasyan,

Is it possible to escape using Clang in the test? Usually build-bots do

not build Clang with LLD
Sure, I will update test case by avoiding clang usage.

[• joshishr]

lib/Driver/GnuLdDriver.cpp

439–442

Yes, enum will be good. Will update the patch.

458

Not all values of --unresolved-symbols option negate --allow-shlib-undefines.
But --allow-shlib-undefines option is only for shared libraries whereas --unresolved-symbols is for relocatable object files, shared libraries as well as executable.
I have tested behavior of all values of --unresolved-symbols in combination with --allow-shlib-undefines and --no-allow-shlib-undefines with gnu ld and gold.

Implementation of --unresolved-symbols option submitted in lld is in compliance with gnu ld and gold.

test/elf/unresolved-symbols.test

9

Sure, I will update test case by avoiding clang usage.

[joerg]

I'm missing test cases for undefined symbols in shared libraries linked against. That's the area where old lld was seriously misbehaving. In short: we should not complain about undefined symbols in shared libraries used as dependencies.

[• joshishr]

In D12226#238096, @joerg wrote:

I'm missing test cases for undefined symbols in shared libraries linked against. That's the area where old lld was seriously misbehaving. In short: we should not complain about undefined symbols in shared libraries used as dependencies.

Test case i have added also tests the behavior of undefined symbols in shared libraries linked against. Default behavior is not complain about undefined symbols in shared libraries used as dependencies. --unresolved-symbols option is not changing this default behavior.

I might not have completely understood your query. Please elaborate more, if this does not answer your query.

[joerg]

The undefined-shlib case makes no sense. That's exactly the case I am talking about.

[• joshishr]

Incorporated Review comments:

1. Use enums instead of strings and use it in Resolver.cpp
2. Avoid using clang in lld test case

[• joshishr]

In D12226#238177, @joerg wrote:

The undefined-shlib case makes no sense. That's exactly the case I am talking about.

--[no]-allow-shlib-undefined option is shared library specific. --unresolved-symbols=ignore-in-object-files option may look equivalent to --allow-shlib-undefined but it is not exactly equivalent since it will report undefined symbols in object files while allowing undefined symbols from dependent shared libraries linked against. So --[no]-allow-shlib-undefined and --unresolved-symbols options serves different unique purpose. Comparatively later option has lot more use case as it allows unresolved symbols in executable as well. This adds use case to defer symbol resolution till run time by pre-loading shared libraries or through dlopen().

Some of the use cases in my test case using --[no]-allow-shlib-undefined may look making no sense but I have added them just for completeness of testing all combinations.

[soumitra]

include/lld/Core/LinkingContext.h

164

Return type should be the enumeration type, and not the size specifier.

214

Parameter type should be the enumeration type and not the size specifier.

358

Member should have the enumeration type, not the size specifier.

lib/Core/LinkingContext.cpp

29

I would recommend using elf::DEFAULT for the initialization.

lib/Core/Resolver.cpp

469

This should ideally be a fall-through case along with the case for 'IGNORE_ALL'.

[• joshishr]

Updated source code with

1. Use enum type for set() and get() APIs
2. Use default enum value for initialization
3. Common switch cases mixed together

include/lld/Core/LinkingContext.h

 
 namespace lld {
 class PassManager;
 class File;
 class Writer;
 class Node;
 class SharedLibraryFile;
 
+enum UnresolvedSymbols : uint8_t {
+    DEFAULT                  = 0,
+    REPORT_ALL               = 1,
+    IGNORE_ALL               = 2,
+    IGNORE_IN_OBJECT_FILES   = 3,
+    IGNORE_IN_SHARED_LIBS    = 4,
+};
+
 /// \brief The LinkingContext class encapsulates "what and how" to link.
 ///
 /// The base class LinkingContext contains the options needed by core linking.
 /// Subclasses of LinkingContext have additional options needed by specific
 /// Writers. For example, ELFLinkingContext has methods that supplies
 /// options to the ELF Writer and ELF Passes.
 class LinkingContext {
 public:
public:
   }
 
   /// Normally, every UndefinedAtom must be replaced by a DefinedAtom or a
   /// SharedLibraryAtom for the link to be successful.  This method controls
   /// whether core linking considers remaining undefines from the shared library
   /// to be an error.
   bool allowShlibUndefines() const { return _allowShlibUndefines; }
 
+  /// Normally, every UndefinedAtom must be replaced by a DefinedAtom or a
+  /// SharedLibraryAtom for the link to be successful.  This method controls
+  /// whether core linking considers remaining undefines from the shared library
+  /// and executable to be an error.
+  UnresolvedSymbols unresolvedSymbols() { return _unresolvedSymbols; }
+
   /// If true, core linking will write the path to each input file to stdout
   /// (i.e. llvm::outs()) as it is used.  This is used to implement the -t
   /// linker option.
   ///
   /// \todo This should be a method core linking calls so that drivers can
   /// format the line as needed.
   bool logInputFiles() const { return _logInputFiles; }
 
public:
   }
   void setPrintRemainingUndefines(bool print) {
     _printRemainingUndefines = print;
   }
   void setAllowRemainingUndefines(bool allow) {
     _allowRemainingUndefines = allow;
   }
   void setAllowShlibUndefines(bool allow) { _allowShlibUndefines = allow; }
+  void setUnresolvedSymbols(UnresolvedSymbols value) { _unresolvedSymbols = value; }
   void setLogInputFiles(bool log) { _logInputFiles = log; }
 
   // Returns true if multiple definitions should not be treated as a
   // fatal error.
   bool getAllowDuplicates() const { return _allowDuplicates; }
 
   void appendLLVMOption(const char *opt) { _llvmOptions.push_back(opt); }
 
protected:
   bool _searchArchivesToOverrideTentativeDefinitions;
   bool _searchSharedLibrariesToOverrideTentativeDefinitions;
   bool _warnIfCoalesableAtomsHaveDifferentCanBeNull;
   bool _warnIfCoalesableAtomsHaveDifferentLoadName;
   bool _printRemainingUndefines;
   bool _allowRemainingUndefines;
   bool _logInputFiles;
   bool _allowShlibUndefines;
+  UnresolvedSymbols _unresolvedSymbols;
   OutputFileType _outputFileType;
   std::vector<StringRef> _deadStripRoots;
   std::map<std::string, std::string> _aliasSymbols;
   std::vector<const char *> _llvmOptions;
   StringRefVector _initialUndefinedSymbols;
   std::vector<std::unique_ptr<Node>> _nodes;
   mutable llvm::BumpPtrAllocator _allocator;
   mutable uint64_t _nextOrdinal;

lib/Core/LinkingContext.cpp

LinkingContext::LinkingContext()
     : _deadStrip(false), _allowDuplicates(false),
       _globalsAreDeadStripRoots(false),
       _searchArchivesToOverrideTentativeDefinitions(false),
       _searchSharedLibrariesToOverrideTentativeDefinitions(false),
       _warnIfCoalesableAtomsHaveDifferentCanBeNull(false),
       _warnIfCoalesableAtomsHaveDifferentLoadName(false),
       _printRemainingUndefines(true), _allowRemainingUndefines(false),
       _logInputFiles(false), _allowShlibUndefines(true),
+      _unresolvedSymbols(UnresolvedSymbols::DEFAULT), 
       _outputFileType(OutputFileType::Default), _nextOrdinal(0) {}
 
 LinkingContext::~LinkingContext() {}
 
 bool LinkingContext::validate(raw_ostream &diagnostics) {
   return validateImpl(diagnostics);
 }
 
 std::error_code LinkingContext::writeFile(const File &linkedFile) const {

lib/Core/Resolver.cpp

bool Resolver::checkUndefines() {
 
   // Warn about unresolved symbols.
   bool foundUndefines = false;
   for (const UndefinedAtom *undef : undefinedAtoms) {
     // Skip over a weak symbol.
     if (undef->canBeNull() != UndefinedAtom::canBeNullNever)
       continue;
 
+    switch (_ctx.unresolvedSymbols()) {
+    case UnresolvedSymbols::IGNORE_ALL:
+    case UnresolvedSymbols::IGNORE_IN_OBJECT_FILES:
+      // IGNORE_ALL: skip over the undefined symbols for executable
+      // IGNORE_IN_OBJECT_FILES: if the unresolved symbol is from shared 
+      // library report it but if it is from a regular object file then 
+      // skip over it
+      if (!isa<SharedLibraryFile>(undef->file()))
+        continue;
+      break;
+
+    case UnresolvedSymbols::IGNORE_IN_SHARED_LIBS:
+      // if the unresolved symbol is from shared library skip over it
+      // But if it is from a regular object file then report it
+      if (isa<SharedLibraryFile>(undef->file()) && _ctx.allowShlibUndefines())
+        continue;
+      break;
+
+    default:
+      break;
+    }
+
     // If this is a library and undefined symbols are allowed on the
     // target platform, skip over it.
     if (isa<SharedLibraryFile>(undef->file()) && _ctx.allowShlibUndefines())
       continue;
 
     // If the undefine is coalesced away, skip over it.
     if (_symbolTable.isCoalescedAway(undef))
       continue;

lib/Driver/GnuLdDriver.cpp

bool GnuLdDriver::parse(llvm::ArrayRef<const char *> args,
   if (parsedArgs.hasArg(OPT_shared)) {
     ctx->setOutputELFType(llvm::ELF::ET_DYN);
     ctx->setAllowShlibUndefines(true);
     ctx->setUseShlibUndefines(false);
     ctx->setPrintRemainingUndefines(false);
     ctx->setAllowRemainingUndefines(true);
   }
 
+  // Handle --unresolved-symbols option
+  if (auto *arg = parsedArgs.getLastArg(OPT_unresolved_symbols)) {
+    StringRef val = arg->getValue();
+    if (val.empty()) {
+      diag << "error: option --unresolved-symbols="
+              " must take one of the following arguments:\n"
+              "\tignore-all, report-all, "
+              "ignore-in-shared-libs, ignore-in-object-files\n";
+      return false;
+    }
+    else if (val.compare("ignore-all") &&
+             val.compare("report-all") &&
+             val.compare("ignore-in-object-files") &&
+             val.compare("ignore-in-shared-libs")) {
+
+      diag << "error: option --unresolved-symbols="
+           << val
+           << " must take one of the following arguments:\n"
+              "\tignore-all, report-all, "
+              "ignore-in-shared-libs, ignore-in-object-files\n";
+      return false;
+    }
+
+    if (!val.compare("report-all")) {
+       ctx->setUnresolvedSymbols(UnresolvedSymbols::REPORT_ALL);
+       ctx->setAllowShlibUndefines(false);
+    }
+    else if (!val.compare("ignore-all")) {
+       ctx->setUnresolvedSymbols(UnresolvedSymbols::IGNORE_ALL);
+    }
+    else if (!val.compare("ignore-in-object-files")) {
+       ctx->setUnresolvedSymbols(UnresolvedSymbols::IGNORE_IN_OBJECT_FILES);
+       ctx->setAllowShlibUndefines(false);
+    }
+    else if (!val.compare("ignore-in-shared-libs")) {
+       ctx->setUnresolvedSymbols(UnresolvedSymbols::IGNORE_IN_SHARED_LIBS);
+    }
+
+    ctx->setPrintRemainingUndefines(true);
+    ctx->setAllowRemainingUndefines(false);
+  }
+
   // Handle --stats.
   if (parsedArgs.hasArg(OPT_stats)) {
     ctx->setCollectStats(true);
   }
 
   // Figure out if the output type is nmagic/omagic
   if (auto *arg =
           parsedArgs.getLastArg(OPT_nmagic, OPT_omagic, OPT_no_omagic)) {

lib/Driver/GnuLdOptions.td

HelpText<"Allow undefined symbols from dynamic"
              " library when creating executables">,
      Group<grp_resolveropt>;
 def use_shlib_undefs: Flag<["--"], "use-shlib-undefines">,
     HelpText<"Resolve undefined symbols from dynamic libraries">,
     Group<grp_resolveropt>;
 def allow_multiple_definition: Flag<["--"], "allow-multiple-definition">,
     HelpText<"Allow multiple definitions">,
     Group<grp_resolveropt>;
+def unresolved_symbols: Joined<["--"], "unresolved-symbols=">,
+     HelpText<"control reporting unresolved symbols">,
+     Group<grp_resolveropt>;
+def unresolved_symbols_alias : Joined<["--"], "unresolved-symbols">,
+     Alias<unresolved_symbols>;
 defm defsym : mDashEq<"defsym",
            "Create a global symbol in the output file "
            "containing the absolute address given by expression">,
            MetaVarName<"symbol=<expression>">,
            Group<grp_resolveropt>;
 
 //===----------------------------------------------------------------------===//
 /// Custom Options

test/elf/unresolved-symbols.test

+#
+# This test creates a executable and tests the options that are used to
+# to create an executable and a shared library
+#
+# This test will fail because there are unresolved symbols from the shared
+# library and we are passing --no-allow-shlib-undefined
+# Test if option is used in correct form
+RUN: lld -flavor gnu -shared %p/Inputs/foo_unresolved-symbols.o -o %p/Inputs/libfoo_unresolved-symbols.so
+
+RUN: not lld -flavor gnu %p/Inputs/main_unresolved-symbols.o -o %t --unresolved-symbols 2> %t4 
+RUN: FileCheck -check-prefix=ERROR %s < %t4
+RUN: FileCheck -check-prefix=OPTIONS %s < %t4
+
+# Test if option is used in correct form
+RUN: not lld -flavor gnu %p/Inputs/main_unresolved-symbols.o -o %t --unresolved-symbols= 2> %t4 
+RUN: FileCheck -check-prefix=ERROR %s < %t4
+RUN: FileCheck -check-prefix=OPTIONS %s < %t4
+
+# Test if option value used is correct
+RUN: not lld -flavor gnu %p/Inputs/main_unresolved-symbols.o -o %t --unresolved-symbols=abcd 2> %t4 
+RUN: FileCheck -check-prefix=ERROR1 %s < %t4
+RUN: FileCheck -check-prefix=OPTIONS %s < %t4
+
+# Test behaviour of --unresolved-symbols=report-all
+RUN: not lld -flavor gnu %p/Inputs/main_unresolved-symbols.o -e main \
+RUN: %p/Inputs/libfoo_unresolved-symbols.so -o %t --unresolved-symbols=report-all 2> %t1
+RUN: FileCheck -check-prefix=UNDEFINED-SHLIB %s < %t1
+RUN: FileCheck -check-prefix=UNDEFINED-EXEC %s < %t1
+
+# Test behaviour of --unresolved-symbols=ignore-all
+RUN: lld -flavor gnu %p/Inputs/main_unresolved-symbols.o -e main \
+RUN: %p/Inputs/libfoo_unresolved-symbols.so -o %t --unresolved-symbols=ignore-all 2> %t2
+
+# Test behaviour of --unresolved-symbols=ignore-in-object-files
+RUN: not lld -flavor gnu %p/Inputs/main_unresolved-symbols.o -e main \
+RUN: %p/Inputs/libfoo_unresolved-symbols.so -o %t --unresolved-symbols=ignore-in-object-files 2> %t2
+RUN: FileCheck -check-prefix=UNDEFINED-SHLIB %s < %t2
+
+# Test behaviour of --unresolved-symbols=ignore-in-shared-libs
+RUN: not lld -flavor gnu %p/Inputs/main_unresolved-symbols.o -e main \
+RUN: %p/Inputs/libfoo_unresolved-symbols.so -o %t --unresolved-symbols=ignore-in-shared-libs 2> %t3
+RUN: FileCheck -check-prefix=UNDEFINED-EXEC %s < %t3
+
+# Test behaviour of --unresolved-symbols=report-all
+# along with --allow-shlib-undefined
+RUN: not lld -flavor gnu %p/Inputs/main_unresolved-symbols.o -e main \
+RUN: %p/Inputs/libfoo_unresolved-symbols.so -o %t --unresolved-symbols=report-all \
+RUN:  --allow-shlib-undefined 2> %t1
+RUN: FileCheck -check-prefix=UNDEFINED-EXEC %s < %t1
+
+# Test behaviour of --unresolved-symbols=ignore-all
+# along with --allow-shlib-undefined
+RUN: lld -flavor gnu %p/Inputs/main_unresolved-symbols.o -e main \
+RUN: %p/Inputs/libfoo_unresolved-symbols.so -o %t --unresolved-symbols=ignore-all \
+RUN: --allow-shlib-undefined 2> %t2
+
+# Test behaviour of --unresolved-symbols=ignore-in-object-files
+# along with --allow-shlib-undefined
+RUN: lld -flavor gnu %p/Inputs/main_unresolved-symbols.o -e main \
+RUN: %p/Inputs/libfoo_unresolved-symbols.so -o %t --unresolved-symbols=ignore-in-object-files \
+RUN: --allow-shlib-undefined 2> %t2
+
+# Test behaviour of --unresolved-symbols=ignore-in-shared-libs
+# along with --allow-shlib-undefined
+RUN: not lld -flavor gnu %p/Inputs/main_unresolved-symbols.o -e main \
+RUN: %p/Inputs/libfoo_unresolved-symbols.so -o %t --unresolved-symbols=ignore-in-shared-libs \
+RUN: --allow-shlib-undefined 2> %t3
+RUN: FileCheck -check-prefix=UNDEFINED-EXEC %s < %t3
+
+# Test behaviour of --unresolved-symbols=report-all
+# along with --no-allow-shlib-undefined
+RUN: not lld -flavor gnu %p/Inputs/main_unresolved-symbols.o -e main \
+RUN: %p/Inputs/libfoo_unresolved-symbols.so -o %t --unresolved-symbols=report-all \
+RUN: --no-allow-shlib-undefined 2> %t1
+RUN: FileCheck -check-prefix=UNDEFINED-SHLIB %s < %t1
+RUN: FileCheck -check-prefix=UNDEFINED-EXEC %s < %t1
+
+# Test behaviour of --unresolved-symbols=ignore-all
+# along with --no-allow-shlib-undefined
+RUN: not lld -flavor gnu %p/Inputs/main_unresolved-symbols.o -e main \
+RUN: %p/Inputs/libfoo_unresolved-symbols.so -o %t --unresolved-symbols=ignore-all \
+RUN: --no-allow-shlib-undefined 2> %t1
+RUN: FileCheck -check-prefix=UNDEFINED-SHLIB %s < %t1
+
+# Test behaviour of --unresolved-symbols=ignore-in-object-files
+# along with --no-allow-shlib-undefined
+RUN: not lld -flavor gnu %p/Inputs/main_unresolved-symbols.o -e main %p/Inputs/libfoo_unresolved-symbols.so -o %t \
+RUN: --unresolved-symbols=ignore-in-object-files --no-allow-shlib-undefined 2> %t2
+RUN: FileCheck -check-prefix=UNDEFINED-SHLIB %s < %t2
+
+# Test behaviour of --unresolved-symbols=ignore-in-shared-libs
+# along with --no-allow-shlib-undefined
+RUN: not lld -flavor gnu %p/Inputs/main_unresolved-symbols.o -e main \
+RUN: %p/Inputs/libfoo_unresolved-symbols.so -o %t --unresolved-symbols=ignore-in-shared-libs \
+RUN: --no-allow-shlib-undefined 2> %t3
+RUN: FileCheck -check-prefix=UNDEFINED-EXEC %s < %t3
+RUN: FileCheck -check-prefix=UNDEFINED-SHLIB %s < %t3
+
+# Test behaviour of --unresolved-symbols=report-all
+# along with -shared
+RUN: not lld -flavor gnu -shared -fpic %p/Inputs/foo_unresolved-symbols.o -o %t \
+RUN: --unresolved-symbols=report-all 2> %t3
+RUN: FileCheck -check-prefix=UNDEFINED-OBJ %s < %t3
+
+# Test behaviour of --unresolved-symbols=ignore-all
+# along with -shared
+RUN: lld -flavor gnu -shared -fpic %p/Inputs/foo_unresolved-symbols.o -o %t \
+RUN: --unresolved-symbols=ignore-all 2> %t3
+
+# Test behaviour of --unresolved-symbols=ignore-in-object-files
+# along with -shared
+RUN: lld -flavor gnu -shared -fpic %p/Inputs/foo_unresolved-symbols.o -o %t \
+RUN: --unresolved-symbols=ignore-in-object-files 2> %t3
+
+# Test behaviour of --unresolved-symbols=ignore-in-shared-libs
+# along with -shared
+RUN: not lld -flavor gnu -shared -fpic %p/Inputs/foo_unresolved-symbols.o -o %t \
+RUN: --unresolved-symbols=ignore-in-shared-libs 2> %t3
+RUN: FileCheck -check-prefix=UNDEFINED-OBJ %s < %t3
+
+ERROR: error: option --unresolved-symbols= must take one of the following arguments:
+ERROR1: error: option --unresolved-symbols=abcd must take one of the following arguments:
+OPTIONS:              ignore-all, report-all, ignore-in-shared-libs, ignore-in-object-files
+UNDEFINED-SHLIB: Undefined symbol: {{.+[\\/]}}libfoo_unresolved-symbols.so: bar
+UNDEFINED-OBJ: Undefined symbol: {{.+[\\/]}}foo_unresolved-symbols.o: bar
+UNDEFINED-EXEC: Undefined symbol: {{.+[\\/]}}main_unresolved-symbols.o: goo
+