[InputGraph][Gnu] Add Linkerscript support.

[MaskRay]

Summary

This adds new functionality in the InputGraph to parse LinkerScripts.

a) Adds support to expand nodes in the InputGraph.
b) Support for Hidden nodes(Nodes that are internal to the InputGraph, and not part of resolution and parsing).
c) Adds a GnuLdScript/ELFGNULdScript node.

With this dynamic glibc helloworld runs fine(dynamic 'C').

As mentioned earlier by Bigcheese, it also links with libstdc++ with appropriate relocations (dynamic) but fails to run.

<-----------------snip---------------->
$type g++
g++ is hashed (/usr/bin/g++)
$cat a.cpp
#include <iostream>
#include <stdio.h>

class A{
public:

A() {
  std::cout << "A" << std::endl;
}

~A() {
  std::cout << "~A" << std::endl;
}

};

int main()
{

A a;
return 0;

}

$g++ -B`pwd` a.cpp
lld -flavor gnu --build-id --eh-frame-hdr -m elf_x86_64 --hash-style=both -dynamic-linker /lib64/ld-linux-x86-64.so.2 -z relro /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/crt1.o /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.8/crtbegin.o -L/usr2/seaswara/tests/lld/extratests/dyncpp -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib -L/lib/../lib -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../.. /tmp/ccPK1vpI.o -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /usr/lib/gcc/x86_64-linux-gnu/4.8/crtend.o /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/crtn.o

$readelf -r a.out

Relocation section '.rela.dyn' at offset 0x796 contains 1 entries:

Offset          Info           Type           Sym. Value    Sym. Name + Addend

0000004b0298 000100000005 R_X86_64_COPY 00000000004b0298 _ZSt4cout + 0

Relocation section '.rela.plt' at offset 0x7ae contains 18 entries:

Offset          Info           Type           Sym. Value    Sym. Name + Addend

0000004ac248 000d00000007 R_X86_64_JUMP_SLO 0000000000000000 _Unwind_Resume + 0
0000004ac250 000300000007 R_X86_64_JUMP_SLO 0000000000000000 _ZNSt8ios_base4InitC1E + 0
0000004ac258 000500000007 R_X86_64_JUMP_SLO 0000000000000000 _ZNSt8ios_base4InitD1E + 0
0000004ac260 000200000007 R_X86_64_JUMP_SLO 0000000000000000 _ZStlsISt11char_traits + 0
0000004ac268 000600000007 R_X86_64_JUMP_SLO 0000000000000000 _ZSt4endlIcSt11char_tr + 0
0000004ac270 000700000007 R_X86_64_JUMP_SLO 0000000000000000 _ZNSolsEPFRSoS_E + 0
0000004ac278 000800000007 R_X86_64_JUMP_SLO 0000000000000000 gxx_personality_v0 + 0
0000004ac280 000000000025 R_X86_64_IRELATIV 0000000000419300
0000004ac288 001700000007 R_X86_64_JUMP_SLO 0000000000000000 gcc_personality_v0 + 0
0000004ac290 000000000025 R_X86_64_IRELATIV 0000000000413860
0000004ac298 000000000025 R_X86_64_IRELATIV 000000000042ef10
0000004ac2a0 000000000025 R_X86_64_IRELATIV 0000000000421140
0000004ac2a8 000000000025 R_X86_64_IRELATIV 000000000042b520
0000004ac2b0 000000000025 R_X86_64_IRELATIV 0000000000432ba0
0000004ac2b8 000000000025 R_X86_64_IRELATIV 00000000004668e0
0000004ac2c0 000000000025 R_X86_64_IRELATIV 0000000000464e50
0000004ac2c8 001900000007 R_X86_64_JUMP_SLO 0000000000000000 _Unwind_Backtrace + 0
0000004ac2d0 001800000007 R_X86_64_JUMP_SLO 0000000000000000 _Unwind_GetIP + 0

$./a.out
unexpected reloc type in static binaryAborted

<------------------snip----------------->

[Bigcheese]

This seems like a really complicated way to do this. Why can't the linker script node just modify the input graph to add the extra nodes?

Also this has the unrelated undef from shared lib changes.

[shankarke]

At the time when the linkerscript node is created or parsed, it has not yet been added to the InputGraph. For this to happen, the linkerscript node needs take InputGraph as one of the arguments, which I think would not be cleaner. Any better ways ?

The shared library changes were needed to have all of the linkerscript changes to work, otherwise it would go in a never ending loop with a helloworld test.

I will split this patch before I commit.

[Bigcheese]

lgtm

[ruiu]

Is helloworld example work related to this linker script patch? It looks this patch does not add a new feature, so it looks to me taht it can link helloworld example only because of undef from shared library change. Is this correct?

[shankarke]

I completely agree that this patch is combining other functionalities. But all these patches as a whole need to be used inorder to run the tests. This was the only reason of merging them all-together in one patch.

I will split this patch into 4 individual commits before I commit.

a) InputGraph functionality to expand elements
b) InputGraph functionality to set the hidden attribute
c) Add an API to linkingcontext on how undefined atoms from shared libraries need to be processed
d) Add LinkerScript nodes.

I dont know how to add this in the testing environment as GnuLD driver would try to look at the files before it adds to the InputGraph.

Any inputs on this would be really helpful.

[ruiu]

It is completely fine to send multiple patches, saying that the second and further patches depends on the first one. That would actually help other developers review patch set. Each step needs to compile and pass all tests if applied in sequence, though.

[shankarke]

I would make all the tests to work when you pick the individual commits. I am working on adding unit tests to the patches so that any new functionality doesnot break things.

include/lld/Core/Error.h

enum class InputGraphError {
   no_more_elements,
   no_more_files
 };
 
 inline llvm::error_code make_error_code(InputGraphError e) {
   return llvm::error_code(static_cast<int>(e), InputGraphErrorCategory());
 }
 
+/// \brief Errors returned by Reader.
+const llvm::error_category &ReaderErrorCategory();
+
+enum class ReaderError {
+  success = 0,
+  unknown_file_format = 1
+};
+
+inline llvm::error_code make_error_code(ReaderError e) {
+  return llvm::error_code(static_cast<int>(e), ReaderErrorCategory());
+}
 } // end namespace lld
 
 namespace llvm {
 
 template <> struct is_error_code_enum<lld::NativeReaderError> : true_type {};
 template <> struct is_error_code_enum<lld::YamlReaderError> : true_type {};
 template <>
 struct is_error_code_enum<lld::LinkerScriptReaderError> : true_type {};
 template <> struct is_error_code_enum<lld::InputGraphError> : true_type {};
+template <> struct is_error_code_enum<lld::ReaderError> : true_type {};
 } // end namespace llvm
 
 #endif

include/lld/Core/File.h

 public:
   virtual ~File();
 
   /// \brief Kinds of files that are supported.
   enum Kind {
     kindObject,            ///< object file (.o)
     kindSharedLibrary,     ///< shared library (.so)
     kindArchiveLibrary,    ///< archive (.a)
-    kindLinkerScript,      ///< linker script
   };
 
   /// \brief Returns file kind.  Need for dyn_cast<> on File objects.
   Kind kind() const {
     return _kind;
   }
 
   /// \brief For error messages and debugging, this returns the path to the file

include/lld/Core/InputGraph.h

public:
   virtual bool addInputElement(std::unique_ptr<InputElement>);
 
   /// \brief Set Ordinals for all the InputElements that form the InputGraph
   virtual bool assignOrdinals();
 
   /// Destructor
   virtual ~InputGraph() {}
 
+  /// Normalize the InputGraph.
+  virtual void normalize();
+
   /// \brief Do postprocessing of the InputGraph if there is a need for the
   /// to provide additional information to the user, also rearranges
   /// InputElements by their ordinals. If an user wants to place an input file
   /// at the desired position, the user can do that
   virtual void doPostProcess();
 
   range<InputElementIterT> inputElements() {
     return make_range(_inputArgs.begin(), _inputArgs.end());

 /// \brief This describes each element in the InputGraph. The Kind
 /// determines what the current node contains.
 class InputElement {
 public:
   /// Each input element in the graph can be a File or a control
   enum class Kind : uint8_t {
     Control,    // Represents a type associated with ControlNodes
     SimpleFile, // Represents a type reserved for internal files
-    File        // Represents a type associated with File Nodes
+    File,       // Represents a type associated with File Nodes
+  };
+
+  /// How does the inputGraph expand the InputElement
+  enum class ExpandType : uint8_t {
+    None,             // Do nothing(Default)
+    ReplaceAndExpand, // Replace current node and expand
+    ExpandOnly        // Expand the current node
   };
 
   /// \brief Initialize the Input Element, The ordinal value of an input Element
   /// is initially set to -1, if the user wants to override its ordinal,
   /// let the user do it
   InputElement(Kind type, int64_t ordinal = -1);
 
   virtual ~InputElement() {}
public:
   virtual void setResolveState(uint32_t state) = 0;
 
   /// \brief Get the resolve state for the element
   virtual uint32_t getResolveState() const = 0;
 
   /// \brief Reset the next index
   virtual void resetNextIndex() = 0;
 
+  /// \brief Is this a hidden node, hidden nodes are not part of
+  /// of the resolver.
+  virtual bool isHidden() const { return false; }
+
+  /// \brief How do we want to expand the current node ?
+  virtual ExpandType expandType() const { return ExpandType::None; }
+
+  /// \brief Get the elements that we want to expand with.
+  virtual range<InputGraph::InputElementIterT> expandElements() {
+    llvm_unreachable("no element to expand");
+  }
+
 protected:
   Kind _kind;              // The type of the Element
   int64_t _ordinal;        // The ordinal value
   int64_t _weight;         // Weight of the file
 };
 
 /// \brief The Control class represents a control node in the InputGraph
 class ControlNode : public InputElement {

include/lld/Core/LinkingContext.h

 
 namespace llvm {
 class Triple;
 }
 
 namespace lld {
 class PassManager;
 class File;
+class SharedLibraryFile;
 class Writer;
 class InputGraph;
 class InputElement;
 
 /// \brief The LinkingContext class encapsulates "what and how" to link.
 ///
 /// The base class LinkingContext contains the options needed by core linking.
 /// Subclasses of LinkingContext have additional options needed by specific
public:
   }
 
   /// Normally, every UndefinedAtom must be replaced by a DefinedAtom or a
   /// SharedLibraryAtom for the link to be successful.  This method controls
   /// whether core linking considers remaining undefines from the shared library
   /// to be an error.
   bool allowShlibUndefines() const { return _allowShlibUndefines; }
 
+  /// Add undefined symbols from shared libraries ?
+  virtual bool addUndefinedAtomsFromSharedLibrary(const SharedLibraryFile *) {
+    return true;
+  }
+
   /// If true, core linking will write the path to each input file to stdout
   /// (i.e. llvm::outs()) as it is used.  This is used to implement the -t
   /// linker option.
   ///
   /// \todo This should be a method core linking calls so that drivers can
   /// format the line as needed.
   bool logInputFiles() const { return _logInputFiles; }
 

include/lld/Driver/GnuLdInputGraph.h

 
 #ifndef LLD_DRIVER_GNU_LD_INPUT_GRAPH_H
 #define LLD_DRIVER_GNU_LD_INPUT_GRAPH_H
 
 #include "lld/Core/InputGraph.h"
 #include "lld/Core/Resolver.h"
 #include "lld/ReaderWriter/ELFLinkingContext.h"
 #include "lld/ReaderWriter/FileArchive.h"
+#include "lld/ReaderWriter/LinkerScript.h"
 
 namespace lld {
 
 /// \brief Represents a ELF File
 class ELFFileNode : public FileNode {
 public:
   ELFFileNode(ELFLinkingContext &ctx, StringRef path,
               std::vector<StringRef> searchPath, int64_t ordinal = -1,
virtual bool dump(raw_ostream &diagnostics) {
     diagnostics << "  contextPath : " << ((_libraryPaths.size()) ? "" : "None")
                 << "\n";
     for (auto path : _libraryPaths)
       diagnostics << "    - " << path << "\n";
     return true;
   }
 
   /// \brief Parse the input file to lld::File.
-  error_code parse(const LinkingContext &ctx, raw_ostream &diagnostics) {
-    ErrorOr<StringRef> filePath = getPath(ctx);
-    if (!filePath)
-      return error_code(filePath);
-
-    if (error_code ec = getBuffer(*filePath))
-      return ec;
-
-    if (ctx.logInputFiles())
-      diagnostics << *filePath << "\n";
-
-    if (filePath->endswith(".objtxt"))
-      return ctx.getYAMLReader().parseFile(_buffer, _files);
-
-    // Identify File type
-    llvm::sys::fs::file_magic FileType =
-        llvm::sys::fs::identify_magic(_buffer->getBuffer());
-
-    switch (FileType) {
-    case llvm::sys::fs::file_magic::elf_relocatable:
-    case llvm::sys::fs::file_magic::elf_shared_object:
-      // Call the default reader to read object files and shared objects
-      return _elfLinkingContext.getDefaultReader().parseFile(_buffer, _files);
-
-    case llvm::sys::fs::file_magic::archive: {
-      // Process archive files. If Whole Archive option is set,
-      // parse all members of the archive.
-      error_code ec;
-      std::unique_ptr<FileArchive> fileArchive(
-          new FileArchive(ctx, std::move(_buffer), ec, _isWholeArchive));
-      if (_isWholeArchive) {
-        fileArchive->parseAllMembers(_files);
-        _archiveFile = std::move(fileArchive);
-      } else {
-        _files.push_back(std::move(fileArchive));
-      }
-      return ec;
-    }
-
-    default:
-      // Process Linker script
-      return _elfLinkingContext.getLinkerScriptReader().parseFile(_buffer,
-                                                                  _files);
-    }
-  }
+  error_code parse(const LinkingContext &, raw_ostream &);
 
   /// \brief This is used by Group Nodes, when there is a need to reset the
   /// the file to be processed next. When handling a group node that contains
   /// Input elements, if the group node has to be reprocessed, the linker needs
   /// to start processing files as part of the inputelement from beginning.
   /// reset the next file index to 0 only if the node is an archive library or
   /// a shared library
   virtual void resetNextIndex() {
private:
   const ELFLinkingContext &_elfLinkingContext;
   bool _isWholeArchive;
   bool _asNeeded;
   bool _isDashlPrefix;
   std::vector<StringRef> _libraryPaths;
   std::unique_ptr<FileArchive> _archiveFile;
 };
 
+/// \brief Parse GNU Linker scripts.
+
+class GNULdScript : public FileNode {
+public:
+  GNULdScript(ELFLinkingContext &ctx, StringRef userPath, int64_t ordinal)
+      : FileNode(userPath, ordinal), _elfLinkingContext(ctx) {}
+
+  static inline bool classof(const InputElement *a) {
+    return a->kind() == InputElement::Kind::File;
+  }
+
+  /// \brief Is this node part of resolution ?
+  virtual bool isHidden() const { return true; }
+
+  /// \brief Validate the options
+  virtual bool validate() {
+    (void)_elfLinkingContext;
+    return true;
+  }
+
+  /// \brief Dump the Linker script.
+  virtual bool dump(raw_ostream &) { return true; }
+
+  /// \brief Parse the linker script.
+  virtual error_code parse(const LinkingContext &, raw_ostream &);
+
+protected:
+  ELFLinkingContext &_elfLinkingContext;
+  std::unique_ptr<script::Parser> _parser;
+  std::unique_ptr<script::Lexer> _lexer;
+  script::LinkerScript *_linkerScript;
+};
+
+/// \brief Handle ELF style with GNU Linker scripts.
+class ELFGNULdScript : public GNULdScript {
+public:
+  ELFGNULdScript(ELFLinkingContext &ctx, StringRef userPath, int64_t ordinal)
+      : GNULdScript(ctx, userPath, ordinal) {}
+
+  virtual error_code parse(const LinkingContext &ctx, raw_ostream &diagnostics);
+
+  virtual ExpandType expandType() const {
+    return InputElement::ExpandType::ExpandOnly;
+  }
+
+  /// Unused functions for ELFGNULdScript Nodes.
+  virtual ErrorOr<File &> getNextFile() {
+    return make_error_code(InputGraphError::no_more_files);
+  }
+
+  /// Return the elements that we would want to expand with.
+  range<InputGraph::InputElementIterT> expandElements() {
+    return make_range(_expandElements.begin(), _expandElements.end());
+  }
+
+  virtual void setResolveState(uint32_t) {}
+
+  virtual uint32_t getResolveState() const {
+    llvm_unreachable("cannot use this function: getResolveState");
+  }
+
+  // Do nothing here.
+  virtual void resetNextIndex() {}
+
+private:
+  InputGraph::InputElementVectorT _expandElements;
+};
+
 /// \brief Represents a ELF control node
 class ELFGroup : public Group {
 public:
   ELFGroup(ELFLinkingContext &ctx, int64_t ordinal)
       : Group(ordinal), _elfLinkingContext(ctx) {}
 
   static inline bool classof(const InputElement *a) {
     return a->kind() == InputElement::Kind::Control;

include/lld/ReaderWriter/ELFLinkingContext.h

 
 #include "lld/ReaderWriter/Reader.h"
 #include "lld/ReaderWriter/Writer.h"
 
 #include "llvm/ADT/Triple.h"
 #include "llvm/Object/ELF.h"
 #include "llvm/Support/ELF.h"
 
+#include <map>
 #include <memory>
 
 namespace lld {
 class DefinedAtom;
 class Reference;
+class SharedLibraryFile;
 
 namespace elf {
 template <typename ELFT> class TargetHandler;
 }
 
 class TargetHandlerBase {
 public:
   virtual ~TargetHandlerBase() {}
public:
   virtual StringRef getInterpreter() const {
     if (_dynamicLinkerArg)
       return _dynamicLinkerPath;
     return getDefaultInterpreter();
   }
 
   virtual Reader &getDefaultReader() const { return *_elfReader; }
 
-  virtual Reader &getLinkerScriptReader() const { return *_linkerScriptReader; }
-
   /// \brief Does the output have dynamic sections.
   virtual bool isDynamic() const;
 
   /// \brief Is the relocation a relative relocation
   virtual bool isRelativeReloc(const Reference &r) const;
 
   template <typename ELFT>
   lld::elf::TargetHandler<ELFT> &getTargetHandler() const {
public:
   void addRpathLink(StringRef path) {
    _rpathLinkList.push_back(path);
   }
 
   range<const StringRef *> getRpathLinkList() const {
     return _rpathLinkList;
   }
 
+  StringRef allocateString(StringRef ref) const {
+    char *x = _allocator.Allocate<char>(ref.size() + 1);
+    memcpy(x, ref.data(), ref.size());
+    x[ref.size()] = '\0';
+    return x;
+  }
+
+  virtual bool addUndefinedAtomsFromSharedLibrary(const SharedLibraryFile *s) {
+    if (_undefinedAtomsFromFile.find(s) != _undefinedAtomsFromFile.end())
+      return false;
+    _undefinedAtomsFromFile[s] = true;
+    return true;
+  }
+
 private:
   ELFLinkingContext() LLVM_DELETED_FUNCTION;
 
 protected:
   ELFLinkingContext(llvm::Triple, std::unique_ptr<TargetHandlerBase>);
 
   virtual Writer &writer() const;
 
protected:
   bool _runLayoutPass;
   bool _useShlibUndefines;
   bool _dynamicLinkerArg;
   bool _noAllowDynamicLibraries;
   OutputMagic _outputMagic;
   StringRefVector _inputSearchPaths;
   std::unique_ptr<Reader> _elfReader;
   std::unique_ptr<Writer> _writer;
-  std::unique_ptr<Reader> _linkerScriptReader;
   StringRef _dynamicLinkerPath;
   StringRefVector _initFunctions;
   StringRefVector _finiFunctions;
   StringRef _sysrootPath;
   StringRef _soname;
   StringRefVector _rpathList;
   StringRefVector _rpathLinkList;
+  std::map<const SharedLibraryFile *, bool> _undefinedAtomsFromFile;
 };
 } // end namespace lld
 
 #endif

include/lld/ReaderWriter/ReaderLinkerScript.h

-//===- lld/ReaderWriter/ReaderLinkerScript.h ------------------------------===//
-//
-//                             The LLVM Linker
-//
-// This file is distributed under the University of Illinois Open Source
-// License. See LICENSE.TXT for details.
-//
-//===----------------------------------------------------------------------===//
-
-#ifndef LLD_READER_WRITER_READER_LINKER_SCRIPT_H
-#define LLD_READER_WRITER_READER_LINKER_SCRIPT_H
-
-#include "lld/Core/LLVM.h"
-#include "lld/ReaderWriter/Reader.h"
-
-namespace lld {
-class File;
-class LinkingContext;
-
-/// \brief ReaderLinkerScript is a class for reading linker scripts
-class ReaderLinkerScript : public Reader {
-public:
-  explicit ReaderLinkerScript(const LinkingContext &context)
-      : Reader(context) {}
-
-  /// \brief Returns a vector of Files that are contained in the archive file
-  ///        pointed to by the Memorybuffer
-  error_code parseFile(std::unique_ptr<MemoryBuffer> &mb,
-                       std::vector<std::unique_ptr<File> > &result) const;
-};
-
-} // end namespace lld
-
-#endif

lib/Core/Error.cpp

virtual llvm::error_condition default_error_condition(int ev) const {
     return llvm::errc::invalid_argument;
   }
 };
 
 const llvm::error_category &lld::InputGraphErrorCategory() {
   static _InputGraphErrorCategory i;
   return i;
 }
+
+class _ReaderErrorCategory : public llvm::_do_message {
+public:
+  virtual const char *name() const { return "lld.inputGraph.parse"; }
+
+  virtual std::string message(int ev) const {
+    if (ReaderError(ev) == ReaderError::success)
+      return "Success";
+    else if (ReaderError(ev) == ReaderError::unknown_file_format)
+      return "File format for the input file is not recognized by this flavor";
+
+    llvm_unreachable("An enumerator of ReaderError does not have a "
+                     "message defined.");
+  }
+
+  virtual llvm::error_condition default_error_condition(int ev) const {
+    if (ReaderError(ev) == ReaderError::success)
+      return llvm::errc::success;
+    return llvm::errc::invalid_argument;
+  }
+};
+
+const llvm::error_category &lld::ReaderErrorCategory() {
+  static _ReaderErrorCategory i;
+  return i;
+}

lib/Core/InputGraph.cpp

for (auto &ie : _inputArgs)
     ie->setOrdinal(++_ordinal);
   return true;
 }
 
 void InputGraph::doPostProcess() {
   std::stable_sort(_inputArgs.begin(), _inputArgs.end(), sortInputElements);
 }
 
+void InputGraph::normalize() {
+  auto iterb = _inputArgs.begin();
+  auto itere = _inputArgs.end();
+  auto currentIter = _inputArgs.begin();
+  bool replaceCurrentNode = false;
+  bool expand = false;
+
+  std::vector<std::unique_ptr<InputElement> > _workInputArgs;
+  while (iterb != itere) {
+    replaceCurrentNode = false;
+    expand = false;
+    InputElement::ExpandType expandType = (*iterb)->expandType();
+    if (expandType == InputElement::ExpandType::ReplaceAndExpand) {
+      replaceCurrentNode = true;
+      expand = true;
+    } else if (expandType == InputElement::ExpandType::ExpandOnly) {
+      replaceCurrentNode = false;
+      expand = true;
+    }
+    currentIter = iterb++;
+    if (expand)
+      _workInputArgs.insert(
+          _workInputArgs.end(),
+          std::make_move_iterator((*currentIter)->expandElements().begin()),
+          std::make_move_iterator((*currentIter)->expandElements().end()));
+    if (!replaceCurrentNode)
+      _workInputArgs.push_back(std::move(*currentIter));
+  }
+  _inputArgs = std::move(_workInputArgs);
+}
+
 bool InputGraph::validate() {
   for (auto &ie : _inputArgs)
     if (!ie->validate())
       return false;
   return true;
 }
 
 bool InputGraph::dump(raw_ostream &diagnostics) {
else if (position == InputGraph::Position::END)
     pos = _inputArgs.size();
   _inputArgs.insert(_inputArgs.begin() + pos, std::move(element));
 }
 
 /// \brief Helper functions for the resolver
 ErrorOr<InputElement *> InputGraph::getNextInputElement() {
   if (_nextElementIndex >= _inputArgs.size())
     return make_error_code(InputGraphError::no_more_elements);
-  return _inputArgs[_nextElementIndex++].get();
+  auto elem = _inputArgs[_nextElementIndex++].get();
+  // Do not return Hidden elements.
+  if (!elem->isHidden())
+    return elem;
+  return getNextInputElement();
 }
 
 /// \brief Set the index on what inputElement has to be returned
 error_code InputGraph::setNextElementIndex(uint32_t index) {
   if (index > _inputArgs.size())
     return make_error_code(llvm::errc::invalid_argument);
   _nextElementIndex = index;
   return error_code::success();

lib/Core/Resolver.cpp

 
 } // namespace
 
 // called before the first atom in any file is added with doAtom()
 void Resolver::doFile(const File &file) {}
 
 void Resolver::handleFile(const File &file) {
   uint32_t resolverState = Resolver::StateNoChange;
+  const SharedLibraryFile *sharedLibraryFile =
+      llvm::dyn_cast<SharedLibraryFile>(&file);
   doFile(file);
   for (const DefinedAtom *atom : file.defined()) {
     doDefinedAtom(*atom);
     resolverState |= StateNewDefinedAtoms;
   }
+  if (!sharedLibraryFile ||
+      _context.addUndefinedAtomsFromSharedLibrary(sharedLibraryFile)) {
     for (const UndefinedAtom *undefAtom : file.undefined()) {
       doUndefinedAtom(*undefAtom);
       resolverState |= StateNewUndefinedAtoms;
     }
+  }
   for (const SharedLibraryAtom *shlibAtom : file.sharedLibrary()) {
     doSharedLibraryAtom(*shlibAtom);
     resolverState |= StateNewSharedLibraryAtoms;
   }
   for (const AbsoluteAtom *absAtom : file.absolute()) {
     doAbsoluteAtom(*absAtom);
     resolverState |= StateNewAbsoluteAtoms;
   }
case File::kindArchiveLibrary:
         file->setOrdinal(_context.getNextOrdinalAndIncrement());
       handleArchiveFile(*file);
       break;
     case File::kindSharedLibrary:
       if (!file->hasOrdinal())
         file->setOrdinal(_context.getNextOrdinalAndIncrement());
       handleSharedLibrary(*file);
       break;
-    case File::kindLinkerScript:
-      llvm_unreachable("linker script should not be returned by nextFile()");
     }
   }
 }
 
 // switch all references to undefined or coalesced away atoms
 // to the new defined atom
 void Resolver::updateReferences() {
   ScopedTask task(getDefaultDomain(), "updateReferences");

lib/Driver/CMakeLists.txt

 tablegen(LLVM WinLinkOptions.inc -gen-opt-parser-defs)
 add_public_tablegen_target(DriverOptionsTableGen)
 
 add_lld_library(lldDriver
   CoreDriver.cpp
   DarwinLdDriver.cpp
   Driver.cpp
   GnuLdDriver.cpp
+  GnuLdInputGraph.cpp
   WinLinkDriver.cpp
   UniversalDriver.cpp
   )
 
 add_dependencies(lldDriver DriverOptionsTableGen)
 
 target_link_libraries(lldDriver
   lldPasses

lib/Driver/Driver.cpp

bool Driver::link(LinkingContext &context, raw_ostream &diagnostics) {
 
   bool fail = false;
 
   // Read inputs
   ScopedTask readTask(getDefaultDomain(), "Read Args");
   TaskGroup tg;
   std::mutex diagnosticsMutex;
   for (auto &ie : inputGraph.inputElements()) {
+    if (ie->isHidden())
+      continue;
     tg.spawn([&] {
       // Writes to the same output stream is not guaranteed to be thread-safe.
       // We buffer the diagnostics output to a separate string-backed output
       // stream, acquire the lock, and then print it out.
       std::string buf;
       llvm::raw_string_ostream stream(buf);
 
       if (error_code ec = ie->parse(context, stream)) {

lib/Driver/GnuLdDriver.cpp

 
 
 // Create OptTable class for parsing actual command line arguments
 class GnuLdOptTable : public llvm::opt::OptTable {
 public:
   GnuLdOptTable() : OptTable(infoTable, llvm::array_lengthof(infoTable)){}
 };
 
+// Get the Input file magic for creating appropriate InputGraph nodes.
+error_code getFileMagic(ELFLinkingContext &ctx, StringRef path,
+                        std::vector<StringRef> &searchPaths,
+                        llvm::sys::fs::file_magic &magic) {
+  error_code ec = llvm::sys::fs::identify_magic(path, magic);
+  if (ec)
+    return ec;
+  switch (magic) {
+  case llvm::sys::fs::file_magic::archive:
+  case llvm::sys::fs::file_magic::elf_relocatable:
+  case llvm::sys::fs::file_magic::elf_shared_object:
+  case llvm::sys::fs::file_magic::unknown:
+    return error_code::success();
+  default:
+    break;
+  }
+  return make_error_code(ReaderError::unknown_file_format);
+}
+
 } // namespace
 
 llvm::ErrorOr<StringRef> ELFFileNode::getPath(const LinkingContext &) const {
   if (!_isDashlPrefix)
     return _path;
   return _elfLinkingContext.searchLibrary(_path, _libraryPaths);
 }
 
for (auto inputArg : *parsedArgs) {
 
     case OPT_end_group:
       dyn_cast<ControlNode>(controlNodeStack.top())->processControlExit();
       controlNodeStack.pop();
       break;
 
     case OPT_INPUT:
     case OPT_l: {
-      std::unique_ptr<InputElement> inputFile(new ELFFileNode(
-          *ctx, inputArg->getValue(), searchPath, index++, isWholeArchive,
-          asNeeded, inputArg->getOption().getID() == OPT_l));
+      bool isDashlPrefix = (inputArg->getOption().getID() == OPT_l);
+      bool isELFFileNode = true;
+      StringRef userPath = inputArg->getValue();
+      std::string resolvedInputPath = userPath;
+      if (isDashlPrefix) {
+        ErrorOr<StringRef> resolvedPath =
+            ctx->searchLibrary(userPath, searchPath);
+        if (!resolvedPath) {
+          diagnostics << " Unable to find library -l" << userPath << "\n";
+          return false;
+        }
+        resolvedInputPath = resolvedPath->str();
+      }
+      llvm::sys::fs::file_magic magic = llvm::sys::fs::file_magic::unknown;
+      error_code ec = getFileMagic(*ctx, resolvedInputPath, searchPath, magic);
+      if (ec) {
+        diagnostics << "lld: unknown input file format for file " << userPath
+                    << "\n";
+        return false;
+      }
+      if ((!userPath.endswith(".objtxt")) &&
+          (magic == llvm::sys::fs::file_magic::unknown))
+        isELFFileNode = false;
+
+      FileNode *inputNode = nullptr;
+      if (isELFFileNode)
+        inputNode = new ELFFileNode(*ctx, userPath, searchPath, index++,
+                                    isWholeArchive, asNeeded, isDashlPrefix);
+      else {
+        inputNode = new ELFGNULdScript(*ctx, resolvedInputPath, index++);
+        ec = inputNode->parse(*ctx, diagnostics);
+        if (ec) {
+          diagnostics << userPath << ": Error parsing linker script"
+                      << "\n";
+          return false;
+        }
+      }
+
+      std::unique_ptr<InputElement> inputFile(inputNode);
       if (controlNodeStack.empty())
         inputGraph->addInputElement(std::move(inputFile));
       else
         dyn_cast<ControlNode>(controlNodeStack.top())
             ->processInputElement(std::move(inputFile));
       break;
     }
 
bool GnuLdDriver::parse(int argc, const char *argv[],
 
   if (ctx->outputFileType() == LinkingContext::OutputFileType::YAML)
     inputGraph->dump(diagnostics);
 
   // Validate the combination of options used.
   if (!ctx->validate(diagnostics))
     return false;
 
+  inputGraph->normalize();
+
   ctx->setInputGraph(std::move(inputGraph));
 
   context.swap(ctx);
 
   return true;
 }
 
 /// Get the default target triple based on either the program name

lib/Driver/GnuLdInputGraph.cpp

+//===- lib/Driver/GnuLdInputGraph.cpp -------------------------------------===//
+//
+//                             The LLVM Linker
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "lld/Driver/GnuLdInputGraph.h"
+#include "lld/ReaderWriter/LinkerScript.h"
+
+using namespace lld;
+
+/// \brief Parse the input file to lld::File.
+error_code ELFFileNode::parse(const LinkingContext &ctx,
+                              raw_ostream &diagnostics) {
+  ErrorOr<StringRef> filePath = getPath(ctx);
+  if (!filePath)
+    return error_code(filePath);
+
+  if (error_code ec = getBuffer(*filePath))
+    return ec;
+
+  if (ctx.logInputFiles())
+    diagnostics << *filePath << "\n";
+
+  if (filePath->endswith(".objtxt"))
+    return ctx.getYAMLReader().parseFile(_buffer, _files);
+
+  // Identify File type
+  llvm::sys::fs::file_magic FileType =
+      llvm::sys::fs::identify_magic(_buffer->getBuffer());
+
+  switch (FileType) {
+  case llvm::sys::fs::file_magic::elf_relocatable:
+  case llvm::sys::fs::file_magic::elf_shared_object:
+    // Call the default reader to read object files and shared objects
+    return _elfLinkingContext.getDefaultReader().parseFile(_buffer, _files);
+
+  case llvm::sys::fs::file_magic::archive: {
+    // Process archive files. If Whole Archive option is set,
+    // parse all members of the archive.
+    error_code ec;
+    std::unique_ptr<FileArchive> fileArchive(
+        new FileArchive(ctx, std::move(_buffer), ec, _isWholeArchive));
+    if (_isWholeArchive) {
+      fileArchive->parseAllMembers(_files);
+      _archiveFile = std::move(fileArchive);
+    } else {
+      _files.push_back(std::move(fileArchive));
+    }
+    return ec;
+  }
+  default:
+    break;
+  }
+
+  return error_code::success();
+}
+
+error_code GNULdScript::parse(const LinkingContext &ctx,
+                              raw_ostream &diagnostics) {
+  ErrorOr<StringRef> filePath = getPath(ctx);
+  if (!filePath)
+    return error_code(filePath);
+
+  if (error_code ec = getBuffer(*filePath))
+    return ec;
+
+  if (ctx.logInputFiles())
+    diagnostics << *filePath << "\n";
+
+  _lexer.reset(new script::Lexer(std::move(_buffer)));
+  _parser.reset(new script::Parser(*_lexer.get()));
+
+  _linkerScript = _parser->parse();
+
+  if (!_linkerScript)
+    return LinkerScriptReaderError::parse_error;
+
+  return error_code::success();
+}
+
+error_code ELFGNULdScript::parse(const LinkingContext &ctx,
+                                 raw_ostream &diagnostics) {
+  int64_t index = 0;
+  std::vector<StringRef> searchPath;
+  if (error_code ec = GNULdScript::parse(ctx, diagnostics))
+    return ec;
+  for (const auto &c : _linkerScript->_commands) {
+    if (auto group = dyn_cast<script::Group>(c)) {
+      std::unique_ptr<InputElement> controlStart(
+          new ELFGroup(_elfLinkingContext, index++));
+      for (auto &path : group->getPaths()) {
+        // TODO : Propagate SearchPath, Set WholeArchive/dashlPrefix
+        auto inputNode = new ELFFileNode(
+            _elfLinkingContext, _elfLinkingContext.allocateString(path._path),
+            searchPath, index++, false, path._asNeeded, false);
+        std::unique_ptr<InputElement> inputFile(inputNode);
+        dyn_cast<ControlNode>(controlStart.get())
+            ->processInputElement(std::move(inputFile));
+      }
+      _expandElements.push_back(std::move(controlStart));
+    }
+  }
+  return error_code::success();
+}

lib/ReaderWriter/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS object)
 
 add_subdirectory(ELF)
 add_subdirectory(MachO)
 add_subdirectory(Native)
 add_subdirectory(PECOFF)
 add_subdirectory(YAML)
 add_lld_library(lldReaderWriter
   CoreLinkingContext.cpp
   LinkerScript.cpp
   Reader.cpp
-  ReaderLinkerScript.cpp
   Writer.cpp
   )
 
 target_link_libraries(lldReaderWriter
   lldCore
   lldPasses
   )

lib/ReaderWriter/ELF/ELFLinkingContext.cpp

 #include "File.h"
 #include "TargetHandler.h"
 #include "Targets.h"
 
 #include "lld/Core/Instrumentation.h"
 #include "lld/Passes/LayoutPass.h"
 #include "lld/Passes/RoundTripNativePass.h"
 #include "lld/Passes/RoundTripYAMLPass.h"
-#include "lld/ReaderWriter/ReaderLinkerScript.h"
 
 #include "llvm/ADT/Triple.h"
 #include "llvm/Support/ELF.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Path.h"
 
 namespace lld {
 

 StringRef ELFLinkingContext::entrySymbolName() const {
   if (_outputELFType == elf::ET_EXEC && _entrySymbolName.empty())
     return "_start";
   return _entrySymbolName;
 }
 
 bool ELFLinkingContext::validateImpl(raw_ostream &diagnostics) {
   _elfReader = createReaderELF(*this);
-  _linkerScriptReader.reset(new ReaderLinkerScript(*this));
   switch (outputFileType()) {
   case LinkingContext::OutputFileType::YAML:
     _writer = createWriterYAML(*this);
     break;
   case LinkingContext::OutputFileType::Native:
     llvm_unreachable("Unimplemented");
     break;
   default:

lib/ReaderWriter/ReaderLinkerScript.cpp

-//===- lib/ReaderWriter/ReaderLinkerScript.cpp ----------------------------===//
-//
-//                             The LLVM Linker
-//
-// This file is distributed under the University of Illinois Open Source
-// License. See LICENSE.TXT for details.
-//
-//===----------------------------------------------------------------------===//
-
-#include "lld/ReaderWriter/ReaderLinkerScript.h"
-
-#include "lld/Core/Error.h"
-#include "lld/Core/File.h"
-#include "lld/ReaderWriter/LinkerScript.h"
-
-using namespace lld;
-using namespace script;
-
-namespace {
-class LinkerScriptFile : public File {
-public:
-  static ErrorOr<std::unique_ptr<LinkerScriptFile> >
-  create(const LinkingContext &context,
-         std::unique_ptr<MemoryBuffer> mb) {
-    std::unique_ptr<LinkerScriptFile> file(
-        new LinkerScriptFile(context, std::move(mb)));
-    file->_script = file->_parser.parse();
-    if (!file->_script)
-      return LinkerScriptReaderError::parse_error;
-    return std::move(file);
-  }
-
-  static inline bool classof(const File *f) {
-    return f->kind() == kindLinkerScript;
-  }
-
-  virtual const LinkingContext &getLinkingContext() const { return _context; }
-
-  virtual const atom_collection<DefinedAtom> &defined() const {
-    return _definedAtoms;
-  }
-
-  virtual const atom_collection<UndefinedAtom> &undefined() const {
-    return _undefinedAtoms;
-  }
-
-  virtual const atom_collection<SharedLibraryAtom> &sharedLibrary() const {
-    return _sharedLibraryAtoms;
-  }
-
-  virtual const atom_collection<AbsoluteAtom> &absolute() const {
-    return _absoluteAtoms;
-  }
-
-  const LinkerScript *getScript() { return _script; }
-
-private:
-  LinkerScriptFile(const LinkingContext &context,
-                   std::unique_ptr<MemoryBuffer> mb)
-      : File(mb->getBufferIdentifier(), kindLinkerScript), _context(context),
-        _lexer(std::move(mb)), _parser(_lexer), _script(nullptr) {}
-
-  const LinkingContext &_context;
-  atom_collection_vector<DefinedAtom> _definedAtoms;
-  atom_collection_vector<UndefinedAtom> _undefinedAtoms;
-  atom_collection_vector<SharedLibraryAtom> _sharedLibraryAtoms;
-  atom_collection_vector<AbsoluteAtom> _absoluteAtoms;
-  Lexer _lexer;
-  Parser _parser;
-  const LinkerScript *_script;
-};
-} // end anon namespace
-
-namespace lld {
-error_code ReaderLinkerScript::parseFile(
-    std::unique_ptr<MemoryBuffer> &mb,
-    std::vector<std::unique_ptr<File> > &result) const {
-  auto lsf = LinkerScriptFile::create(_context, std::move(mb));
-  if (!lsf)
-    return lsf;
-  const LinkerScript *ls = (*lsf)->getScript();
-  result.push_back(std::move(*lsf));
-  for (const auto &c : ls->_commands) {
-    if (auto group = dyn_cast<lld::script::Group>(c))
-      for (const auto &path : group->getPaths()) {
-        OwningPtr<MemoryBuffer> opmb;
-        if (error_code ec =
-                MemoryBuffer::getFileOrSTDIN(path._path, opmb))
-          return ec;
-        std::unique_ptr<MemoryBuffer> eachMB(opmb.take());
-        if (error_code ec =
-                _context.getDefaultReader().parseFile(eachMB, result))
-          return ec;
-      }
-  }
-  return error_code::success();
-}
-} // end namespace lld

unittests/DriverTests/GnuLdDriverTest.cpp

protected:
   virtual const LinkingContext *linkingContext() { return _context.get(); }
 };
 
 TEST_F(GnuLdParserTest, Empty) {
   EXPECT_FALSE(parse("ld", nullptr));
   EXPECT_EQ(linkingContext(), nullptr);
   EXPECT_EQ("No input files\n", errorMessage());
 }
-
-TEST_F(GnuLdParserTest, Basic) {
-  EXPECT_TRUE(parse("ld", "infile.o", nullptr));
-  EXPECT_NE(linkingContext(), nullptr);
-  EXPECT_EQ("a.out", linkingContext()->outputPath());
-  EXPECT_EQ(1, inputFileCount());
-  EXPECT_EQ("infile.o", inputFile(0));
-  EXPECT_FALSE(_context->outputFileType() ==
-               LinkingContext::OutputFileType::YAML);
-}
-
-TEST_F(GnuLdParserTest, ManyOptions) {
-  EXPECT_TRUE(parse("ld", "-entry", "_start", "-o", "outfile",
-                    "--output-filetype=yaml", "infile.o", nullptr));
-  EXPECT_NE(linkingContext(), nullptr);
-  EXPECT_EQ("outfile", linkingContext()->outputPath());
-  EXPECT_EQ("_start", linkingContext()->entrySymbolName());
-  EXPECT_TRUE(_context->outputFileType() ==
-              LinkingContext::OutputFileType::YAML);
-}
-
 }  // end anonymous namespace