[LLD/ELF2] Initial support for local symbols in symtab

[davide]

Summary

Maybe this can be shrunk, and local.s needs some cleanup. Would appreciate comments, as I have other patches building on top of this one (I'll propose later if we're OK with the scaffolding).

[silvas]

test/elf2/local.s

15

Offset doesn't seem to matter for this test?

17–18

Do we care about these indices? We aren't checking that 5 and 4 are a particular thing.

33

I don't think we need to check the exact string table index.

[denis-protivensky]

ELF/InputFiles.cpp

78

return llvm::make_range... for consistency

ELF/Writer.cpp

538–539

Maybe better to rename BufStart to something like BufVisible since it points to the start of visible symbols?

621–622

Looks like Syms = BufStart = pointer to the buffer with STN_UNDEF and local symbols already skipped, so you need to pass Syms + NumVisible as the array end.

[• rafael]

Please git-clang-format it.

The current behavior is equivalent to --discard-all. Please add that
option and use it in the existing tests.

For local.s, use FileCheck variables to check what Link points to or
just use "Link :". We test that in other tests already.

There is duplicated code between getNonLocalSymbols and
getLocalSymbols. Can you create a getSymbols(bool Local) helper?

This crashes trying to add local symbols to the dynamic symbol table:

$ cat t.s

.global foo

foo:
bar:
$ clang -c t.s
$ ./build/bin/lld -flavor gnu2 t.o -o t.so -shared
lld: /home/espindola/llvm/llvm/include/llvm/MC/StringTableBuilder.h:54:
size_t llvm::StringTableBuilder::getOffset(llvm::StringRef) const:
Assertion `I != StringIndexMap.end() && "String is not in table!"'
failed.

uint8_t *BufStart = Buf;

The name is not very appropriate anymore. It now points to the start
of the global symbols.

Cheers,
Rafael

[davide]

Hopefully addressed (almost) all the comments.

• Cleaned up tests
• Fixup crash reported by Rafael + added a new test for that
• Rename variables

I'll refactor getLocalSymbols() and getNonLocalSymbols() in a separate NFC patch. I put an XXX on top of the function so I won't forget.
Also, I think I ran clang-format correctly on my svn checkout, but please double check.

[davide]

Fix typo in comment: s/dynsm/dynsym.
Also, previous patch introduces -discard-all option and uses it in test, as proposed by Rafael.

[davide]

• Clean up tests a little bit more.

[• rafael]

Looking pretty good.

Is there a big reason for not refactoring the duplicated code in this patch?

ELF/InputFiles.cpp

75

s/XXX/FIXME/

Please do fix it on the next patch.

84

This should be >=. Which is one of the reason I asked to avoid the code duplication.

test/elf2/local-dynamic.s

5

Instead of -s you can use

llvm-leadobj -t -dyn-symbols.

[davide]

New patch:

• refactor to reduce duplication (thank you for keeping me honest)
• use -dynsym instead of -s for test

[• rafael]

LGTM with a nit.

ELF/InputFiles.cpp

66

We normally put {} in both the if and the else or in neither.

71

You don't need this special case. The line bellow becomes

return llvm::make_range(Syms.begin() + 1, Syms.begin() + 1);

Which is a valid empty range.

With that you can also drop the {

ELF/Config.h

 namespace lld {
 namespace elf2 {
 
 struct Configuration {
   llvm::StringRef OutputFile;
   llvm::StringRef DynamicLinker;
   std::string RPath;
   bool Shared = false;
+  bool DiscardAll = false;
 };
 
 extern Configuration *Config;
 
 } // namespace elf2
 } // namespace lld
 
 #endif

ELF/Driver.cpp

void LinkerDriver::link(ArrayRef<const char *> ArgsArr) {
   for (auto *Arg : Args.filtered(OPT_rpath))
     RPaths.push_back(Arg->getValue());
   if (!RPaths.empty())
     Config->RPath = llvm::join(RPaths.begin(), RPaths.end(), ":");
 
   if (Args.hasArg(OPT_shared))
     Config->Shared = true;
 
+  if (Args.hasArg(OPT_discard_all))
+    Config->DiscardAll = true;
+
   // Create a list of input files.
   std::vector<MemoryBufferRef> Inputs;
 
   for (auto *Arg : Args.filtered(OPT_INPUT)) {
     StringRef Path = Arg->getValue();
     Inputs.push_back(openFile(Path));
   }
 

ELF/InputFiles.h

 public:
   typedef typename llvm::object::ELFFile<ELFT>::Elf_Shdr Elf_Shdr;
   typedef typename llvm::object::ELFFile<ELFT>::Elf_Sym_Range Elf_Sym_Range;
 
   llvm::object::ELFFile<ELFT> *getObj() const { return ELFObj.get(); }
 
   uint16_t getEMachine() const { return getObj()->getHeader()->e_machine; }
 
+  StringRef getStringTable() const { return StringTable; }
+
 protected:
   std::unique_ptr<llvm::object::ELFFile<ELFT>> ELFObj;
   const Elf_Shdr *Symtab = nullptr;
   StringRef StringTable;
   Elf_Sym_Range getNonLocalSymbols();
 
   void openELF(MemoryBufferRef MB);
 };
public:
 
   SymbolBody *getSymbolBody(uint32_t SymbolIndex) {
     uint32_t FirstNonLocal = this->Symtab->sh_info;
     if (SymbolIndex < FirstNonLocal)
       return nullptr;
     return SymbolBodies[SymbolIndex - FirstNonLocal]->getReplacement();
   }
 
+  Elf_Sym_Range getLocalSymbols();
+
 private:
   void initializeChunks();
   void initializeSymbols();
 
   SymbolBody *createSymbolBody(StringRef StringTable, const Elf_Sym *Sym);
 
   // List of all chunks defined by this file.
   std::vector<SectionChunk<ELFT> *> Chunks;

ELF/InputFiles.cpp

 typename ELFData<ELFT>::Elf_Sym_Range ELFData<ELFT>::getNonLocalSymbols() {
   if (!Symtab)
     return Elf_Sym_Range(nullptr, nullptr);
 
   ErrorOr<StringRef> StringTableOrErr =
       ELFObj->getStringTableForSymtab(*Symtab);
   error(StringTableOrErr.getError());
   StringTable = *StringTableOrErr;
 
   Elf_Sym_Range Syms = ELFObj->symbols(Symtab);
   uint32_t NumSymbols = std::distance(Syms.begin(), Syms.end());
   uint32_t FirstNonLocal = Symtab->sh_info;
   if (FirstNonLocal > NumSymbols)
     error("Invalid sh_info in symbol table");
   return llvm::make_range(Syms.begin() + FirstNonLocal, Syms.end());
 }
 
+// XXX: Factor out common code between getLocalSymbols()
+//      and getNonLocalSymbols().
+template <class ELFT>
+typename ObjectFile<ELFT>::Elf_Sym_Range ObjectFile<ELFT>::getLocalSymbols() {
+  if (!this->Symtab)
+    return Elf_Sym_Range(nullptr, nullptr);
+  Elf_Sym_Range Syms = this->ELFObj->symbols(this->Symtab);
+  uint32_t NumSymbols = std::distance(Syms.begin(), Syms.end());
+  uint32_t LastLocal = this->Symtab->sh_info - 1;
+  if (LastLocal > NumSymbols)
+    error("Invalid sh_info in symbol table");
+
+  // No local symbols.
+  if (LastLocal == 0)
+    return Elf_Sym_Range(nullptr, nullptr);
+
+  // Skip over dummy symbol.
+  return llvm::make_range(Syms.begin() + 1, Syms.begin() + LastLocal + 1);
+}
+
 template <class ELFT> void elf2::ObjectFile<ELFT>::parse() {
   this->openELF(MB);
 
   // Read section and symbol tables.
   initializeChunks();
   initializeSymbols();
 }
 

ELF/Options.td

 def dynamic_linker : Separate<["-"], "dynamic-linker">,
      HelpText<"Which dynamic linker to use">;
 
 def rpath : Separate<["-"], "rpath">,
      HelpText<"Add a DT_RUNPATH to the output">;
 
 def shared : Flag<["-"], "shared">,
      HelpText<"Build a shared object">;
+
+def discard_all : Flag<["-"], "discard-all">,
+     HelpText<"Delete all local symbols">;

ELF/Writer.cpp

 };
 
 template <class ELFT> class Writer;
 
 template <class ELFT>
 class SymbolTableSection final : public OutputSectionBase<ELFT::Is64Bits> {
 public:
   typedef typename ELFFile<ELFT>::Elf_Sym Elf_Sym;
+  typedef typename ELFFile<ELFT>::Elf_Sym_Range Elf_Sym_Range;
   typedef typename OutputSectionBase<ELFT::Is64Bits>::uintX_t uintX_t;
   SymbolTableSection(Writer<ELFT> &W, SymbolTable &Table,
                      StringTableSection<ELFT::Is64Bits> &StrTabSec)
       : OutputSectionBase<ELFT::Is64Bits>(
             StrTabSec.isDynamic() ? ".dynsym" : ".symtab",
             StrTabSec.isDynamic() ? SHT_DYNSYM : SHT_SYMTAB,
             StrTabSec.isDynamic() ? (uintX_t)SHF_ALLOC : 0),
         Table(Table), StrTabSec(StrTabSec), W(W) {
     typedef OutputSectionBase<ELFT::Is64Bits> Base;
     typename Base::HeaderT &Header = this->Header;
 
-    // For now the only local symbol is going to be the one at index 0
-    Header.sh_info = 1;
-
     Header.sh_entsize = sizeof(Elf_Sym);
     Header.sh_addralign = ELFT::Is64Bits ? 8 : 4;
   }
 
   void finalize() override {
     this->Header.sh_size = getNumSymbols() * sizeof(Elf_Sym);
     this->Header.sh_link = StrTabSec.getSectionIndex();
+    this->Header.sh_info = NumLocals + 1;
   }
 
   void writeTo(uint8_t *Buf) override;
 
   const SymbolTable &getSymTable() const { return Table; }
 
-  void addSymbol(StringRef Name) {
+  void addSymbol(StringRef Name, bool isLocal = false) {
     StrTabSec.add(Name);
     ++NumVisible;
+    if (isLocal)
+      ++NumLocals;
   }
 
   StringTableSection<ELFT::Is64Bits> &getStrTabSec() { return StrTabSec; }
   unsigned getNumSymbols() const { return NumVisible + 1; }
 
 private:
   SymbolTable &Table;
   StringTableSection<ELFT::Is64Bits> &StrTabSec;
   unsigned NumVisible = 0;
+  unsigned NumLocals = 0;
   const Writer<ELFT> &W;
 };
 
 template <class ELFT>
 class HashTableSection final : public OutputSectionBase<ELFT::Is64Bits> {
   typedef typename ELFFile<ELFT>::Elf_Word Elf_Word;
 
 public:

 // The writer writes a SymbolTable result to a file.
 template <class ELFT> class Writer {
 public:
   typedef typename ELFFile<ELFT>::uintX_t uintX_t;
   typedef typename ELFFile<ELFT>::Elf_Shdr Elf_Shdr;
   typedef typename ELFFile<ELFT>::Elf_Ehdr Elf_Ehdr;
   typedef typename ELFFile<ELFT>::Elf_Phdr Elf_Phdr;
   typedef typename ELFFile<ELFT>::Elf_Sym Elf_Sym;
+  typedef typename ELFFile<ELFT>::Elf_Sym_Range Elf_Sym_Range;
   Writer(SymbolTable *T)
       : SymTabSec(*this, *T, StrTabSec), DynSymSec(*this, *T, DynStrSec),
         HashSec(DynSymSec), DynamicSec(*T, HashSec) {}
   void run();
 
   const OutputSection<ELFT> &getBSS() const {
     assert(BSSSec);
     return *BSSSec;
static bool includeInSymtab(const SymbolBody &B) {
   if (!B.isUsedInRegularObj())
     return false;
   uint8_t V = B.getMostConstrainingVisibility();
   if (V != STV_DEFAULT && V != STV_PROTECTED)
     return false;
   return true;
 }
 
 template <class ELFT> void SymbolTableSection<ELFT>::writeTo(uint8_t *Buf) {
-  uint8_t *BufStart = Buf;
+  Buf += sizeof(Elf_Sym);
 
+  // All symbols with STB_LOCAL binding precede the weak and global symbols.
+  // .dynsm only contains global symbols.
+  if (!Config->DiscardAll && !StrTabSec.isDynamic()) {
+    for (const std::unique_ptr<ObjectFileBase> &FileB :
+         Table.getObjectFiles()) {
+      auto &File = cast<ObjectFile<ELFT>>(*FileB);
+      Elf_Sym_Range Syms = File.getLocalSymbols();
+      for (const Elf_Sym &Sym : Syms) {
+        auto *ESym = reinterpret_cast<Elf_Sym *>(Buf);
+        ErrorOr<StringRef> SymName = Sym.getName(File.getStringTable());
+        ESym->st_name = (SymName) ? StrTabSec.getFileOff(*SymName) : 0;
+        ESym->st_value = Sym.st_value;
+        ESym->st_size = Sym.st_size;
         Buf += sizeof(Elf_Sym);
+      }
+    }
+  }
+
+  uint8_t *GlobalStart = Buf;
+
   for (auto &P : Table.getSymbols()) {
     StringRef Name = P.first;
     Symbol *Sym = P.second;
     SymbolBody *Body = Sym->Body;
     if (!includeInSymtab(*Body))
       continue;
     const Elf_Sym &InputSym = cast<ELFSymbolBody<ELFT>>(Body)->Sym;
 
for (auto &P : Table.getSymbols()) {
     Buf += sizeof(Elf_Sym);
   }
 
   // The order the global symbols are in is not defined. We can use an arbitrary
   // order, but it has to be reproducible. That is true even when cross linking.
   // The default hashing of StringRef produces different results on 32 and 64
   // bit systems so we sort by st_name. That is arbitrary but deterministic.
   // FIXME: Experiment with passing in a custom hashing instead.
-  auto *Syms = reinterpret_cast<Elf_Sym *>(BufStart);
-  ++Syms;
-  array_pod_sort(Syms, Syms + NumVisible, compareSym<ELFT>);
+  auto *Syms = reinterpret_cast<Elf_Sym *>(GlobalStart);
+  array_pod_sort(Syms, Syms + NumVisible - NumLocals, compareSym<ELFT>);
 }
 
 template <bool Is64Bits>
 template <endianness E>
 void OutputSectionBase<Is64Bits>::writeHeaderTo(
     typename ELFFile<ELFType<E, Is64Bits>>::Elf_Shdr *SHdr) {
   SHdr->sh_name = Header.sh_name;
   SHdr->sh_type = Header.sh_type;
if (!Sec) {
       OutputSections.push_back(Sec);
     }
     return Sec;
   };
 
   const SymbolTable &Symtab = SymTabSec.getSymTable();
   for (const std::unique_ptr<ObjectFileBase> &FileB : Symtab.getObjectFiles()) {
     auto &File = cast<ObjectFile<ELFT>>(*FileB);
+    if (!Config->DiscardAll) {
+      Elf_Sym_Range Syms = File.getLocalSymbols();
+      for (const Elf_Sym &Sym : Syms) {
+        ErrorOr<StringRef> SymName = Sym.getName(File.getStringTable());
+        if (SymName)
+          SymTabSec.addSymbol(*SymName, true);
+      }
+    }
     for (SectionChunk<ELFT> *C : File.getChunks()) {
       if (!C)
         continue;
       const Elf_Shdr *H = C->getSectionHdr();
       OutputSection<ELFT> *Sec =
           getSection(C->getSectionName(), H->sh_type, H->sh_flags);
       Sec->addChunk(C);
     }

test/elf2/basic64be.s

 # RUN: llvm-mc -filetype=obj -triple=powerpc64-unknown-linux %s -o %t
-# RUN: lld -flavor gnu2 %t -o %t2
+# RUN: lld -flavor gnu2 -discard-all %t -o %t2
 # RUN: llvm-readobj -file-headers -sections -program-headers %t2 | FileCheck %s
 # REQUIRES: ppc
 
 # exits with return code 42 on linux
 .section        ".opd","aw"
 .global _start
 _start:
 .quad   .Lfoo,.TOC.@tocbase,0

test/elf2/local-dynamic.s

+// Check that local symbols are not inserted into dynamic table.
+// RUN: llvm-mc -filetype=obj -triple=x86_64-pc-linux %s -o %t
+// RUN: lld -flavor gnu2 %t -shared -o %t1.so
+// RUN: llvm-readobj -t -s %t1.so | FileCheck %s
+// REQUIRES: x86
+
+// CHECK:     Name: .dynsym (35)
+// CHECK-NEXT:     Type: SHT_DYNSYM (0xB)
+// CHECK-NEXT:     Flags [ (0x2)
+// CHECK-NEXT:       SHF_ALLOC (0x2)
+// CHECK-NEXT:     ]
+// CHECK-NEXT:     Address:
+// CHECK-NEXT:     Offset:
+// CHECK-NEXT:     Size:
+// CHECK-NEXT:     Link:
+// CHECK-NEXT:     Info: 1
+
+// CHECK:     Name: .symtab (71)
+// CHECK-NEXT:     Type: SHT_SYMTAB (0x2)
+// CHECK-NEXT:     Flags [ (0x0)
+// CHECK-NEXT:     ]
+// CHECK-NEXT:     Address:
+// CHECK-NEXT:     Offset:
+// CHECK-NEXT:     Size:
+// CHECK-NEXT:     Link:
+// CHECK-NEXT:     Info: 4
+
+// CHECK: Symbols [
+// CHECK-NEXT:   Symbol {
+// CHECK-NEXT:     Name:  (0)
+// CHECK-NEXT:     Value: 0x0
+// CHECK-NEXT:     Size: 0
+// CHECK-NEXT:     Binding: Local (0x0)
+// CHECK-NEXT:     Type: None (0x0)
+// CHECK-NEXT:     Other: 0
+// CHECK-NEXT:     Section: Undefined (0x0)
+// CHECK-NEXT:   }
+// CHECK-NEXT:   Symbol {
+// CHECK-NEXT:     Name: blah (49)
+// CHECK-NEXT:     Value: 0x0
+// CHECK-NEXT:     Size: 0
+// CHECK-NEXT:     Binding: Local (0x0)
+// CHECK-NEXT:     Type: None (0x0)
+// CHECK-NEXT:     Other: 0
+// CHECK-NEXT:     Section: Undefined (0x0)
+// CHECK-NEXT:   }
+// CHECK-NEXT:   Symbol {
+// CHECK-NEXT:     Name: foo (31)
+// CHECK-NEXT:     Value: 0x0
+// CHECK-NEXT:     Size: 0
+// CHECK-NEXT:     Binding: Local (0x0)
+// CHECK-NEXT:     Type: None (0x0)
+// CHECK-NEXT:     Other: 0
+// CHECK-NEXT:     Section: Undefined (0x0)
+// CHECK-NEXT:   }
+// CHECK-NEXT:   Symbol {
+// CHECK-NEXT:     Name: goo (27)
+// CHECK-NEXT:     Value: 0x0
+// CHECK-NEXT:     Size: 0
+// CHECK-NEXT:     Binding: Local (0x0)
+// CHECK-NEXT:     Type: None (0x0)
+// CHECK-NEXT:     Other: 0
+// CHECK-NEXT:     Section: Undefined (0x0)
+// CHECK-NEXT:   }
+// CHECK-NEXT:   Symbol {
+// CHECK-NEXT:     Name: _start (7)
+// CHECK-NEXT:     Value: 0x1000
+// CHECK-NEXT:     Size: 0
+// CHECK-NEXT:     Binding: Global (0x1)
+// CHECK-NEXT:     Type: None (0x0)
+// CHECK-NEXT:     Other: 0
+// CHECK-NEXT:     Section: .text (0x1)
+// CHECK-NEXT:   }
+// CHECK-NEXT: ]
+
+.global _start
+_start:
+
+blah:
+foo:
+goo:
+
+

test/elf2/local.s

+// Check that symbol table is correctly populated with local symbols.
+// RUN: llvm-mc -filetype=obj -triple=x86_64-pc-linux %s -o %t
+// RUN: lld -flavor gnu2 %t -o %t1
+// RUN: llvm-readobj -t -s %t1 | FileCheck %s
+// REQUIRES: x86
+
+// Check that Info is equal to the number of local symbols.
+// CHECK:   Section {
+// CHECK:     Name: .symtab
+// CHECK-NEXT:     Type: SHT_SYMTAB
+// CHECK-NEXT:     Flags [
+// CHECK-NEXT:     ]
+// CHECK-NEXT:     Address:
+// CHECK-NEXT:     Offset:
+// CHECK-NEXT:     Size:
+// CHECK-NEXT:     Link:
+// CHECK-NEXT:     Info: 4
+
+// CHECK: Symbols [
+// CHECK-NEXT:   Symbol {
+// CHECK-NEXT:     Name:
+// CHECK-NEXT:     Value: 0x0
+// CHECK-NEXT:     Size: 0
+// CHECK-NEXT:     Binding: Local
+// CHECK-NEXT:     Type: None
+// CHECK-NEXT:     Other: 0
+// CHECK-NEXT:     Section: Undefined
+// CHECK-NEXT:   }
+// CHECK-NEXT:   Symbol {
+// CHECK-NEXT:     Name: blah
+// CHECK-NEXT:     Value: 0x0
+// CHECK-NEXT:     Size: 0
+// CHECK-NEXT:     Binding: Local
+// CHECK-NEXT:     Type: None
+// CHECK-NEXT:     Other: 0
+// CHECK-NEXT:     Section: Undefined
+// CHECK-NEXT:   }
+// CHECK-NEXT:   Symbol {
+// CHECK-NEXT:     Name: foo
+// CHECK-NEXT:     Value: 0x0
+// CHECK-NEXT:     Size: 0
+// CHECK-NEXT:     Binding: Local
+// CHECK-NEXT:     Type: None
+// CHECK-NEXT:     Other: 0
+// CHECK-NEXT:     Section: Undefined
+// CHECK-NEXT:   }
+// CHECK-NEXT:   Symbol {
+// CHECK-NEXT:     Name: goo
+// CHECK-NEXT:     Value: 0x0
+// CHECK-NEXT:     Size: 0
+// CHECK-NEXT:     Binding: Local
+// CHECK-NEXT:     Type: None
+// CHECK-NEXT:     Other: 0
+// CHECK-NEXT:     Section: Undefined
+// CHECK-NEXT:   }
+// CHECK-NEXT:   Symbol {
+// CHECK-NEXT:     Name: _start
+// CHECK-NEXT:     Value: 0x11000
+// CHECK-NEXT:     Size: 0
+// CHECK-NEXT:     Binding: Global
+// CHECK-NEXT:     Type: None
+// CHECK-NEXT:     Other: 0
+// CHECK-NEXT:     Section: .text
+// CHECK-NEXT:   }
+// CHECK-NEXT: ]
+
+.global _start
+_start:
+
+blah:
+foo:
+goo:

test/elf2/resolution.s

 // RUN: llvm-mc -filetype=obj -triple=x86_64-pc-linux %s -o %t
 // RUN: llvm-mc -filetype=obj -triple=x86_64-pc-linux %p/Inputs/resolution.s -o %t2
-// RUN: lld -flavor gnu2 %t %t2 -o %t3
+// RUN: lld -flavor gnu2 -discard-all %t %t2 -o %t3
 // RUN: llvm-readobj -t %t3 | FileCheck %s
 // REQUIRES: x86
 
 // This is an exhaustive test for checking which symbol is kept when two
 // have the same name. Each symbol has a different size which is used
 // to see which one was chosen.
 
 // CHECK:      Symbols [