[ObjectYAML] Let dynamic entries use section names as values

[amontanez]

Summary

This change adds functionality to yaml2obj that allows dynamic entries to have string values via the String field. In this change, using a section name in the String field tells yaml2obj to use that section's address (pulled from the section header).

This is the first patch seeking to implement the associated RFC here: http://lists.llvm.org/pipermail/llvm-dev/2019-January/129231.html

Added:

• String values treated as section names that are converted to addresses.

[amontanez]

Rebase, reduced patch contents to only include one feature addition. Rather than using a unified Value field, there's now Value and String fields. This was done because a unified Value field that could accept strings and numbers made it somewhat difficult for obj2yaml to output an appropriately formatted value. This is up for discussion, though, as I'm not sure what I've done is the best solution.

[jhenderson]

In D57975#1400108, @amontanez wrote:

Rebase, reduced patch contents to only include one feature addition. Rather than using a unified Value field, there's now Value and String fields. This was done because a unified Value field that could accept strings and numbers made it somewhat difficult for obj2yaml to output an appropriately formatted value. This is up for discussion, though, as I'm not sure what I've done is the best solution.

Honestly, I feel that a unified field would be better (note, I haven't thought much about how that would affect the implementation). I think that matches what we already do with other fields like Section's Info etc. It is also more intuitive, and you don't have to handle the case of both being specified. I also don't think it matters if yaml2obj -> obj2yaml produces the output identical to the input, as long as it is still valid. In this case, it could (should?) use the actual address of the referenced section, rather than trying to figure out the section name.

llvm/lib/ObjectYAML/ELFYAML.cpp

666–667

Why has this assert disappeared? It seems unrelated to the change?

llvm/test/tools/yaml2obj/dynamic-entries.yaml

2

Nit: Avoid mixing single and double dash long args in the same test. I prefer double-dash to avoid ambiguity with option grouping, but it's not a big deal.

llvm/test/tools/yaml2obj/dynamic-entry-invalid-val.yaml

19

I'd specify the name of the section being searched for in this error message (i.e. "xxyyzz").

llvm/tools/yaml2obj/yaml2elf.cpp

388

Why can't this be done now by calling getDotDynStrSecNo?

417

Assuming you stick with different fields for string and value, I think this and the other comments and error messages in this area might need a bit of tweaking.

780

This is a personal thing, so feel free to ignore, but I find it easier to read if this explicitly compares against null.

llvm/include/llvm/ObjectYAML/ELFYAML.h

 };
 
 struct SectionOrType {
   StringRef sectionNameOrType;
 };
 
 struct DynamicEntry {
   ELF_DYNTAG Tag;
-  llvm::yaml::Hex64 Val;
+  Optional<llvm::yaml::Hex64> Val;
+  // Optional, used to specify string-based values.
+  Optional<StringRef> StrVal;
 };
 
 struct Section {
   enum class SectionKind {
     Dynamic,
     Group,
     RawContent,
     Relocation,

 };
 
 template <>
 struct MappingTraits<ELFYAML::LocalGlobalWeakSymbols> {
   static void mapping(IO &IO, ELFYAML::LocalGlobalWeakSymbols &Symbols);
 };
 
 template <> struct MappingTraits<ELFYAML::DynamicEntry> {
-  static void mapping(IO &IO, ELFYAML::DynamicEntry &Rel);
+  static void mapping(IO &IO, ELFYAML::DynamicEntry &Dyn);
+  static StringRef validate(IO &IO, ELFYAML::DynamicEntry &Dyn);
 };
 
 template <> struct MappingTraits<ELFYAML::Relocation> {
   static void mapping(IO &IO, ELFYAML::Relocation &Rel);
 };
 
 template <>
 struct MappingTraits<std::unique_ptr<ELFYAML::Section>> {

llvm/lib/ObjectYAML/ELFYAML.cpp

default:
     llvm_unreachable("Unsupported architecture");
   }
 #undef ELF_RELOC
   IO.enumFallback<Hex32>(Value);
 }
 
 void ScalarEnumerationTraits<ELFYAML::ELF_DYNTAG>::enumeration(
     IO &IO, ELFYAML::ELF_DYNTAG &Value) {
-  assert(IO.getContext() && "The IO context is not initialized");
-
 // TODO: For simplicity we do not handle target specific flags. They are
 // still supported and will be shown as a raw numeric values in the output.
 #define MIPS_DYNAMIC_TAG(name, value)
 #define HEXAGON_DYNAMIC_TAG(name, value)
 #define PPC64_DYNAMIC_TAG(name, value)
 // Also ignore marker tags such as DT_HIOS (maps to DT_VERNEEDNUM), etc.
 #define DYNAMIC_TAG_MARKER(name, value)
 
struct NormalizedMips64RelType {
   ELFYAML::ELF_REL Type2;
   ELFYAML::ELF_REL Type3;
   ELFYAML::ELF_RSS SpecSym;
 };
 
 } // end anonymous namespace
 
 void MappingTraits<ELFYAML::DynamicEntry>::mapping(IO &IO,
-                                                   ELFYAML::DynamicEntry &Rel) {
-  assert(IO.getContext() && "The IO context is not initialized");
+                                                   ELFYAML::DynamicEntry &Dyn) {
+  IO.mapRequired("Tag", Dyn.Tag);
+  IO.mapOptional("Value", Dyn.Val);
+  IO.mapOptional("String", Dyn.StrVal);
+}
 
-  IO.mapRequired("Tag", Rel.Tag);
-  IO.mapRequired("Value", Rel.Val);
+StringRef MappingTraits<ELFYAML::DynamicEntry>::validate(
+    IO &io, ELFYAML::DynamicEntry &Dyn) {
+  if (Dyn.Val.hasValue() && Dyn.StrVal.hasValue())
+    return "Dynamic entry must have a `String` or a numeric `Value`";
+  else if (!Dyn.Val.hasValue() && !Dyn.StrVal.hasValue())
+    return "Dynamic cannot have both a `String` and numeric `Value`";
+
+  return StringRef();
 }
 
 void MappingTraits<ELFYAML::Relocation>::mapping(IO &IO,
                                                  ELFYAML::Relocation &Rel) {
   const auto *Object = static_cast<ELFYAML::Object *>(IO.getContext());
   assert(Object && "The IO context is not initialized");
 
   IO.mapRequired("Offset", Rel.Offset);

llvm/test/tools/yaml2obj/dynamic-entries.yaml

+# RUN: yaml2obj %s -o %t
+# RUN: llvm-readobj -dynamic %t | FileCheck %s --check-prefix=DYNAMIC
+
+!ELF
+FileHeader:
+  Class:           ELFCLASS64
+  Data:            ELFDATA2LSB
+  Type:            ET_DYN
+  Machine:         EM_X86_64
+Sections:
+  - Name: .somestrings
+    Type: SHT_STRTAB
+    Flags: [ SHF_ALLOC, SHF_WRITE ]
+    #        "\0libsomething.so\0"
+    Content: "006c6962736f6d657468696e672e736f00"
+    Address: 0x1000
+  - Name: .dynamic
+    Type: SHT_DYNAMIC
+    Flags: [ SHF_ALLOC, SHF_WRITE ]
+    Link: .somestrings
+    Entries:
+      - Tag: DT_STRSZ
+        Value: 17
+      - Tag: DT_STRTAB
+        String: .somestrings
+# llvm-readobj requires some program headers to read .dynamic entries.
+ProgramHeaders:
+  - Type: PT_LOAD
+    Flags: [ PF_R ]
+    VAddr: 0x0000
+    PAddr: 0x0000
+    Align: 8
+    Sections:
+      - Section: .dynamic
+      - Section: .dynstr
+  - Type: PT_DYNAMIC
+    Flags: [ PF_X, PF_R ]
+    VAddr: 0x0000
+    PAddr: 0x0000
+    Sections:
+      - Section: .dynamic
+  - Type: PT_LOAD
+    Flags: [ PF_R ]
+    VAddr: 0x1000
+    PAddr: 0x1000
+    Align: 8
+    Sections:
+      - Section: .somestrings
+
+# DYNAMIC: STRSZ 17 (bytes)
+# DYNAMIC: STRTAB 0x1000{{$}}
+# DYNAMIC: NULL 0x0{{$}}

llvm/test/tools/yaml2obj/dynamic-entry-invalid-val.yaml

+# RUN: not yaml2obj %s -o %t 2>&1 | FileCheck %s
+
+!ELF
+FileHeader:
+  Class:           ELFCLASS64
+  Data:            ELFDATA2LSB
+  Type:            ET_DYN
+  Machine:         EM_X86_64
+Sections:
+  - Name: .dynamic
+    Type: SHT_DYNAMIC
+    Flags: [ SHF_ALLOC, SHF_WRITE ]
+    Entries:
+      - Tag: DT_STRSZ
+        Value: 17
+      - Tag: DT_STRTAB
+        String: xxyyzz
+
+# CHECK: error: String is not a valid section name

llvm/tools/obj2yaml/elf2yaml.cpp

class ELFDumper {
   std::error_code dumpSymbol(const Elf_Sym *Sym, const Elf_Shdr *SymTab,
                              StringRef StrTable, ELFYAML::Symbol &S);
   std::error_code dumpCommonSection(const Elf_Shdr *Shdr, ELFYAML::Section &S);
   std::error_code dumpCommonRelocationSection(const Elf_Shdr *Shdr,
                                               ELFYAML::RelocationSection &S);
   template <class RelT>
   std::error_code dumpRelocation(const RelT *Rel, const Elf_Shdr *SymTab,
                                  ELFYAML::Relocation &R);
-  
   ErrorOr<ELFYAML::DynamicSection *> dumpDynamicSection(const Elf_Shdr *Shdr);
   ErrorOr<ELFYAML::RelocationSection *> dumpRelocSection(const Elf_Shdr *Shdr);
   ErrorOr<ELFYAML::RawContentSection *>
   dumpContentSection(const Elf_Shdr *Shdr);
   ErrorOr<ELFYAML::NoBitsSection *> dumpNoBitsSection(const Elf_Shdr *Shdr);
   ErrorOr<ELFYAML::Group *> dumpGroup(const Elf_Shdr *Shdr);
   ErrorOr<ELFYAML::MipsABIFlags *> dumpMipsABIFlags(const Elf_Shdr *Shdr);
 
ELFDumper<ELFT>::dumpDynamicSection(const Elf_Shdr *Shdr) {
   auto S = make_unique<ELFYAML::DynamicSection>();
   if (std::error_code EC = dumpCommonSection(Shdr, *S))
     return EC;
 
   auto DynTagsOrErr = Obj.template getSectionContentsAsArray<Elf_Dyn>(Shdr);
   if (!DynTagsOrErr)
     return errorToErrorCode(DynTagsOrErr.takeError());
 
-  for (const Elf_Dyn &Dyn : *DynTagsOrErr)
-    S->Entries.push_back({(ELFYAML::ELF_DYNTAG)Dyn.getTag(), Dyn.getVal()});
+  for (const Elf_Dyn &Dyn : *DynTagsOrErr) {
+    ELFYAML::DynamicEntry Ent;
+    Ent.Tag = Dyn.getTag();
+    Ent.Val = Dyn.getVal();
+    S->Entries.push_back(Ent);
+  }
 
   return S.release();
 }
 
 template <class ELFT>
 ErrorOr<ELFYAML::RelocationSection *>
 ELFDumper<ELFT>::dumpRelocSection(const Elf_Shdr *Shdr) {
   auto S = make_unique<ELFYAML::RelocationSection>();

llvm/tools/yaml2obj/yaml2elf.cpp

class ELFState {
   StringTableBuilder DotStrtab{StringTableBuilder::ELF};
 
   /// The future ".shstrtab" section.
   StringTableBuilder DotShStrtab{StringTableBuilder::ELF};
 
   /// The future ".dynstr" section.
   StringTableBuilder DotDynstr{StringTableBuilder::ELF};
 
+  /// Retain a pointer to the ELFYAML::DynamicSection as state.
+  /// This is required as the .dynamic section isn't written until after the
+  /// .dynstr section.
+  ELFYAML::DynamicSection *YamlDynamicSection = nullptr;
+  unsigned DynamicSectionIdx = 0;
+
   NameToIdxMap SN2I;
   NameToIdxMap SymN2I;
   const ELFYAML::Object &Doc;
 
   bool buildSectionIndex();
   bool buildSymbolIndex(std::size_t &StartIndex,
                         const std::vector<ELFYAML::Symbol> &Symbols);
   void initELFHeader(Elf_Ehdr &Header);
   void initProgramHeaders(std::vector<Elf_Phdr> &PHeaders);
   bool initSectionHeaders(std::vector<Elf_Shdr> &SHeaders,
                           ContiguousBlobAccumulator &CBA);
   void initSymtabSectionHeader(Elf_Shdr &SHeader, SymtabType STType,
                                ContiguousBlobAccumulator &CBA);
   void initStrtabSectionHeader(Elf_Shdr &SHeader, StringRef Name,
                                StringTableBuilder &STB,
                                ContiguousBlobAccumulator &CBA);
+  void initDynamicSectionHeader(Elf_Shdr &SHeader);
+  bool finalizeDynamicSection(Elf_Shdr &DynamicHdr,
+                              std::vector<Elf_Shdr> &AllShdrs,
+                              ContiguousBlobAccumulator &CBA);
   void setProgramHeaderLayout(std::vector<Elf_Phdr> &PHeaders,
                               std::vector<Elf_Shdr> &SHeaders);
   void addSymbols(const std::vector<ELFYAML::Symbol> &Symbols,
                   std::vector<Elf_Sym> &Syms, unsigned SymbolBinding,
                   const StringTableBuilder &Strtab);
   void writeSectionContent(Elf_Shdr &SHeader,
                            const ELFYAML::RawContentSection &Section,
                            ContiguousBlobAccumulator &CBA);
   bool writeSectionContent(Elf_Shdr &SHeader,
                            const ELFYAML::RelocationSection &Section,
                            ContiguousBlobAccumulator &CBA);
   bool writeSectionContent(Elf_Shdr &SHeader, const ELFYAML::Group &Group,
                            ContiguousBlobAccumulator &CBA);
   bool writeSectionContent(Elf_Shdr &SHeader,
                            const ELFYAML::MipsABIFlags &Section,
                            ContiguousBlobAccumulator &CBA);
   void writeSectionContent(Elf_Shdr &SHeader,
                            const ELFYAML::DynamicSection &Section,
                            ContiguousBlobAccumulator &CBA);
   bool hasDynamicSymbols() const;
+  bool hasDynamicEntries() const;
   SmallVector<const char *, 5> implicitSectionNames() const;
 
   // - SHT_NULL entry (placed first, i.e. 0'th entry)
   // - symbol table (.symtab) (defaults to after last yaml section)
   // - string table (.strtab) (defaults to after .symtab)
   // - section header string table (.shstrtab) (defaults to after .strtab)
   // - dynamic symbol table (.dynsym) (defaults to after .shstrtab)
   // - dynamic string table (.dynstr) (defaults to after .dynsym)
+  // - dynamic section (.dynamic) (defaults to after .dynstr)
   unsigned getDotSymTabSecNo() const { return SN2I.get(".symtab"); }
   unsigned getDotStrTabSecNo() const { return SN2I.get(".strtab"); }
   unsigned getDotShStrTabSecNo() const { return SN2I.get(".shstrtab"); }
   unsigned getDotDynSymSecNo() const { return SN2I.get(".dynsym"); }
   unsigned getDotDynStrSecNo() const { return SN2I.get(".dynstr"); }
   unsigned getSectionCount() const { return SN2I.size() + 1; }
 
   ELFState(const ELFYAML::Object &D) : Doc(D) {}
else if (auto S = dyn_cast<ELFYAML::RelocationSection>(Sec.get())) {
         return false;
     } else if (auto S = dyn_cast<ELFYAML::NoBitsSection>(Sec.get())) {
       SHeader.sh_entsize = 0;
       SHeader.sh_size = S->Size;
       // SHT_NOBITS section does not have content
       // so just to setup the section offset.
       CBA.getOSAndAlignedOffset(SHeader.sh_offset, SHeader.sh_addralign);
     } else if (auto S = dyn_cast<ELFYAML::DynamicSection>(Sec.get())) {
-      writeSectionContent(SHeader, *S, CBA);
+      YamlDynamicSection = S;
+      DynamicSectionIdx = SHeaders.size();
+      initDynamicSectionHeader(SHeader);
     } else
       llvm_unreachable("Unknown section type");
 
     SHeaders.push_back(SHeader);
   }
   return true;
 }
 
void ELFState<ELFT>::initStrtabSectionHeader(Elf_Shdr &SHeader, StringRef Name,
   SHeader.sh_name = DotShStrtab.getOffset(Name);
   SHeader.sh_type = ELF::SHT_STRTAB;
   STB.write(CBA.getOSAndAlignedOffset(SHeader.sh_offset, SHeader.sh_addralign));
   SHeader.sh_size = STB.getSize();
   SHeader.sh_addralign = 1;
 }
 
 template <class ELFT>
+void ELFState<ELFT>::initDynamicSectionHeader(Elf_Shdr &SHeader) {
+  SHeader.sh_name = DotShStrtab.getOffset(YamlDynamicSection->Name);
+  SHeader.sh_type = ELF::SHT_DYNAMIC;
+  if (YamlDynamicSection->EntSize)
+    SHeader.sh_entsize = *YamlDynamicSection->EntSize;
+  else
+    SHeader.sh_entsize = sizeof(Elf_Dyn);
+
+  if (SHeader.sh_addralign < sizeof(Elf_Dyn)/2)
+    SHeader.sh_addralign = sizeof(Elf_Dyn)/2;
+  // TODO: Link .dynamic to .dynstr by default.
+}
+
+template <class ELFT>
+bool ELFState<ELFT>::finalizeDynamicSection(Elf_Shdr &DynamicHdr,
+                                            std::vector<Elf_Shdr> &AllShdrs,
+                                            ContiguousBlobAccumulator &CBA) {
+  std::vector<Elf_Dyn> Dyn;
+  auto printYamlError = [](ELFYAML::DynamicEntry YamlObj, StringRef After) {
+    yaml::Output Yout(errs());
+    Yout << YamlObj;
+    WithColor::error() << After << "\n";
+  };
+
+  for (const auto &YamlEntry : YamlDynamicSection->Entries) {
+    Elf_Dyn Ent;
+    Ent.d_tag = YamlEntry.Tag;
+
+    // If a section name, use that section's sh_addr.
+    if (YamlEntry.StrVal.hasValue()) {
+      unsigned SectionIdx;
+      if (!SN2I.lookup(*YamlEntry.StrVal, SectionIdx)) {
+        Ent.d_un.d_val = AllShdrs[SectionIdx].sh_addr;
+      } else {
+        printYamlError(YamlEntry,
+                       "String is not a valid section name");
+        return false;
+      }
+
+    // Otherwise, attempt to treat as a number.
+    } else if (YamlEntry.Val.hasValue()) {
+      Ent.d_un.d_val = *YamlEntry.Val;
+    } else {
+      printYamlError(YamlEntry,
+                     "Value is not a number or valid section name");
+      return false;
+    }
+    Dyn.push_back(Ent);
+  }
+
+  // Ensure DT_NULL is present.
+  if (!Dyn.empty() && Dyn.back().getTag() != ELF::DT_NULL) {
+    Elf_Dyn NullEnt;
+    NullEnt.d_tag = ELF::DT_NULL;
+    NullEnt.d_un.d_val = 0;
+    Dyn.push_back(NullEnt);
+  }
+
+  writeArrayData(
+      CBA.getOSAndAlignedOffset(DynamicHdr.sh_offset, DynamicHdr.sh_addralign),
+      makeArrayRef(Dyn));
+  DynamicHdr.sh_size = arrayDataSize(makeArrayRef(Dyn));
+
+  return true;
+}
+
+template <class ELFT>
 void ELFState<ELFT>::setProgramHeaderLayout(std::vector<Elf_Phdr> &PHeaders,
                                             std::vector<Elf_Shdr> &SHeaders) {
   uint32_t PhdrIdx = 0;
   for (auto &YamlPhdr : Doc.ProgramHeaders) {
     auto &PHeader = PHeaders[PhdrIdx++];
 
     if (YamlPhdr.Sections.size())
       PHeader.p_offset = UINT32_MAX;
bool ELFState<ELFT>::writeSectionContent(Elf_Shdr &SHeader,
   Flags.ases = Section.ASEs;
   Flags.flags1 = Section.Flags1;
   Flags.flags2 = Section.Flags2;
   OS.write((const char *)&Flags, sizeof(Flags));
 
   return true;
 }
 
-template <class ELFT>
-void ELFState<ELFT>::writeSectionContent(Elf_Shdr &SHeader,
-                                         const ELFYAML::DynamicSection &Section,
-                                         ContiguousBlobAccumulator &CBA) {
-  typedef typename ELFT::Addr Elf_Addr;
-  assert(Section.Type == llvm::ELF::SHT_DYNAMIC &&
-         "Section type is not SHT_DYNAMIC");
-
-  SHeader.sh_size = 2 * sizeof(Elf_Addr) * Section.Entries.size();
-  if (Section.EntSize)
-    SHeader.sh_entsize = *Section.EntSize;
-  else
-    SHeader.sh_entsize = sizeof(Elf_Dyn);
-
-  auto &OS = CBA.getOSAndAlignedOffset(SHeader.sh_offset, SHeader.sh_addralign);
-  for (const ELFYAML::DynamicEntry &DE : Section.Entries) {
-    Elf_Addr Tag = (Elf_Addr)DE.Tag;
-    OS.write((const char *)&Tag, sizeof(Elf_Addr));
-    Elf_Addr Val = (Elf_Addr)DE.Val;
-    OS.write((const char *)&Val, sizeof(Elf_Addr));
-  }
-}
-
 template <class ELFT> bool ELFState<ELFT>::buildSectionIndex() {
   for (unsigned i = 0, e = Doc.Sections.size(); i != e; ++i) {
     StringRef Name = Doc.Sections[i]->Name;
     DotShStrtab.add(Name);
     // "+ 1" to take into account the SHT_NULL entry.
     if (SN2I.addName(Name, i + 1)) {
       WithColor::error() << "Repeated section name: '" << Name
                          << "' at YAML section number " << i << ".\n";
if (State.hasDynamicSymbols()) {
     Index = State.SN2I.get(".dynsym");
     State.initSymtabSectionHeader(SHeaders[Index], SymtabType::Dynamic, CBA);
     SHeaders[Index].sh_flags |= ELF::SHF_ALLOC;
     Index = State.SN2I.get(".dynstr");
     State.initStrtabSectionHeader(SHeaders[Index], ".dynstr", State.DotDynstr, CBA);
     SHeaders[Index].sh_flags |= ELF::SHF_ALLOC;
   }
 
+  // Write .dynamic section header and contents.
+  if (State.hasDynamicEntries()) {
+    Elf_Shdr &DynamicHdr = SHeaders[State.DynamicSectionIdx];
+    if (!State.finalizeDynamicSection(DynamicHdr, SHeaders, CBA))
+      return 1;
+  }
+
   // Now we can decide segment offsets
   State.setProgramHeaderLayout(PHeaders, SHeaders);
 
   OS.write((const char *)&Header, sizeof(Header));
   writeArrayData(OS, makeArrayRef(PHeaders));
   writeArrayData(OS, makeArrayRef(SHeaders));
   CBA.writeBlobToStream(OS);
   return 0;
 }
 
 template <class ELFT> bool ELFState<ELFT>::hasDynamicSymbols() const {
   return Doc.DynamicSymbols.Global.size() > 0 ||
          Doc.DynamicSymbols.Weak.size() > 0 ||
          Doc.DynamicSymbols.Local.size() > 0;
 }
 
+template <class ELFT> bool ELFState<ELFT>::hasDynamicEntries() const {
+  return YamlDynamicSection;
+}
+
 template <class ELFT>
 SmallVector<const char *, 5> ELFState<ELFT>::implicitSectionNames() const {
   if (!hasDynamicSymbols())
     return {".symtab", ".strtab", ".shstrtab"};
   return {".symtab", ".strtab", ".shstrtab", ".dynsym", ".dynstr"};
 }
 
 static bool is64Bit(const ELFYAML::Object &Doc) {