[Object] Create MutableObject class for doing Object Mutations [Part 2]

[abrachet]

Summary

Create section sub classes for ELF

[jhenderson]

It looks like in this patch, you haven't uploaded the diff versus D64281, it's versus trunk. As previously explained, I think it's important you do the diff versus the previous patch, so that it's obvious what this adds over the previous one.

I also don't think this is the right next step, or at least it's too much for one patch. I think a better thing to do would be to add only the bare minimum number of sections you need to be able to remove a section from a very basic ELF (e.g. one with just a .data section in with no symbols). In this case, you probably just need a string table section, so that you can update it's contents following the removal of sections. That would allow you to write unit tests that test some useful functionality. Subsequent patches can add proper handling of other section types, so that the ELF is not invalidated following removal of certain sections, but in the first instance, you don't need to handle everything.

[MaskRay]

If you have branches a, b, and c (a depends on master, b depends on a, c depends on b):

# after syncing master, a, b and c became stale.
git rebase --onto master 'a^' a
  arc diff 'HEAD^'  # create/update a revision based on the diff master..a
git rebase --onto a 'b^' b  # this also switches to b
  arc diff 'HEAD^'  # create/update a revision based on the diff a..b     master..a is not included in the revision.
git rebase --onto b 'c^' c  # this also switches to b
  arc diff 'HEAD^'  # create/update a revision based on the diff b..c

Perhaps there is a better way to do things but this is how I handle dependent revisions.

In D64281, I still want to see how you are going to use MutableRange. Do you have a design what the end result will be?

[abrachet]

In D64968#1595107, @MaskRay wrote:

If you have branches a, b, and c (a depends on master, b depends on a, c depends on b):

# after syncing master, a, b and c became stale.
git rebase --onto master 'a^' a
  arc diff 'HEAD^'  # create/update a revision based on the diff master..a
git rebase --onto a 'b^' b  # this also switches to b
  arc diff 'HEAD^'  # create/update a revision based on the diff a..b     master..a is not included in the revision.
git rebase --onto b 'c^' c  # this also switches to b
  arc diff 'HEAD^'  # create/update a revision based on the diff b..c

Perhaps there is a better way to do things but this is how I handle dependent revisions.

In D64281, I still want to see how you are going to use MutableRange. Do you have a design what the end result will be?

Thanks @MaskRay, thats very helpful I was having a hard time managing this.

I ended up just abandoning this revision for now because D64281 has changed so drastically and I wasn't ready for a part 2 to that yet so I didn't want this one to just be dangling here. That revision still uses a data structure called MutableRange, but it has changed a lot. I'll work on trying to improve the performance of batch operations on it in a bit. But I've drastically changed my whole project so I have put that off for now.

llvm/include/llvm/Object/MutableELFObject.h

+//===-- MutableELFObject.h --------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file declares the MutableELFObject template class.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_OBJECT_MUTABLEELFOBJECT_H
+#define LLVM_OBJECT_MUTABLEELFOBJECT_H
+
+#include "ELFObjectFile.h"
+#include "ELFTypes.h"
+#include "MutableObject.h"
+#include "llvm/MC/StringTableBuilder.h"
+
+namespace llvm {
+namespace object {
+namespace elf {
+
+template <typename ELFT> class MutableELFObject;
+template <typename ELFT> class MutableSection;
+template <typename ELFT> class MutableStrTabSection;
+template <typename ELFT> class MutableSectionIndexSection;
+template <typename ELFT> class MutableSymbolTableSection;
+template <typename ELFT> class MutableRelocationSection;
+template <typename ELFT> class MutableDynRelocationSection;
+template <typename ELFT> class MutableGroupSection;
+
+template <typename ELFT>
+class MutableSection : public MutableSectionBase, public Elf_Shdr_Impl<ELFT> {
+
+public:
+  MutableSection(const MutableSection &) = default;
+  MutableSection(std::string Name, ArrayRef<const uint8_t> Data,
+                 Elf_Shdr_Impl<ELFT> SecHeader)
+      : MutableSectionBase(Name, Data), Elf_Shdr_Impl<ELFT>(SecHeader) {}
+  ~MutableSection() override {}
+
+  static bool classof(const MutableSectionBase *Sec) {
+    return isa<MutableELFObject<ELFT>>(Sec->OwningObject);
+  }
+
+  virtual std::unique_ptr<MutableSectionBase> clone() const override {
+    return make_unique<MutableSection<ELFT>>(*this);
+  }
+};
+
+template <typename ELFT>
+class MutableStrTabSection : public MutableSection<ELFT> {
+
+public:
+  MutableStrTabSection(const MutableStrTabSection &) = default;
+  MutableStrTabSection(std::string Name, ArrayRef<const uint8_t> Data,
+                       Elf_Shdr_Impl<ELFT> SecHeader)
+      : MutableSection<ELFT>(Name, Data, SecHeader) {
+    assert(SecHeader.sh_type == ELF::SHT_STRTAB && "Expected .strtab section");
+  }
+  ~MutableStrTabSection() override {}
+
+  StringRef getName(uint64_t Index) const { return this->Data.data() + Index; }
+
+  uint64_t findIndex(StringRef Name) const;
+
+  static bool classof(const MutableSection<ELFT> *S) {
+    return S->sh_type == ELF::SHT_STRTAB;
+  }
+
+  std::unique_ptr<MutableSectionBase> clone() const override {
+    // return std::unique_ptr<MutableSectionBase>(new
+    // MutableStrTabSection<ELFT>(*this));
+    return make_unique<MutableStrTabSection<ELFT>>(*this);
+  }
+};
+
+template <typename ELFT>
+class MutableSectionIndexSection : public MutableSection<ELFT> {
+
+public:
+  MutableSectionIndexSection(const MutableSectionIndexSection &) = default;
+  MutableSectionIndexSection(std::string Name, ArrayRef<const uint8_t> Data,
+                             Elf_Shdr_Impl<ELFT> SecHeader)
+      : MutableSection<ELFT>(Name, Data, SecHeader) {
+    assert(SecHeader.sh_type == ELF::SHT_SYMTAB_SHNDX && "Incorrect sh_type");
+  }
+  ~MutableSectionIndexSection() override {}
+
+  static bool classof(const MutableSection<ELFT> *S) {
+    return S->sh_type == ELF::SHT_SYMTAB_SHNDX;
+  }
+
+  std::unique_ptr<MutableSectionBase> clone() const override {
+    return make_unique<MutableSectionIndexSection<ELFT>>(*this);
+  }
+};
+
+template <typename ELFT>
+class MutableSymbolTableSection : public MutableSection<ELFT> {
+
+public:
+  MutableSymbolTableSection(const MutableSymbolTableSection &) = default;
+  MutableSymbolTableSection(std::string Name, ArrayRef<const uint8_t> Data,
+                            Elf_Shdr_Impl<ELFT> SecHeader)
+      : MutableSection<ELFT>(Name, Data, SecHeader) {}
+  ~MutableSymbolTableSection() override {}
+
+  static bool classof(const MutableSection<ELFT> *S) {
+    return S->sh_type == ELF::SHT_SYMTAB;
+  }
+
+  std::unique_ptr<MutableSectionBase> clone() const override {
+    return make_unique<MutableSymbolTableSection<ELFT>>(*this);
+  }
+};
+
+template <typename ELFT>
+class MutableRelocationSection : public MutableSection<ELFT> {
+
+public:
+  MutableRelocationSection(const MutableRelocationSection &) = default;
+  MutableRelocationSection(std::string Name, ArrayRef<const uint8_t> Data,
+                           Elf_Shdr_Impl<ELFT> SecHeader)
+      : MutableSection<ELFT>(Name, Data, SecHeader) {
+    assert(!(SecHeader.sh_flags & ELF::SHF_ALLOC) &&
+           "Relocation section shouldn't be allocated");
+    assert(SecHeader.sh_type == ELF::SHT_REL ||
+           SecHeader.sh_type == ELF::SHT_RELA && "Incorrect sh_type");
+  }
+  ~MutableRelocationSection() override {}
+
+  static bool classof(const MutableSection<ELFT> *S) {
+    if (S->Flags & ELF::SHF_ALLOC)
+      return false;
+    return S->sh_type == ELF::SHT_REL || S->sh_type == ELF::SHT_RELA;
+  }
+
+  std::unique_ptr<MutableSectionBase> clone() const override {
+    return make_unique<MutableRelocationSection<ELFT>>(*this);
+  }
+};
+
+template <typename ELFT>
+class MutableDynRelocationSection : public MutableSection<ELFT> {
+
+public:
+  MutableDynRelocationSection(const MutableDynRelocationSection &) = default;
+  MutableDynRelocationSection(std::string Name, ArrayRef<const uint8_t> Data,
+                              Elf_Shdr_Impl<ELFT> SecHeader)
+      : MutableSection<ELFT>(Name, Data, SecHeader) {
+    assert(SecHeader.sh_flags & ELF::SHF_ALLOC &&
+           "Dynamic relocation section should be allocated");
+    assert(SecHeader.sh_type == ELF::SHT_REL ||
+           SecHeader.sh_type == ELF::SHT_RELA && "Incorrect sh_type");
+  }
+  ~MutableDynRelocationSection() override {}
+
+  static bool classof(const MutableSection<ELFT> *S) {
+    if (!(S->Flags & ELF::SHF_ALLOC))
+      return false;
+    return S->sh_type == ELF::SHT_REL || S->sh_type == ELF::SHT_RELA;
+  }
+
+  std::unique_ptr<MutableSectionBase> clone() const override {
+    return make_unique<MutableDynRelocationSection<ELFT>>(*this);
+  }
+};
+
+template <typename ELFT>
+class MutableGroupSection : public MutableSection<ELFT> {
+
+public:
+  MutableGroupSection(const MutableGroupSection &) = default;
+  MutableGroupSection(std::string Name, ArrayRef<const uint8_t> Data,
+                      Elf_Shdr_Impl<ELFT> SecHeader)
+      : MutableSection<ELFT>(Name, Data, SecHeader) {
+    assert(SecHeader.sh_type == ELF::SHT_GROUP && "Incorrect sh_type");
+  }
+  ~MutableGroupSection() override {}
+
+  static bool classof(const MutableSection<ELFT> *S) {
+    return S->sh_type == ELF::SHT_GROUP;
+  }
+
+  std::unique_ptr<MutableSectionBase> clone() const override {
+    return make_unique<MutableGroupSection<ELFT>>(*this);
+  }
+};
+
+template <typename ELFT>
+class MutableSegment : public MutableSegmentBase, public Elf_Phdr_Impl<ELFT> {
+
+public:
+  MutableSegment(const MutableSegment &) = default;
+  MutableSegment(ArrayRef<const uint8_t> Data, Elf_Phdr_Impl<ELFT> ProgHeader)
+      : MutableSectionBase(Data), Elf_Phdr_Impl<ELFT>(ProgHeader) {}
+  ~MutableSegment() override {}
+
+  static bool classof(const MutableSectionBase *Sec) {
+    return isa<MutableELFObject<ELFT>>(Sec->OwningObject);
+  }
+
+  std::unique_ptr<MutableSegmentBase> clone() const override {
+    return make_unique<MutableSegment<ELFT>>(*this);
+  }
+};
+
+template <typename ELFT> class MutableELFObject : public MutableObject {
+  Elf_Ehdr_Impl<ELFT> Header;
+
+public:
+  MutableELFObject(const ObjectFile &ObjFile,
+                   SectionList::VecType OriginalSections,
+                   SegmentList::VecType OriginalSegments,
+                   SymbolList::VecType OriginalSymbols,
+                   Elf_Ehdr_Impl<ELFT> Header)
+      : MutableObject(ObjFile, std::move(OriginalSections),
+                      std::move(OriginalSegments), std::move(OriginalSymbols)),
+        Header(Header) {}
+
+public:
+  static Expected<std::unique_ptr<MutableObject>>
+  createMutableObject(const ELFObjectFile<ELFT> &ObjFile);
+
+  const ELFObjectFile<ELFT> *getObjectFile() const { return &ObjFile; }
+};
+
+template <typename ELFT>
+static std::unique_ptr<MutableSection<ELFT>>
+makeSection(StringRef Name, ArrayRef<const uint8_t> Data,
+            Elf_Shdr_Impl<ELFT> SecHeader) {
+  switch (SecHeader.sh_type) {
+  case ELF::SHT_STRTAB:
+    return make_unique<MutableStrTabSection<ELFT>>(Name, Data, SecHeader);
+  case ELF::SHT_SYMTAB_SHNDX:
+    return make_unique<MutableSectionIndexSection<ELFT>>(Name, Data, SecHeader);
+  case ELF::SHT_SYMTAB:
+    return make_unique<MutableSymbolTableSection<ELFT>>(Name, Data, SecHeader);
+  case ELF::SHT_REL:
+  case ELF::SHT_RELA:
+    if (SecHeader.sh_flags & ELF::SHF_ALLOC)
+      return make_unique<MutableDynRelocationSection<ELFT>>(Name, Data,
+                                                            SecHeader);
+    else
+      return make_unique<MutableRelocationSection<ELFT>>(Name, Data, SecHeader);
+  case ELF::SHT_GROUP:
+    return make_unique<MutableGroupSection<ELFT>>(Name, Data, SecHeader);
+  }
+  return make_unique<MutableSection<ELFT>>(Name, Data, SecHeader);
+}
+
+template <typename ELFT>
+Expected<std::unique_ptr<MutableObject>>
+MutableELFObject<ELFT>::createMutableObject(
+    const ELFObjectFile<ELFT> &ObjFile) {
+  auto &Header = *ObjFile.getELFFile()->getHeader();
+  const uint8_t *Base = reinterpret_cast<const uint8_t *>(
+      ObjFile.getMemoryBufferRef().getBufferStart());
+
+  ArrayRef<const Elf_Shdr_Impl<ELFT>> SectionHeaders(
+      reinterpret_cast<const Elf_Shdr_Impl<ELFT> *>(Base + Header.e_shoff),
+      Header.e_shnum);
+
+  const uint8_t *Shstrndx = nullptr;
+  if (Header.e_shstrndx) {
+    size_t Index = Header.e_shstrndx == ELF::SHN_XINDEX
+                       ? SectionHeaders[0].sh_size
+                       : Header.e_shstrndx;
+    Shstrndx = Base + SectionHeaders[Index].sh_offset;
+  }
+  auto SectionName = [Shstrndx](size_t StrOffset) -> StringRef {
+    if (!Shstrndx)
+      return StringRef();
+    return StringRef(reinterpret_cast<const char *>(Shstrndx) + StrOffset);
+  };
+
+  SectionList::VecType OriginalSections;
+  for (const auto &SecHeader : SectionHeaders)
+    OriginalSections.push_back(makeSection(
+        SectionName(SecHeader.sh_name),
+        ArrayRef<const uint8_t>(Base + SecHeader.sh_offset, SecHeader.sh_size),
+        SecHeader));
+
+  SegmentList::VecType OriginalSegments;
+
+  return std::unique_ptr<MutableObject>(new MutableELFObject<ELFT>(
+      ObjFile, std::move(OriginalSections), SegmentList::VecType(),
+      SymbolList::VecType(), Header));
+}
+
+} // namespace elf
+} // namespace object
+} // namespace llvm
+
+#endif // LLVM_OBJECT_MUTABLEELFOBJECT_H

llvm/include/llvm/Object/MutableObject.h

+//===-- MutableObject.h -----------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Class for doing mutations on object files.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_OBJECT_MUTABLEOBJECT_H
+#define LLVM_OBJECT_MUTABLEOBJECT_H
+
+#include "ObjectFile.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/MutableRange.h"
+#include <string>
+
+namespace llvm {
+namespace object {
+
+struct MutableSectionBase;
+struct MutableSegmentBase;
+struct MutableSymbolBase;
+class MutableObject;
+
+struct MutableSectionBase {
+  const MutableObject *OwningObject = nullptr;
+  MutableSegmentBase *OwnedSegment = nullptr;
+
+  // Name is mutable because the original name can be recovered using
+  // sh_name or similar. This makes it more effecient add a prefix to
+  // all sections for example.
+  mutable std::string Name;
+  ArrayRef<const uint8_t> Data;
+
+  MutableSectionBase(std::string Name, ArrayRef<const uint8_t> Data)
+      : Name(Name), Data(Data) {}
+  virtual ~MutableSectionBase() {}
+
+  virtual std::unique_ptr<MutableSectionBase> clone() const {
+    return make_unique<MutableSectionBase>(*this);
+  }
+};
+
+struct MutableSegmentBase {
+  const MutableObject *OwningObject = nullptr;
+  MutableSegmentBase *OwnedSegment = nullptr;
+  ArrayRef<const uint8_t> Data;
+
+  MutableSegmentBase(ArrayRef<const uint8_t> Data) : Data(Data) {}
+  virtual ~MutableSegmentBase() {}
+  virtual std::unique_ptr<MutableSegmentBase> clone() const;
+};
+
+struct MutableSymbolBase {
+  const MutableObject *OwningObject = nullptr;
+  MutableSegmentBase *OwnedSegment = nullptr;
+  mutable std::string Name;
+
+  MutableSymbolBase(const MutableObject &OwningObject, std::string Name)
+      : Name(Name) {}
+  virtual ~MutableSymbolBase() {}
+  virtual std::unique_ptr<MutableSymbolBase> clone() const;
+};
+
+// Custom cloner for MutableRange because MutableXXXBase classes
+// are non trivially copyable.
+template <typename T> struct MutableEntryCloner {
+  T operator()(const T *Other) { return (*Other)->clone(); }
+};
+
+// MutableRange doesn't take ownership of the underlying container
+// this is a small wrapper that does.
+template <typename T>
+struct OwningMutableRange
+    : public MutableRange<std::vector<T>, MutableEntryCloner<T>> {
+  using VecType = std::vector<T>;
+  using Base = MutableRange<VecType, MutableEntryCloner<T>>;
+
+  OwningMutableRange(VecType Vec) : Base(Vec), OwnedVec(std::move(Vec)) {}
+
+private:
+  const VecType OwnedVec;
+};
+
+class MutableObject {
+protected:
+  const ObjectFile &ObjFile;
+
+  using SectionList = OwningMutableRange<std::unique_ptr<MutableSectionBase>>;
+  using SegmentList = OwningMutableRange<std::unique_ptr<MutableSegmentBase>>;
+  using SymbolList = OwningMutableRange<std::unique_ptr<MutableSymbolBase>>;
+  SectionList Sections;
+  SegmentList Segments;
+  SymbolList Symbols;
+
+  MutableObject(const ObjectFile &ObjFile,
+                SectionList::VecType OriginalSections,
+                SegmentList::VecType OriginalSegments,
+                SymbolList::VecType OriginalSymbols);
+
+public:
+  static Expected<std::unique_ptr<MutableObject>>
+  createMutableObject(const ObjectFile &ObjFile);
+
+  const ObjectFile *getObjectFile() const { return &ObjFile; }
+
+  SectionList &sections() { return Sections; }
+  SegmentList &segments() { return Segments; }
+  SymbolList &symbols() { return Symbols; }
+};
+
+} // namespace object
+} // namespace llvm
+
+#endif // LLVM_OBJECT_MUTABLEOBJECT_H

llvm/lib/Object/CMakeLists.txt

add_llvm_library(LLVMObject
   ELFObjectFile.cpp
   Error.cpp
   IRObjectFile.cpp
   IRSymtab.cpp
   MachOObjectFile.cpp
   MachOUniversal.cpp
   Minidump.cpp
   ModuleSymbolTable.cpp
+  MutableObject.cpp
   Object.cpp
   ObjectFile.cpp
   RecordStreamer.cpp
   RelocationResolver.cpp
   SymbolicFile.cpp
   SymbolSize.cpp
   WasmObjectFile.cpp
   WindowsMachineFlag.cpp

llvm/lib/Object/MutableObject.cpp

+//===- MutableObject.cpp - An object file which can be mutated ------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Object/MutableObject.h"
+
+using namespace llvm;
+using namespace object;
+
+MutableObject::MutableObject(const ObjectFile &ObjFile,
+                             SectionList::VecType OriginalSections,
+                             SegmentList::VecType OriginalSegments,
+                             SymbolList::VecType OriginalSymbols)
+    : ObjFile(ObjFile), Sections(std::move(OriginalSections)),
+      Segments(std::move(OriginalSegments)),
+      Symbols(std::move(OriginalSymbols)) {}
+
+Expected<std::unique_ptr<MutableObject>>
+MutableObject::createMutableObject(const ObjectFile &ObjFile) {
+  // Hacky way to expose the constructor publicly for make_unique.
+  struct PublicCtor : public MutableObject {
+    PublicCtor(const ObjectFile &ObjFile, SectionList::VecType Secs,
+               SegmentList::VecType Segs, SymbolList::VecType Syms)
+        : MutableObject(ObjFile, std::move(Secs), std::move(Segs),
+                        std::move(Syms)) {}
+  };
+
+  return llvm::make_unique<PublicCtor>(ObjFile, SectionList::VecType(),
+                                       SegmentList::VecType(),
+                                       SymbolList::VecType());
+}