Extend obj2yaml to optionally preserve raw __LINKEDIT/__DATA segments.

[aprantl]

Summary

I am planning to upstream MachOObjectFile code to support Darwin
chained fixups. In order to test the new parser features we need a way
to produce correct (and incorrect) chained fixups. Right now the only
tool that can produce them is the Darwin linker. To avoid having to
check in binary files, this patch allows obj2yaml to print a hexdump
of the raw LINKEDIT and DATA segment, which both allows to
bootstrap the parser and enables us to easily create malformed inputs
to test error handling in the parser.

This patch adds two new options to obj2yaml:

-raw-data-segment
-raw-linkedit-segment

[alexander-shaposhnikov]

llvm/lib/Object/MachOObjectFile.cpp

2052

static ?

llvm/tools/obj2yaml/macho2yaml.cpp

654

in the future someone might want to dump other segments too, though I'm not sure what would be the best solution here

[aprantl]

Put function into inline namespace.

llvm/lib/Object/MachOObjectFile.cpp

2052

Good point! My understanding is that a template cannot also be a static function, but I should at least put it into the anonymous namespace.

llvm/tools/obj2yaml/macho2yaml.cpp

654

These two are all I need right now, but if these options proliferate in the future, we should probably pass a list of raw segment names or an option struct.

[alexander-shaposhnikov]

llvm/tools/obj2yaml/macho2yaml.cpp

654

Yeah, I also think that passing a list would be a good option. Somehow it feels like it'd be better than the newly introduced boolean flags - in this case we won't need to change the command line options in the future or keep them around just for compatibility. E.g. one could use --raw-segments=<segment name 1>,<segment name 2> instead of
'-raw-data-segment -raw-linkedit-segment' + the code will improve a little bit - maybe switch now ?

[aprantl]

Generalize command line options to use a bitvector to list all raw segments.

[alexander-shaposhnikov]

thanks, generally looks good to me, i think this feature is very useful.
I'm wondering if we can add tests for this flag ? If I'm not mistaken you had a test in your first revision but it appears to have disappeared in the latest one.

[aprantl]

Indeed. I forgot to git-add the file!

[alexander-shaposhnikov]

llvm/tools/obj2yaml/obj2yaml.cpp

27

I'd like just to point out that in general users can specify their one segment names,
e.g. via attribute((section("CUSTOMSEG,text")), if you use enum here they won't be able to dump them
without modifying the tool
(Phabricator doesn't like double underscore, the name above is <double underscore>CUSTOMSEG)

[alexander-shaposhnikov]

llvm/tools/obj2yaml/obj2yaml.cpp

27

typo: their own

[aprantl]

llvm/tools/obj2yaml/obj2yaml.cpp

27

Yeah, I guess if someone wanted to use such a feature, they will have to generalize this further.

llvm/include/llvm/Object/MachO.h

public:
   bool isSectionCompressed(DataRefImpl Sec) const override;
   bool isSectionText(DataRefImpl Sec) const override;
   bool isSectionData(DataRefImpl Sec) const override;
   bool isSectionBSS(DataRefImpl Sec) const override;
   bool isSectionVirtual(DataRefImpl Sec) const override;
   bool isSectionBitcode(DataRefImpl Sec) const override;
   bool isDebugSection(DataRefImpl Sec) const override;
 
+  /// Return the raw contents of an entire segment.
+  ArrayRef<uint8_t> getSegmentContents(StringRef SegmentName) const;
+
   /// When dsymutil generates the companion file, it strips all unnecessary
   /// sections (e.g. everything in the _TEXT segment) by omitting their body
   /// and setting the offset in their corresponding load command to zero.
   ///
   /// While the load command itself is valid, reading the section corresponds
   /// to reading the number of bytes specified in the load command, starting
   /// from offset 0 (i.e. the Mach-O header at the beginning of the file).
   bool isSectionStripped(DataRefImpl Sec) const override;

llvm/include/llvm/ObjectYAML/MachOYAML.h

 };
 
 struct Object {
   bool IsLittleEndian;
   FileHeader Header;
   std::vector<LoadCommand> LoadCommands;
   std::vector<Section> Sections;
   LinkEditData LinkEdit;
+  Optional<llvm::yaml::BinaryRef> RawLinkEditSegment;
   DWARFYAML::Data DWARF;
 };
 
 struct FatHeader {
   llvm::yaml::Hex32 magic;
   uint32_t nfat_arch;
 };
 

llvm/lib/Object/MachOObjectFile.cpp

bool MachOObjectFile::isDebugSection(DataRefImpl Sec) const {
   }
   StringRef SectionName = SectionNameOrErr.get();
   return SectionName.startswith("__debug") ||
          SectionName.startswith("__zdebug") ||
          SectionName.startswith("__apple") || SectionName == "__gdb_index" ||
          SectionName == "__swift_ast";
 }
 
+namespace {
+template <typename LoadCommandType>
+ArrayRef<uint8_t> getSegmentContents(const MachOObjectFile &Obj,
+                                     MachOObjectFile::LoadCommandInfo LoadCmd,
+                                     StringRef SegmentName) {
+  auto SegmentOrErr = getStructOrErr<LoadCommandType>(Obj, LoadCmd.Ptr);
+  if (!SegmentOrErr) {
+    consumeError(SegmentOrErr.takeError());
+    return {};
+  }
+  auto &Segment = SegmentOrErr.get();
+  if (StringRef(Segment.segname, 16).startswith(SegmentName))
+    return arrayRefFromStringRef(Obj.getData().slice(
+        Segment.fileoff, Segment.fileoff + Segment.filesize));
+  return {};
+}
+} // namespace
+
+ArrayRef<uint8_t>
+MachOObjectFile::getSegmentContents(StringRef SegmentName) const {
+  for (auto LoadCmd : load_commands()) {
+    ArrayRef<uint8_t> Contents;
+    switch (LoadCmd.C.cmd) {
+    case MachO::LC_SEGMENT:
+      Contents = ::getSegmentContents<MachO::segment_command>(*this, LoadCmd,
+                                                              SegmentName);
+      break;
+    case MachO::LC_SEGMENT_64:
+      Contents = ::getSegmentContents<MachO::segment_command_64>(*this, LoadCmd,
+                                                                 SegmentName);
+      break;
+    default:
+      continue;
+    }
+    if (!Contents.empty())
+      return Contents;
+  }
+  return {};
+}
+
 unsigned MachOObjectFile::getSectionID(SectionRef Sec) const {
   return Sec.getRawDataRefImpl().d.a;
 }
 
 bool MachOObjectFile::isSectionVirtual(DataRefImpl Sec) const {
   uint32_t Flags = getSectionFlags(*this, Sec);
   unsigned SectionType = Flags & MachO::SECTION_TYPE;
   return SectionType == MachO::S_ZEROFILL ||

llvm/lib/ObjectYAML/MachOEmitter.cpp

#include "llvm/BinaryFormat/MachO.def"
     auto BytesRemaining = LC.Data.load_command_data.cmdsize - BytesWritten;
     if (BytesRemaining > 0) {
       ZeroFillBytes(OS, BytesRemaining);
     }
   }
 }
 
 Error MachOWriter::writeSectionData(raw_ostream &OS) {
+  uint64_t LinkEditOff = 0;
   for (auto &LC : Obj.LoadCommands) {
     switch (LC.Data.load_command_data.cmd) {
     case MachO::LC_SEGMENT:
     case MachO::LC_SEGMENT_64:
       uint64_t segOff = is64Bit ? LC.Data.segment_command_64_data.fileoff
                                 : LC.Data.segment_command_data.fileoff;
       if (0 ==
           strncmp(&LC.Data.segment_command_data.segname[0], "__LINKEDIT", 16)) {
         FoundLinkEditSeg = true;
+        LinkEditOff = segOff;
+        if (Obj.RawLinkEditSegment)
+          continue;
         writeLinkEditData(OS);
       }
       for (auto &Sec : LC.Sections) {
         ZeroToOffset(OS, Sec.offset);
         // Zero Fill any data between the end of the last thing we wrote and the
         // start of this section.
         if (OS.tell() - fileStart > Sec.offset && Sec.offset != (uint32_t)0)
           return createStringError(
case MachO::LC_SEGMENT_64:
       }
       uint64_t segSize = is64Bit ? LC.Data.segment_command_64_data.filesize
                                  : LC.Data.segment_command_data.filesize;
       ZeroToOffset(OS, segOff + segSize);
       break;
     }
   }
 
+  if (Obj.RawLinkEditSegment) {
+    ZeroToOffset(OS, LinkEditOff);
+    if (OS.tell() - fileStart > LinkEditOff || !LinkEditOff)
+      return createStringError(errc::invalid_argument,
+                               "section offsets don't line up");
+    Obj.RawLinkEditSegment->writeAsBinary(OS);
+  }
   return Error::success();
 }
 
 // The implementation of makeRelocationInfo and makeScatteredRelocationInfo is
 // consistent with how libObject parses MachO binary files. For the reference
 // see getStruct, getRelocation, getPlainRelocationPCRel,
 // getPlainRelocationLength and related methods in MachOObjectFile.cpp
 static MachO::any_relocation_info

llvm/lib/ObjectYAML/MachOYAML.cpp

void MappingTraits<MachOYAML::Object>::mapping(IO &IO,
   IO.mapOptional("IsLittleEndian", Object.IsLittleEndian,
                  sys::IsLittleEndianHost);
   Object.DWARF.IsLittleEndian = Object.IsLittleEndian;
 
   IO.mapRequired("FileHeader", Object.Header);
   Object.DWARF.Is64BitAddrSize = Object.Header.magic == MachO::MH_MAGIC_64 ||
                                  Object.Header.magic == MachO::MH_CIGAM_64;
   IO.mapOptional("LoadCommands", Object.LoadCommands);
+
+  if (Object.RawLinkEditSegment || !IO.outputting())
+    IO.mapOptional("__LINKEDIT", Object.RawLinkEditSegment);
   if(!Object.LinkEdit.isEmpty() || !IO.outputting())
     IO.mapOptional("LinkEditData", Object.LinkEdit);
 
   if(!Object.DWARF.isEmpty() || !IO.outputting())
     IO.mapOptional("DWARF", Object.DWARF);
 
   if (IO.getContext() == &Object)
     IO.setContext(nullptr);

llvm/tools/obj2yaml/macho2yaml.cpp

class MachODumper {
   template <typename StructType>
   Expected<const char *> processLoadCommandData(
       MachOYAML::LoadCommand &LC,
       const llvm::object::MachOObjectFile::LoadCommandInfo &LoadCmd,
       MachOYAML::Object &Y);
 
   const object::MachOObjectFile &Obj;
   std::unique_ptr<DWARFContext> DWARFCtx;
+  unsigned RawSegments;
   void dumpHeader(std::unique_ptr<MachOYAML::Object> &Y);
   Error dumpLoadCommands(std::unique_ptr<MachOYAML::Object> &Y);
   void dumpLinkEdit(std::unique_ptr<MachOYAML::Object> &Y);
   void dumpRebaseOpcodes(std::unique_ptr<MachOYAML::Object> &Y);
   void dumpBindOpcodes(std::vector<MachOYAML::BindOpcode> &BindOpcodes,
                        ArrayRef<uint8_t> OpcodeBuffer, bool Lazy = false);
   void dumpExportTrie(std::unique_ptr<MachOYAML::Object> &Y);
   void dumpSymbols(std::unique_ptr<MachOYAML::Object> &Y);
 
   template <typename SectionType>
   Expected<MachOYAML::Section> constructSectionCommon(SectionType Sec,
                                                       size_t SecIndex);
   template <typename SectionType>
   Expected<MachOYAML::Section> constructSection(SectionType Sec,
                                                 size_t SecIndex);
   template <typename SectionType, typename SegmentType>
   Expected<const char *>
   extractSections(const llvm::object::MachOObjectFile::LoadCommandInfo &LoadCmd,
                   std::vector<MachOYAML::Section> &Sections,
                   MachOYAML::Object &Y);
 
 public:
   MachODumper(const object::MachOObjectFile &O,
-              std::unique_ptr<DWARFContext> DCtx)
-      : Obj(O), DWARFCtx(std::move(DCtx)) {}
+              std::unique_ptr<DWARFContext> DCtx, unsigned RawSegments)
+      : Obj(O), DWARFCtx(std::move(DCtx)), RawSegments(RawSegments) {}
   Expected<std::unique_ptr<MachOYAML::Object>> dump();
 };
 
 #define HANDLE_LOAD_COMMAND(LCName, LCValue, LCStruct)                         \
   case MachO::LCName:                                                          \
     memcpy((void *)&(LC.Data.LCStruct##_data), LoadCmd.Ptr,                    \
            sizeof(MachO::LCStruct));                                           \
     if (Obj.isLittleEndian() != sys::IsLittleEndianHost)                       \
for (; reinterpret_cast<const void *>(Curr) < End; Curr++) {
     SectionType Sec;
     memcpy((void *)&Sec, Curr, sizeof(SectionType));
     if (Obj.isLittleEndian() != sys::IsLittleEndianHost)
       MachO::swapStruct(Sec);
     // For MachO section indices start from 1.
     if (Expected<MachOYAML::Section> S =
             constructSection(Sec, Sections.size() + 1)) {
       StringRef SecName(S->sectname);
+
+      // Copy data sections if requested.
+      if ((RawSegments & RawSegments::data) &&
+          StringRef(S->segname).startswith("__DATA"))
+        S->content =
+            yaml::BinaryRef(Obj.getSectionContents(Sec.offset, Sec.size));
+
       if (SecName.startswith("__debug_")) {
         // If the DWARF section cannot be successfully parsed, emit raw content
         // instead of an entry in the DWARF section of the YAML.
         if (Error Err = dumpDebugSection(SecName, *DWARFCtx.get(), Y.DWARF))
           consumeError(std::move(Err));
         else
           S->content.reset();
       }

 }
 
 Expected<std::unique_ptr<MachOYAML::Object>> MachODumper::dump() {
   auto Y = std::make_unique<MachOYAML::Object>();
   Y->IsLittleEndian = Obj.isLittleEndian();
   dumpHeader(Y);
   if (Error Err = dumpLoadCommands(Y))
     return std::move(Err);
+  if (RawSegments & RawSegments::linkedit)
+    Y->RawLinkEditSegment =
+        yaml::BinaryRef(Obj.getSegmentContents("__LINKEDIT"));
+  else
     dumpLinkEdit(Y);
 
   return std::move(Y);
 }
 
 void MachODumper::dumpHeader(std::unique_ptr<MachOYAML::Object> &Y) {
   Y->Header.magic = Obj.getHeader().magic;
   Y->Header.cputype = Obj.getHeader().cputype;
   Y->Header.cpusubtype = Obj.getHeader().cpusubtype;
void MachODumper::dumpSymbols(std::unique_ptr<MachOYAML::Object> &Y) {
   StringRef RemainingTable = Obj.getStringTableData();
   while (RemainingTable.size() > 0) {
     auto SymbolPair = RemainingTable.split('\0');
     RemainingTable = SymbolPair.second;
     LEData.StringTable.push_back(SymbolPair.first);
   }
 }
 
-Error macho2yaml(raw_ostream &Out, const object::MachOObjectFile &Obj) {
+Error macho2yaml(raw_ostream &Out, const object::MachOObjectFile &Obj,
+                 unsigned RawSegments) {
   std::unique_ptr<DWARFContext> DCtx = DWARFContext::create(Obj);
-  MachODumper Dumper(Obj, std::move(DCtx));
+  MachODumper Dumper(Obj, std::move(DCtx), RawSegments);
   Expected<std::unique_ptr<MachOYAML::Object>> YAML = Dumper.dump();
   if (!YAML)
     return YAML.takeError();
 
   yaml::YamlObjectFile YAMLFile;
   YAMLFile.MachO = std::move(YAML.get());
 
   yaml::Output Yout(Out);
   Yout << YAMLFile;
   return Error::success();
 }
 
-Error macho2yaml(raw_ostream &Out, const object::MachOUniversalBinary &Obj) {
+Error macho2yaml(raw_ostream &Out, const object::MachOUniversalBinary &Obj,
+                 unsigned RawSegments) {
   yaml::YamlObjectFile YAMLFile;
   YAMLFile.FatMachO.reset(new MachOYAML::UniversalBinary());
   MachOYAML::UniversalBinary &YAML = *YAMLFile.FatMachO;
   YAML.Header.magic = Obj.getMagic();
   YAML.Header.nfat_arch = Obj.getNumberOfObjects();
 
   for (auto Slice : Obj.objects()) {
     MachOYAML::FatArch arch;
     arch.cputype = Slice.getCPUType();
     arch.cpusubtype = Slice.getCPUSubType();
     arch.offset = Slice.getOffset();
     arch.size = Slice.getSize();
     arch.align = Slice.getAlign();
     arch.reserved = Slice.getReserved();
     YAML.FatArchs.push_back(arch);
 
     auto SliceObj = Slice.getAsObjectFile();
     if (!SliceObj)
       return SliceObj.takeError();
 
     std::unique_ptr<DWARFContext> DCtx = DWARFContext::create(*SliceObj.get());
-    MachODumper Dumper(*SliceObj.get(), std::move(DCtx));
+    MachODumper Dumper(*SliceObj.get(), std::move(DCtx), RawSegments);
     Expected<std::unique_ptr<MachOYAML::Object>> YAMLObj = Dumper.dump();
     if (!YAMLObj)
       return YAMLObj.takeError();
     YAML.Slices.push_back(*YAMLObj.get());
   }
 
   yaml::Output Yout(Out);
   Yout << YAML;
   return Error::success();
 }
 
-Error macho2yaml(raw_ostream &Out, const object::Binary &Binary) {
+Error macho2yaml(raw_ostream &Out, const object::Binary &Binary,
+                 unsigned RawSegments) {
   if (const auto *MachOObj = dyn_cast<object::MachOUniversalBinary>(&Binary))
-    return macho2yaml(Out, *MachOObj);
+    return macho2yaml(Out, *MachOObj, RawSegments);
 
   if (const auto *MachOObj = dyn_cast<object::MachOObjectFile>(&Binary))
-    return macho2yaml(Out, *MachOObj);
+    return macho2yaml(Out, *MachOObj, RawSegments);
 
   llvm_unreachable("unexpected Mach-O file format");
 }

llvm/tools/obj2yaml/obj2yaml.h

 #include "llvm/Object/COFF.h"
 #include "llvm/Object/Minidump.h"
 #include "llvm/Object/Wasm.h"
 #include "llvm/Object/XCOFFObjectFile.h"
 #include "llvm/Support/raw_ostream.h"
 #include "llvm/Support/MemoryBufferRef.h"
 #include <system_error>
 
+enum RawSegments : unsigned { none = 0, data = 1, linkedit = 1 << 1 };
 std::error_code coff2yaml(llvm::raw_ostream &Out,
                           const llvm::object::COFFObjectFile &Obj);
 llvm::Error elf2yaml(llvm::raw_ostream &Out,
                          const llvm::object::ObjectFile &Obj);
-llvm::Error macho2yaml(llvm::raw_ostream &Out,
-                           const llvm::object::Binary &Obj);
+llvm::Error macho2yaml(llvm::raw_ostream &Out, const llvm::object::Binary &Obj,
+                       unsigned RawSegments);
 llvm::Error minidump2yaml(llvm::raw_ostream &Out,
                           const llvm::object::MinidumpFile &Obj);
 llvm::Error xcoff2yaml(llvm::raw_ostream &Out,
                        const llvm::object::XCOFFObjectFile &Obj);
 std::error_code wasm2yaml(llvm::raw_ostream &Out,
                           const llvm::object::WasmObjectFile &Obj);
 llvm::Error archive2yaml(llvm::raw_ostream &Out, llvm::MemoryBufferRef Source);
 

llvm/tools/obj2yaml/obj2yaml.cpp

-//===------ utils/obj2yaml.cpp - obj2yaml conversion tool -------*- C++ -*-===//
+//===------ utils/obj2yaml.cpp - obj2yaml conversion tool -----------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "obj2yaml.h"
 #include "llvm/BinaryFormat/Magic.h"
 #include "llvm/Object/Archive.h"
 #include "llvm/Object/COFF.h"
 #include "llvm/Object/Minidump.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Errc.h"
 #include "llvm/Support/InitLLVM.h"
 
 using namespace llvm;
 using namespace llvm::object;
 
+static cl::opt<std::string>
+    InputFilename(cl::Positional, cl::desc("<input file>"), cl::init("-"));
+static cl::bits<RawSegments> RawSegment(
+    "raw-segment",
+    cl::desc("Mach-O: dump the raw contents of the listed segments instead of "
+             "parsing them:"),
+    cl::values(clEnumVal(data, "__DATA"), clEnumVal(linkedit, "__LINKEDIT")));
+
 static Error dumpObject(const ObjectFile &Obj) {
   if (Obj.isCOFF())
     return errorCodeToError(coff2yaml(outs(), cast<COFFObjectFile>(Obj)));
 
   if (Obj.isXCOFF())
     return xcoff2yaml(outs(), cast<XCOFFObjectFile>(Obj));
 
   if (Obj.isELF())
Expected<std::unique_ptr<Binary>> BinOrErr =
       createBinary(MemBuf, /*Context=*/nullptr);
   if (!BinOrErr)
     return BinOrErr.takeError();
 
   Binary &Binary = *BinOrErr->get();
   // Universal MachO is not a subclass of ObjectFile, so it needs to be handled
   // here with the other binary types.
   if (Binary.isMachO() || Binary.isMachOUniversalBinary())
-    return macho2yaml(outs(), Binary);
+    return macho2yaml(outs(), Binary, RawSegment.getBits());
   if (ObjectFile *Obj = dyn_cast<ObjectFile>(&Binary))
     return dumpObject(*Obj);
   if (MinidumpFile *Minidump = dyn_cast<MinidumpFile>(&Binary))
     return minidump2yaml(outs(), *Minidump);
 
   return Error::success();
 }
 
 static void reportError(StringRef Input, Error Err) {
   if (Input == "-")
     Input = "<stdin>";
   std::string ErrMsg;
   raw_string_ostream OS(ErrMsg);
   logAllUnhandledErrors(std::move(Err), OS);
   OS.flush();
   errs() << "Error reading file: " << Input << ": " << ErrMsg;
   errs().flush();
 }
 
-cl::opt<std::string> InputFilename(cl::Positional, cl::desc("<input file>"),
-                                   cl::init("-"));
-
 int main(int argc, char *argv[]) {
   InitLLVM X(argc, argv);
   cl::ParseCommandLineOptions(argc, argv);
 
   if (Error Err = dumpInput(InputFilename)) {
     reportError(InputFilename, std::move(Err));
     return 1;
   }
 
   return 0;
 }