[lld-macho] Handle command-line option -sectcreate SEG SECT FILE

[gkm]

Summary

Handle command-line option -sectcreate SEG SECT FILE, which inputs a binary blob from FILE into SEG,SECT

[int3]

lld/MachO/InputFiles.cpp

313

none of the other InputFile constructors modify inputSections directly; I think it would better fit the pattern if OpaqueFile instead inserted the isec in its subsections map (at an offset of zero)

lld/test/MachO/sectcreate.s

5

would be interesting to test multiple -sectcreate flags in sequence, e.g. checking that we lay out their sections in sequence as well. Also, does ld64 interleave them amongst the other object files on the command line, or do the created sections all appear at the end? Would be good to match that behavior

9–10

FileCheck does a substring match, so you could drop the hex bytes and just match against the ASCII

[gkm]

lld/MachO/InputFiles.cpp

313

Quite right. Fixed.

lld/test/MachO/sectcreate.s

5

ld64 does not interleave. It places them all at the end, after the input files. I augmented the test case.

[gkm]

Amend according to review feedback

[gkm]

Shuffle option sequence to demonstrate merging of discontiguous -sectcreate sections

[int3]

lgtm modulo those changes

lld/MachO/InputFiles.cpp

313

nit: subsections.emplace_back({0, isec});

lld/test/MachO/sectcreate.s

5

Thanks for checking! I think it's worthwhile for us to codify this behavior -- we could put %t.o in between the sectcreate flags below to demonstrate that interleaving doesn't occur.

12

let's check that __text appears before the additional sections

[gkm]

lld/MachO/InputFiles.cpp

313

I followed prior practice in InputFile::parseSections. If I follow your recommendation, it seems I should also fix InputFile::parseSections, but then I am straying from the purpose of this diff into cleanup territory. Perhaps we should rather save cleanups for a separate diff, which could also include explicit const auto.

[int3]

lld/MachO/InputFiles.cpp

313

I think it's worthwhile to make new code conform to the linter. That way, even if we don't explicitly do cleanups, we will still organically move to the new convention in the course of normal code churn from development. But I'm fine with migrating to emplace in a separate diff.

[gkm]

lld/MachO/InputFiles.cpp

313

I gave-up on subsections.emplace_back(). None of these compiled:

subsections.emplace_back({{0, isec}});
subsections.emplace_back({0, isec});
subsections.emplace_back(0, isec);

subsections is a vector of maps, and the emplacement incantation is beyond me. Clues welcome. Here are the relevant decls:

using SubsectionMap = std::map<uint32_t, InputSection *>;
class InputFile {
  std::vector<SubsectionMap> subsections;
};

[gkm]

test case: move object file to end of arg list + add a .data section in order to prove that -sectcreate sections are place after sections from object files

[int3]

lld/MachO/InputFiles.cpp

313

hm I'm not 100% familiar with what the C++ spec says about nested initialization lists... that not compiling is probably the reason why push_back was used elsewhere

lld/MachO/Driver.cpp

for (const auto &arg : args) {
     case OPT_e:
     case OPT_F:
     case OPT_L:
     case OPT_headerpad:
     case OPT_install_name:
     case OPT_Z:
     case OPT_arch:
     case OPT_syslibroot:
+    case OPT_sectcreate:
       // handled elsewhere
       break;
     default:
       warnIfUnimplementedOption(opt);
       break;
     }
   }
 
bool macho::link(llvm::ArrayRef<const char *> argsArr, bool canExitEarly,
   if (config->outputType == MH_EXECUTE && !isa<Defined>(config->entry)) {
     error("undefined symbol: " + config->entry->getName());
     return false;
   }
 
   createSyntheticSections();
   symtab->addDSOHandle(in.header);
 
+  for (opt::Arg *arg : args.filtered(OPT_sectcreate)) {
+    StringRef segName = arg->getValue(0);
+    StringRef sectName = arg->getValue(1);
+    StringRef fileName = arg->getValue(2);
+    Optional<MemoryBufferRef> buffer = readFile(fileName);
+    if (buffer)
+      inputFiles.push_back(make<OpaqueFile>(*buffer, segName, sectName));
+  }
+
   // Initialize InputSections.
   for (InputFile *file : inputFiles) {
     for (SubsectionMap &map : file->subsections) {
       for (auto &p : map) {
         InputSection *isec = p.second;
         inputSections.push_back(isec);
       }
     }

lld/MachO/InputFiles.h

 using SubsectionMap = std::map<uint32_t, InputSection *>;
 
 class InputFile {
 public:
   enum Kind {
     ObjKind,
     DylibKind,
     ArchiveKind,
+    OpaqueKind,
   };
 
   virtual ~InputFile() = default;
   Kind kind() const { return fileKind; }
   StringRef getName() const { return mb.getBufferIdentifier(); }
 
   MemoryBufferRef mb;
   std::vector<Symbol *> symbols;

 
 // .o file
 class ObjFile : public InputFile {
 public:
   explicit ObjFile(MemoryBufferRef mb);
   static bool classof(const InputFile *f) { return f->kind() == ObjKind; }
 };
 
+// command-line -sectcreate file
+class OpaqueFile : public InputFile {
+public:
+  explicit OpaqueFile(MemoryBufferRef mb, StringRef segName,
+                      StringRef sectName);
+  static bool classof(const InputFile *f) { return f->kind() == OpaqueKind; }
+};
+
 // .dylib file
 class DylibFile : public InputFile {
 public:
   explicit DylibFile(std::shared_ptr<llvm::MachO::InterfaceFile> interface,
                      DylibFile *umbrella = nullptr);
 
   // Mach-O dylibs can re-export other dylibs as sub-libraries, meaning that the
   // symbols in those sub-libraries will be available under the umbrella

lld/MachO/InputFiles.cpp

for (size_t idx : altEntrySymIdxs) {
     const structs::nlist_64 &sym = nList[idx];
     SubsectionMap &subsecMap = subsections[sym.n_sect - 1];
     uint32_t off = sym.n_value - sectionHeaders[sym.n_sect - 1].addr;
     InputSection *subsec = findContainingSubsection(subsecMap, &off);
     symbols[idx] = createDefined(sym, subsec, off);
   }
 }
 
+OpaqueFile::OpaqueFile(MemoryBufferRef mb, StringRef segName,
+                       StringRef sectName)
+    : InputFile(OpaqueKind, mb) {
+  InputSection *isec = make<InputSection>();
+  isec->file = this;
+  isec->name = sectName.take_front(16);
+  isec->segname = segName.take_front(16);
+  const auto *buf = reinterpret_cast<const uint8_t *>(mb.getBufferStart());
+  isec->data = {buf, mb.getBufferSize()};
+  subsections.push_back({{0, isec}});
+}
+
 ObjFile::ObjFile(MemoryBufferRef mb) : InputFile(ObjKind, mb) {
   auto *buf = reinterpret_cast<const uint8_t *>(mb.getBufferStart());
   auto *hdr = reinterpret_cast<const mach_header_64 *>(mb.getBufferStart());
 
   if (const load_command *cmd = findCommand(hdr, LC_SEGMENT_64)) {
     auto *c = reinterpret_cast<const segment_command_64 *>(cmd);
     sectionHeaders = ArrayRef<section_64>{
         reinterpret_cast<const section_64 *>(c + 1), c->nsects};

lld/MachO/Options.td

def force_load : Separate<["-"], "force_load">,
      Flags<[HelpHidden]>,
      Group<grp_libs>;
 
 def grp_content : OptionGroup<"content">, HelpText<"ADDITIONAL CONTENT">;
 
 def sectcreate : MultiArg<["-"], "sectcreate", 3>,
      MetaVarName<"<segment> <section> <file>">,
      HelpText<"Create <section> in <segment> from the contents of <file>">,
-     Flags<[HelpHidden]>,
      Group<grp_content>;
 def segcreate : MultiArg<["-"], "segcreate", 3>,
      MetaVarName<"<segment> <section> <file>">,
      Alias<sectcreate>,
      HelpText<"Alias for -sectcreate">,
      Flags<[HelpHidden]>,
      Group<grp_content>;
 def filelist : Separate<["-"], "filelist">,

lld/test/MachO/sectcreate.s

+# REQUIRES: x86
+# RUN: llvm-mc -filetype=obj -triple=x86_64-apple-darwin %s -o %t.o
+# RUN: echo "-sectcreate 1.1" >%t1
+# RUN: echo "-sectcreate 1.2" >%t2
+# RUN: echo "-sectcreate 2" >%t3
+# RUN: lld -flavor darwinnew -Z \
+# RUN:     -sectcreate SEG SEC1 %t1 \
+# RUN:     -sectcreate SEG SEC2 %t3 \
+# RUN:     -sectcreate SEG SEC1 %t2 \
+# RUN:     -o %t %t.o
+# RUN: llvm-objdump -s %t | FileCheck %s
+
+# CHECK: Contents of section __text:
+# CHECK: Contents of section __data:
+# CHECK: my string!.
+# CHECK: Contents of section SEC1:
+# CHECK: -sectcreate 1.1.
+# CHECK: -sectcreate 1.2.
+# CHECK: Contents of section SEC2:
+# CHECK: -sectcreate 2.
+
+.text
+.global _main
+_main:
+  mov $0, %eax
+  ret
+
+.data
+.global my_string
+my_string:
+  .string "my string!"