[WebAssembly] Allow for the creation of user-defined custom sections

[sunfish]

Summary

This patch adds a way for users to create their own custom sections to be added to wasm files. At the LLVM IR layer, they are defined through the "wasm.custom_sections" named metadata. The expected use case for this is bindings generators such as wasm-bindgen.

[sbc100]

Sorry maybe I'm jumping the gun here because haven't assigned a reviewer yet..

Can we have test in test/MC/WebAssembly too?

lib/MC/WasmObjectWriter.cpp

995

would ArrayRef<char> ? Or ArrayRef<uint8_t> work better here?

[sunfish]

Add an MC test.

[sunfish]

In D43097#1002844, @sbc100 wrote:

Sorry maybe I'm jumping the gun here because haven't assigned a reviewer yet..

You're good. For LLVM patches, I'm accustomed to posting the patch first, and letting reviewers choose themselves, or in any case figuring it out later.

Can we have test in test/MC/WebAssembly too?

Added.

lib/MC/WasmObjectWriter.cpp

995

I initially did exactly that, with ArrayRef and uint8_t, however writeBytes takes a SmallVectorImpl<char>.

[ncw]

lib/MC/WasmObjectWriter.cpp

995

It could be a bit less ugly with a typedef/"using=" for the std::pair (and in fact for all these vectors here which have rather long types)

[sbc100]

lib/MC/WasmObjectWriter.cpp

995

Why does writeBytes not take an ArrayRef? There is an overload that takes a StringRef..

[sunfish]

Created a struct to replace the noisy std::pair code.

[sunfish]

I think it's because writeBytes is defined to be passed data from an MCFragment's getContents(), which is a SmallVectorImpl<char>. That's the case here too, though the code that does the getContents() is a little separated from the writeBytes. Following @ncw's suggestion, I created a struct type, which not only avoids repeating the type name, but also makes the use of it cleaner than std::pair.

[sunfish]

This user-defined custom sections patch should be ready to go now.

[sbc100]

lib/MC/WasmObjectWriter.cpp

234

This seems like it makes the code more complex than it needs to be.

If you must change it from char* how about just StringRef Name = "" and use Name.empty() to check for its presence?

Is ths part of the change needed for the new feature?

1077

Use ArrayRef<char>, here and in WasmCustomSection?

[ncw]

How do you expect the linker to handle these custom sections? We currently discard them. Will there be semantics for how to merge them, if there are name clashes, etc...?

I can't see a problem with the code and approach - don't wait on my review - but I'm just a bit hazy on the end-to-end story for bindings generators (which presumably will have to hook into things at the C level via a new attribute?)

[sbc100]

In D43097#1033939, @ncw wrote:

How do you expect the linker to handle these custom sections? We currently discard them. Will there be semantics for how to merge them, if there are name clashes, etc...?

I can't see a problem with the code and approach - don't wait on my review - but I'm just a bit hazy on the end-to-end story for bindings generators (which presumably will have to hook into things at the C level via a new attribute?)

I imagine that sections with the same name should be concatenated (with relocations as needed). At least this is what I think the linker currently does for debug sections in ELF. I could be wrong though.

[sbc100]

This should probably be coordinated with https://reviews.llvm.org/D44184 since they both add custom sections. Maybe this should land first and the dwarf change can be simplified?

[sbc100]

@sunfish do you have some time to look at this again and get it landed? I would like to start work on the lld-side, so the we can also get debug info working. (I can start before this lands of course, so just FYI really.)

[sbc100]

LGTM, with comments.

[sbc100]

Ping. Can we get this landed?

[sbc100]

I rebased and cleaned this up a litte: https://reviews.llvm.org/D45297. OK if I land?

[sbc100]

I landed this change in https://reviews.llvm.org/D45297

lib/MC/WasmObjectWriter.cpp

Out << "Off=" << Offset << ", Sym=" << *Symbol << ", Addend=" << Addend
         << ", FixupSection=" << FixupSection->getSectionName();
   }
 
 #if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
   LLVM_DUMP_METHOD void dump() const { print(dbgs()); }
 #endif
 };
 
+struct WasmCustomSection {
+  StringRef Name;
+  const SmallVectorImpl<char> &Contents;
+
+  WasmCustomSection(StringRef Name, const SmallVectorImpl<char> &Contents)
+    : Name(Name), Contents(Contents) {}
+};
+
 #if !defined(NDEBUG)
 raw_ostream &operator<<(raw_ostream &OS, const WasmRelocationEntry &Rel) {
   Rel.print(OS);
   return OS;
 }
 #endif
 
 class WasmObjectWriter : public MCObjectWriter {
class WasmObjectWriter : public MCObjectWriter {
 
   // TargetObjectWriter wrappers.
   bool is64Bit() const { return TargetObjectWriter->is64Bit(); }
   unsigned getRelocType(const MCValue &Target, const MCFixup &Fixup) const {
     return TargetObjectWriter->getRelocType(Target, Fixup);
   }
 
   void startSection(SectionBookkeeping &Section, unsigned SectionId,
-                    const char *Name = nullptr);
+                    Optional<StringRef> Name = Optional<StringRef>());
   void endSection(SectionBookkeeping &Section);
 
 public:
   WasmObjectWriter(std::unique_ptr<MCWasmObjectTargetWriter> MOTW,
                    raw_pwrite_stream &OS)
       : MCObjectWriter(OS, /*IsLittleEndian=*/true),
         TargetObjectWriter(std::move(MOTW)) {}
 
private:
   void writeDataSection(ArrayRef<WasmDataSegment> Segments);
   void writeCodeRelocSection();
   void writeDataRelocSection();
   void writeLinkingMetaDataSection(
       ArrayRef<WasmDataSegment> Segments, uint32_t DataSize,
       ArrayRef<std::pair<StringRef, uint32_t>> SymbolFlags,
       ArrayRef<std::pair<uint16_t, uint32_t>> InitFuncs,
       const std::map<StringRef, std::vector<WasmComdatEntry>>& Comdats);
+  void writeUserCustomSections(
+      const std::vector<WasmCustomSection> &CustomSections);
 
   uint32_t getProvisionalValue(const WasmRelocationEntry &RelEntry);
   void applyRelocations(ArrayRef<WasmRelocationEntry> Relocations,
                         uint64_t ContentsOffset);
 
   void writeRelocations(ArrayRef<WasmRelocationEntry> Relocations);
   uint32_t getRelocationIndexValue(const WasmRelocationEntry &RelEntry);
   uint32_t getFunctionType(const MCSymbolWasm& Symbol);
   uint32_t registerFunctionType(const MCSymbolWasm& Symbol);
 };
 
 } // end anonymous namespace
 
 WasmObjectWriter::~WasmObjectWriter() {}
 
 // Write out a section header and a patchable section size field.
 void WasmObjectWriter::startSection(SectionBookkeeping &Section,
                                     unsigned SectionId,
-                                    const char *Name) {
-  assert((Name != nullptr) == (SectionId == wasm::WASM_SEC_CUSTOM) &&
+                                    Optional<StringRef> Name) {
+  assert((Name.hasValue()) == (SectionId == wasm::WASM_SEC_CUSTOM) &&
          "Only custom sections can have names");
 
-  DEBUG(dbgs() << "startSection " << SectionId << ": " << Name << "\n");
+  DEBUG(dbgs() << "startSection " << SectionId;
+        if (Name.hasValue())
+          dbgs() << ": " << Name.getValue();
+         dbgs() << "\n");
   encodeULEB128(SectionId, getStream());
 
   Section.SizeOffset = getStream().tell();
 
   // The section size. We don't know the size yet, so reserve enough space
   // for any 32-bit value; we'll patch it later.
   encodeULEB128(UINT32_MAX, getStream());
 
   // The position where the section starts, for measuring its size.
   Section.ContentsOffset = getStream().tell();
 
   // Custom sections in wasm also have a string identifier.
   if (SectionId == wasm::WASM_SEC_CUSTOM) {
-    assert(Name);
-    writeString(StringRef(Name));
+    writeString(Name.getValue());
   }
 }
 
 // Now that the section is complete and we know how big it is, patch up the
 // section size field at the start of the section.
 void WasmObjectWriter::endSection(SectionBookkeeping &Section) {
   uint64_t Size = getStream().tell() - Section.ContentsOffset;
   if (uint32_t(Size) != Size)

 void WasmObjectWriter::writeCodeRelocSection() {
   // See: https://github.com/WebAssembly/tool-conventions/blob/master/Linking.md
   // for descriptions of the reloc sections.
 
   if (CodeRelocations.empty())
     return;
 
   SectionBookkeeping Section;
-  startSection(Section, wasm::WASM_SEC_CUSTOM, "reloc.CODE");
+  startSection(Section, wasm::WASM_SEC_CUSTOM, StringRef("reloc.CODE"));
 
   encodeULEB128(wasm::WASM_SEC_CODE, getStream());
   encodeULEB128(CodeRelocations.size(), getStream());
 
   writeRelocations(CodeRelocations);
 
   endSection(Section);
 }
 
 void WasmObjectWriter::writeDataRelocSection() {
   // See: https://github.com/WebAssembly/tool-conventions/blob/master/Linking.md
   // for descriptions of the reloc sections.
 
   if (DataRelocations.empty())
     return;
 
   SectionBookkeeping Section;
-  startSection(Section, wasm::WASM_SEC_CUSTOM, "reloc.DATA");
+  startSection(Section, wasm::WASM_SEC_CUSTOM, StringRef("reloc.DATA"));
 
   encodeULEB128(wasm::WASM_SEC_DATA, getStream());
   encodeULEB128(DataRelocations.size(), getStream());
 
   writeRelocations(DataRelocations);
 
   endSection(Section);
 }
 
 void WasmObjectWriter::writeLinkingMetaDataSection(
     ArrayRef<WasmDataSegment> Segments, uint32_t DataSize,
     ArrayRef<std::pair<StringRef, uint32_t>> SymbolFlags,
     ArrayRef<std::pair<uint16_t, uint32_t>> InitFuncs,
     const std::map<StringRef, std::vector<WasmComdatEntry>>& Comdats) {
   SectionBookkeeping Section;
-  startSection(Section, wasm::WASM_SEC_CUSTOM, "linking");
+  startSection(Section, wasm::WASM_SEC_CUSTOM, StringRef("linking"));
   SectionBookkeeping SubSection;
 
   if (SymbolFlags.size() != 0) {
     startSection(SubSection, wasm::WASM_SYMBOL_INFO);
     encodeULEB128(SymbolFlags.size(), getStream());
     for (auto Pair: SymbolFlags) {
       writeString(Pair.first);
       encodeULEB128(Pair.second, getStream());
for (const auto &C : Comdats) {
       }
     }
     endSection(SubSection);
   }
 
   endSection(Section);
 }
 
+void WasmObjectWriter::writeUserCustomSections(
+    const std::vector<WasmCustomSection> &CustomSections
+) {
+  for (const auto &CustomSection : CustomSections) {
+    SectionBookkeeping Section;
+    startSection(Section, wasm::WASM_SEC_CUSTOM, CustomSection.Name);
+    writeBytes(CustomSection.Contents);
+    endSection(Section);
+  }
+}
+
 uint32_t WasmObjectWriter::getFunctionType(const MCSymbolWasm& Symbol) {
   assert(Symbol.isFunction());
   assert(TypeIndices.count(&Symbol));
   return TypeIndices[&Symbol];
 }
 
 uint32_t WasmObjectWriter::registerFunctionType(const MCSymbolWasm& Symbol) {
   assert(Symbol.isFunction());
void WasmObjectWriter::writeObject(MCAssembler &Asm,
   SmallVector<WasmFunction, 4> Functions;
   SmallVector<uint32_t, 4> TableElems;
   SmallVector<WasmImport, 4> Imports;
   SmallVector<WasmExport, 4> Exports;
   SmallVector<std::pair<StringRef, uint32_t>, 4> SymbolFlags;
   SmallVector<std::pair<uint16_t, uint32_t>, 2> InitFuncs;
   std::map<StringRef, std::vector<WasmComdatEntry>> Comdats;
   SmallVector<WasmDataSegment, 4> DataSegments;
+  std::vector<WasmCustomSection> CustomSections;
   uint32_t DataSize = 0;
 
   // For now, always emit the memory import, since loads and stores are not
   // valid without it. In the future, we could perhaps be more clever and omit
   // it if there are no loads or stores.
   MCSymbolWasm *MemorySym =
       cast<MCSymbolWasm>(Ctx.getOrCreateSymbol("__linear_memory"));
   WasmImport MemImport;
if ((!WS.isDefined() && !WS.isComdat()) ||
       }
 
       Imports.push_back(Import);
     }
   }
 
   for (MCSection &Sec : Asm) {
     auto &Section = static_cast<MCSectionWasm &>(Sec);
+
+    if (cast<MCSectionWasm>(Sec).getSectionName().startswith(".custom_section.")) {
+      if (Section.getFragmentList().empty())
+        continue;
+      if (Section.getFragmentList().size() != 1)
+        report_fatal_error("only one .custom_section section fragment supported");
+      const MCFragment &Frag = *Section.begin();
+      if (Frag.hasInstructions() || Frag.getKind() != MCFragment::FT_Data)
+        report_fatal_error("only data supported in .custom_section section");
+      const auto &DataFrag = cast<MCDataFragment>(Frag);
+      if (!DataFrag.getFixups().empty())
+        report_fatal_error("fixups not supported in .custom_section section");
+      StringRef UserName = Section.getSectionName().substr(16);
+      const SmallVectorImpl<char> &Contents = DataFrag.getContents();
+      CustomSections.push_back(WasmCustomSection(UserName, Contents));
+      continue;
+    }
+
     if (!Section.isWasmData())
       continue;
 
     // .init_array sections are handled specially elsewhere.
     if (cast<MCSectionWasm>(Sec).getSectionName().startswith(".init_array"))
       continue;
 
     uint32_t SegmentIndex = DataSegments.size();
void WasmObjectWriter::writeObject(MCAssembler &Asm,
   writeExportSection(Exports);
   writeElemSection(TableElems);
   writeCodeSection(Asm, Layout, Functions);
   writeDataSection(DataSegments);
   writeCodeRelocSection();
   writeDataRelocSection();
   writeLinkingMetaDataSection(DataSegments, DataSize, SymbolFlags,
                               InitFuncs, Comdats);
+  writeUserCustomSections(CustomSections);
 
   // TODO: Translate the .comment section to the output.
   // TODO: Translate debug sections to the output.
 }
 
 std::unique_ptr<MCObjectWriter>
 llvm::createWasmObjectWriter(std::unique_ptr<MCWasmObjectTargetWriter> MOTW,
                              raw_pwrite_stream &OS) {
   return llvm::make_unique<WasmObjectWriter>(std::move(MOTW), OS);
 }

lib/Target/WebAssembly/WebAssemblyAsmPrinter.cpp

 #include "llvm/CodeGen/AsmPrinter.h"
 #include "llvm/CodeGen/MachineConstantPool.h"
 #include "llvm/CodeGen/MachineInstr.h"
 #include "llvm/CodeGen/MachineModuleInfoImpls.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/IR/GlobalVariable.h"
 #include "llvm/MC/MCContext.h"
 #include "llvm/MC/MCStreamer.h"
+#include "llvm/MC/MCSectionWasm.h"
 #include "llvm/MC/MCSymbol.h"
 #include "llvm/MC/MCSymbolWasm.h"
 #include "llvm/MC/MCSymbolELF.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/TargetRegistry.h"
 #include "llvm/Support/raw_ostream.h"
 using namespace llvm;
 
if (!G.hasInitializer() && G.hasExternalLinkage()) {
         uint16_t Size = M.getDataLayout().getTypeAllocSize(G.getValueType());
         if (TM.getTargetTriple().isOSBinFormatELF())
           getTargetStreamer()->emitGlobalImport(G.getGlobalIdentifier());
         OutStreamer->emitELFSize(getSymbol(&G),
                                  MCConstantExpr::create(Size, OutContext));
       }
     }
   }
+
+  if (const NamedMDNode* Named = M.getNamedMetadata("wasm.custom_sections")) {
+    for (const Metadata *MD : Named->operands()) {
+      const MDTuple *Tuple = dyn_cast<MDTuple>(MD);
+      if (!Tuple || Tuple->getNumOperands() != 2)
+        continue;
+      const MDString *Name = dyn_cast<MDString>(Tuple->getOperand(0));
+      const MDString *Contents = dyn_cast<MDString>(Tuple->getOperand(1));
+      if (!Name || !Contents)
+        continue;
+
+      OutStreamer->PushSection();
+      std::string SectionName = (".custom_section." + Name->getString()).str();
+      MCSectionWasm *mySection =
+        OutContext.getWasmSection(SectionName, SectionKind::getMetadata());
+      OutStreamer->SwitchSection(mySection);
+      OutStreamer->EmitBytes(Contents->getString());
+      OutStreamer->PopSection();
+    }
+  }
 }
 
 void WebAssemblyAsmPrinter::EmitConstantPool() {
   assert(MF->getConstantPool()->getConstants().empty() &&
          "WebAssembly disables constant pools");
 }
 
 void WebAssemblyAsmPrinter::EmitJumpTableInfo() {

test/CodeGen/WebAssembly/custom-sections.ll

+; RUN: llc < %s -asm-verbose=false | FileCheck %s
+
+; Test the mechanism for defining user custom sections.
+
+target datalayout = "e-m:e-p:32:32-i64:64-n32:64-S128"
+target triple = "wasm32-unknown-unknown-wasm"
+
+!0 = !{ !"red", !"foo" }
+!1 = !{ !"green", !"bar" }
+!2 = !{ !"green", !"qux" }
+!wasm.custom_sections = !{ !0, !1, !2 }
+
+; CHECK:      .section  .custom_section.red,"",@
+; CHECK-NEXT: .ascii  "foo"
+
+; CHECK:      .section  .custom_section.green,"",@
+; CHECK-NEXT: .ascii  "bar"
+
+; CHECK:      .section  .custom_section.green,"",@
+; CHECK-NEXT: .ascii  "qux"

test/MC/WebAssembly/custom-sections.ll

+; RUN: llc -filetype=obj %s -o - | llvm-readobj -s | FileCheck %s
+
+; Test the mechanism for defining user custom sections.
+
+target triple = "wasm32-unknown-unknown-wasm"
+
+!0 = !{ !"red", !"foo" }
+!1 = !{ !"green", !"bar" }
+!2 = !{ !"green", !"qux" }
+!wasm.custom_sections = !{ !0, !1, !2 }
+
+; CHECK:  Section {
+; CHECK:    Type: CUSTOM (0x0)
+; CHECK:    Size: 7
+; CHECK:    Offset: 86
+; CHECK:    Name: red
+; CHECK:  }
+; CHECK:  Section {
+; CHECK:    Type: CUSTOM (0x0)
+; CHECK:    Size: 12
+; CHECK:    Offset: 99
+; CHECK:    Name: green
+; CHECK:  }