[llvm-dsymutil] Add support for __swift_ast MachO DWARF section

[fjricci]

Summary

Xcode's dsymutil emits a __swift_ast DWARF section, which is required for debugging,
and which contains a byte-for-byte dump of the swiftmodule file.
Add this feature to llvm-dsymutil.

Tested with gobjdump --dwarf=info -s, by verifying that the contents of
__DWARF.__swift_ast match between Xcode's dsymutil and llvm-dsymutil
(Xcode's dwarfdump and llvm-dwarfdump don't currently recognize the
__swift_ast section).

[JDevlieghere]

This LGTM. We should probably also have a test for this. Committing a swift binary upstream might not be the best idea but I don't see an alternative.

tools/dsymutil/DwarfLinker.cpp

716 ↗

(On Diff #117521)

I think you can also simplify this to MS->SwitchSection

718 ↗

(On Diff #117521)

And MS->EmitBytes

[aprantl]

Perhaps it's possible to do assembler or LLVM IR instead of a binary?

[fjricci]

Presumably, at least the swiftmodule would need to be pre-generated, since it's swift AST

[aprantl]

That's true, though for testing dsymutil, we could just have a bogus section with some recognizable string in it, since all it does is copy the section data.

[fjricci]

I have a test case working with an IR file, but it looks like most of the existing dsymutil tests just use an object file - should I do that here as well? (I don't think we get much value from using an IR file over an object file)

[fjricci]

Looks like they check in binaries as well. I'll upload a version with a binary for now, and I can swap it out if we decide we want to.

[fjricci]

Add test

[fjricci]

Asm->OutStreamer -> MS

[aprantl]

lib/MC/MCObjectFileInfo.cpp

217 ↗

(On Diff #117728)

this comment doesn't really add anything. I would either explain what that section is or remove it.

tools/dsymutil/DwarfLinker.cpp

3491 ↗

(On Diff #117728)

. at the end, llvm coding guides always want full setences.

[fjricci]

Clean up comments

[fjricci]

Fix -no-output mode

[JDevlieghere]

test/tools/dsymutil/swift-ast.test

1 ↗

(On Diff #117733)

Why is this necessary?

2 ↗

(On Diff #117733)

You can use %T for a temporary directory unique to your test.

9 ↗

(On Diff #117733)

For consistency with the other tests, please include the actual compile command. (i.e. Compile with:)

tools/dsymutil/DwarfLinker.cpp

716 ↗

(On Diff #117728)

The swift AST section must be 32-bit aligned. Our internal code has Section->setAlignment(1 << 5);. though I think the operand is in bytes (rather than bits). Maybe I'm wrong or maybe this is because LLDB has stricter requirements. Either way we'd want to do the same here.

717 ↗

(On Diff #117728)

No {} around single line if/for.

[fjricci]

Address comments

[JDevlieghere]

Thanks Francis, LGTM!

[fjricci]

The test fails on non apple-x86_64 builders, reverted the patch for now

[fjricci]

I think that postfixing the binary name with ".macho.x86_64" will work, that seems to be what the other tests do, although I can't seem to find a lit config relating to that.

[fjricci]

Weirdly, appending .macho.x86_64 fixed the armv7/thumbv7 buildbots, but not aarch64. Will try to diagnose.

[fjricci]

They fail with:
while processing dwarf streamer init
error: : error: unable to get target for 'x86_64-apple-darwin', see --version and --triple.

[JDevlieghere]

Moving your test to the X86 subfolder will solve this. They are automatically disabled (see lit.local.cfg) when there's no X86 backend, which is what's causing some bots to fail.

llvm/trunk/include/llvm/MC/MCObjectFileInfo.h

protected:
   MCSection *DwarfCUIndexSection;
   MCSection *DwarfTUIndexSection;
 
   /// Section for newer gnu pubnames.
   MCSection *DwarfGnuPubNamesSection;
   /// Section for newer gnu pubtypes.
   MCSection *DwarfGnuPubTypesSection;
 
+  // Section for Swift AST
+  MCSection *DwarfSwiftASTSection;
+
   MCSection *COFFDebugSymbolsSection;
   MCSection *COFFDebugTypesSection;
 
   /// Extra TLS Variable Data section.
   ///
   /// If the target needs to put additional information for a TLS variable,
   /// it'll go here.
   MCSection *TLSExtraDataSection;
public:
   MCSection *getDwarfStrDWOSection() const { return DwarfStrDWOSection; }
   MCSection *getDwarfLineDWOSection() const { return DwarfLineDWOSection; }
   MCSection *getDwarfLocDWOSection() const { return DwarfLocDWOSection; }
   MCSection *getDwarfStrOffDWOSection() const { return DwarfStrOffDWOSection; }
   MCSection *getDwarfStrOffSection() const { return DwarfStrOffSection; }
   MCSection *getDwarfAddrSection() const { return DwarfAddrSection; }
   MCSection *getDwarfCUIndexSection() const { return DwarfCUIndexSection; }
   MCSection *getDwarfTUIndexSection() const { return DwarfTUIndexSection; }
+  MCSection *getDwarfSwiftASTSection() const { return DwarfSwiftASTSection; }
 
   MCSection *getCOFFDebugSymbolsSection() const {
     return COFFDebugSymbolsSection;
   }
   MCSection *getCOFFDebugTypesSection() const {
     return COFFDebugTypesSection;
   }
 

llvm/trunk/lib/MC/MCObjectFileInfo.cpp

void MCObjectFileInfo::initMachOMCObjectFileInfo(const Triple &T) {
   // 16 character section limit...
   DwarfAccelNamespaceSection =
       Ctx->getMachOSection("__DWARF", "__apple_namespac", MachO::S_ATTR_DEBUG,
                            SectionKind::getMetadata(), "namespac_begin");
   DwarfAccelTypesSection =
       Ctx->getMachOSection("__DWARF", "__apple_types", MachO::S_ATTR_DEBUG,
                            SectionKind::getMetadata(), "types_begin");
 
+  DwarfSwiftASTSection =
+      Ctx->getMachOSection("__DWARF", "__swift_ast", MachO::S_ATTR_DEBUG,
+                           SectionKind::getMetadata());
+
   DwarfAbbrevSection =
       Ctx->getMachOSection("__DWARF", "__debug_abbrev", MachO::S_ATTR_DEBUG,
                            SectionKind::getMetadata(), "section_abbrev");
   DwarfInfoSection =
       Ctx->getMachOSection("__DWARF", "__debug_info", MachO::S_ATTR_DEBUG,
                            SectionKind::getMetadata(), "section_info");
   DwarfLineSection =
       Ctx->getMachOSection("__DWARF", "__debug_line", MachO::S_ATTR_DEBUG,

llvm/trunk/test/tools/dsymutil/Inputs/swift-ast.swiftmodule

+SWIFTMODULE DATA

llvm/trunk/test/tools/dsymutil/X86/swift-ast-x86_64.test

+RUN: llvm-dsymutil -oso-prepend-path %p/.. %p/../Inputs/swift-ast.macho.x86_64 -o %T/swift-ast.dSYM -verbose | FileCheck %s --check-prefix=DSYMUTIL
+RUN: llvm-readobj -sections -section-data %T/swift-ast.dSYM/Contents/Resources/DWARF/swift-ast.macho.x86_64 | FileCheck %s --check-prefix=READOBJ
+
+The tested object file has been created by the dummy Swift code:
+let x = 1
+
+Compiled with: swiftc /tmp/test.swift -Onone -target x86_64-apple-macosx10.9 -c
+Linked with: ld swift-ast.o -add_ast_path Inputs/swift-ast.swiftmodule -arch x86_64 -lSystem -macosx_version_min 10.9.0
+
+DSYMUTIL: filename:{{.*}}swift-ast.swiftmodule
+DSYMUTIL: DEBUG MAP OBJECT:{{.*}}swift-ast.swiftmodule
+
+READOBJ: Name:{{.*}}__swift_ast
+READOBJ: |SWIFTMODULE DATA|

llvm/trunk/tools/dsymutil/BinaryHolder.h

class BinaryHolder {
   /// corresponding archive to memory and return the MemoryBufferRefs
   /// corresponding to the described member. Multiple buffers are
   /// returned when there are multiple architectures available for the
   /// requested file.
   ErrorOr<std::vector<MemoryBufferRef>>
   MapArchiveAndGetMemberBuffers(StringRef Filename,
                                 sys::TimePoint<std::chrono::seconds> Timestamp);
 
+  void changeBackingMemoryBuffer(std::unique_ptr<MemoryBuffer> &&MemBuf);
+  ErrorOr<const object::ObjectFile &> getObjfileForArch(const Triple &T);
+
+public:
+  BinaryHolder(bool Verbose) : Verbose(Verbose) {}
+
   /// Return the MemoryBufferRef that holds the memory mapping for the
   /// given \p Filename. This function will try to parse archive
   /// member specifications of the form /path/to/archive.a(member.o).
   ///
   /// The returned MemoryBufferRefs points to a buffer owned by this
   /// object. The buffer is valid until the next call to
   /// GetMemoryBufferForFile() on this object.
   /// Multiple buffers are returned when there are multiple
   /// architectures available for the requested file.
   ErrorOr<std::vector<MemoryBufferRef>>
   GetMemoryBuffersForFile(StringRef Filename,
                           sys::TimePoint<std::chrono::seconds> Timestamp);
 
-  void changeBackingMemoryBuffer(std::unique_ptr<MemoryBuffer> &&MemBuf);
-  ErrorOr<const object::ObjectFile &> getObjfileForArch(const Triple &T);
-
-public:
-  BinaryHolder(bool Verbose) : Verbose(Verbose) {}
-
   /// Get the ObjectFiles designated by the \p Filename. This
   /// might be an archive member specification of the form
   /// /path/to/archive.a(member.o).
   ///
   /// Calling this function invalidates the previous mapping owned by
   /// the BinaryHolder. Multiple buffers are returned when there are
   /// multiple architectures available for the requested file.
   ErrorOr<std::vector<const object::ObjectFile *>>

llvm/trunk/tools/dsymutil/DebugMap.h

public:
   const_iterator begin() const { return Objects.begin(); }
 
   const_iterator end() const { return Objects.end(); }
 
   /// This function adds an DebugMapObject to the list owned by this
   /// debug map.
   DebugMapObject &
   addDebugMapObject(StringRef ObjectFilePath,
-                    sys::TimePoint<std::chrono::seconds> Timestamp);
+                    sys::TimePoint<std::chrono::seconds> Timestamp,
+                    uint8_t Type);
 
   const Triple &getTriple() const { return BinaryTriple; }
 
   StringRef getBinaryPath() const { return BinaryPath; }
 
   void print(raw_ostream &OS) const;
 
 #ifndef NDEBUG
public:
   const DebugMapEntry *lookupObjectAddress(uint64_t Address) const;
 
   llvm::StringRef getObjectFilename() const { return Filename; }
 
   sys::TimePoint<std::chrono::seconds> getTimestamp() const {
     return Timestamp;
   }
 
+  uint8_t getType() const { return Type; }
+
   iterator_range<StringMap<SymbolMapping>::const_iterator> symbols() const {
     return make_range(Symbols.begin(), Symbols.end());
   }
 
   void print(raw_ostream &OS) const;
 #ifndef NDEBUG
   void dump() const;
 #endif
 private:
   friend class DebugMap;
   /// DebugMapObjects can only be constructed by the owning DebugMap.
   DebugMapObject(StringRef ObjectFilename,
-                 sys::TimePoint<std::chrono::seconds> Timestamp);
+                 sys::TimePoint<std::chrono::seconds> Timestamp, uint8_t Type);
 
   std::string Filename;
   sys::TimePoint<std::chrono::seconds> Timestamp;
   StringMap<SymbolMapping> Symbols;
   DenseMap<uint64_t, DebugMapEntry *> AddressToMapping;
+  uint8_t Type;
 
   /// For YAMLIO support.
   ///@{
   friend yaml::MappingTraits<dsymutil::DebugMapObject>;
   friend yaml::SequenceTraits<std::vector<std::unique_ptr<DebugMapObject>>>;
   DebugMapObject() = default;
 
 public:

llvm/trunk/tools/dsymutil/DebugMap.cpp

 #include <algorithm>
 
 namespace llvm {
 namespace dsymutil {
 
 using namespace llvm::object;
 
 DebugMapObject::DebugMapObject(StringRef ObjectFilename,
-                               sys::TimePoint<std::chrono::seconds> Timestamp)
-    : Filename(ObjectFilename), Timestamp(Timestamp) {}
+                               sys::TimePoint<std::chrono::seconds> Timestamp,
+                               uint8_t Type)
+    : Filename(ObjectFilename), Timestamp(Timestamp), Type(Type) {}
 
 bool DebugMapObject::addSymbol(StringRef Name, Optional<uint64_t> ObjectAddress,
                                uint64_t LinkedAddress, uint32_t Size) {
   auto InsertResult = Symbols.insert(
       std::make_pair(Name, SymbolMapping(ObjectAddress, LinkedAddress, Size)));
 
   if (ObjectAddress && InsertResult.second)
     AddressToMapping[*ObjectAddress] = &*InsertResult.first;

 }
 
 #ifndef NDEBUG
 void DebugMapObject::dump() const { print(errs()); }
 #endif
 
 DebugMapObject &
 DebugMap::addDebugMapObject(StringRef ObjectFilePath,
-                            sys::TimePoint<std::chrono::seconds> Timestamp) {
-  Objects.emplace_back(new DebugMapObject(ObjectFilePath, Timestamp));
+                            sys::TimePoint<std::chrono::seconds> Timestamp,
+                            uint8_t Type) {
+  Objects.emplace_back(new DebugMapObject(ObjectFilePath, Timestamp, Type));
   return *Objects.back();
 }
 
 const DebugMapObject::DebugMapEntry *
 DebugMapObject::lookupSymbol(StringRef SymbolName) const {
   StringMap<SymbolMapping>::const_iterator Sym = Symbols.find(SymbolName);
   if (Sym == Symbols.end())
     return nullptr;
for (const auto &Sym : ErrOrObjectFile->symbols()) {
         if (!Name)
           consumeError(Name.takeError());
         continue;
       }
       SymbolAddresses[*Name] = Address;
     }
   }
 
-  dsymutil::DebugMapObject Res(Path, sys::toTimePoint(Timestamp));
+  dsymutil::DebugMapObject Res(Path, sys::toTimePoint(Timestamp), MachO::N_OSO);
   for (auto &Entry : Entries) {
     auto &Mapping = Entry.second;
     Optional<uint64_t> ObjAddress;
     if (Mapping.ObjectAddress)
       ObjAddress = *Mapping.ObjectAddress;
     auto AddressIt = SymbolAddresses.find(Entry.first);
     if (AddressIt != SymbolAddresses.end())
       ObjAddress = AddressIt->getValue();
     Res.addSymbol(Entry.first, ObjAddress, Mapping.BinaryAddress, Mapping.Size);
   }
   return Res;
 }
 }
 }

llvm/trunk/tools/dsymutil/DwarfLinker.cpp

public:
 
   /// Emit the abbreviation table \p Abbrevs to the debug_abbrev section.
   void emitAbbrevs(const std::vector<std::unique_ptr<DIEAbbrev>> &Abbrevs,
                    unsigned DwarfVersion);
 
   /// Emit the string table described by \p Pool.
   void emitStrings(const NonRelocatableStringpool &Pool);
 
+  /// Emit the swift_ast section stored in \p Buffers.
+  void emitSwiftAST(const std::vector<MemoryBufferRef> &Buffers);
+
   /// Emit debug_ranges for \p FuncRange by translating the
   /// original \p Entries.
   void emitRangesEntries(
       int64_t UnitPcOffset, uint64_t OrigLowPc,
       const FunctionIntervals::const_iterator &FuncRange,
       const std::vector<DWARFDebugRangeList::RangeListEntry> &Entries,
       unsigned AddressSize);
 

 void DwarfStreamer::emitStrings(const NonRelocatableStringpool &Pool) {
   Asm->OutStreamer->SwitchSection(MOFI->getDwarfStrSection());
   for (auto *Entry = Pool.getFirstEntry(); Entry;
        Entry = Pool.getNextEntry(Entry))
     Asm->OutStreamer->EmitBytes(
         StringRef(Entry->getKey().data(), Entry->getKey().size() + 1));
 }
 
+/// Emit the swift_ast section stored in \p Buffers.
+void DwarfStreamer::emitSwiftAST(const std::vector<MemoryBufferRef> &Buffers) {
+  MCSection *SwiftASTSection = MOFI->getDwarfSwiftASTSection();
+  SwiftASTSection->setAlignment(1 << 5);
+  MS->SwitchSection(SwiftASTSection);
+  for (auto Buf : Buffers)
+    MS->EmitBytes(Buf.getBuffer());
+}
+
 /// Emit the debug_range section contents for \p FuncRange by
 /// translating the original \p Entries. The debug_range section
 /// format is totally trivial, consisting just of pairs of address
 /// sized addresses describing the ranges.
 void DwarfStreamer::emitRangesEntries(
     int64_t UnitPcOffset, uint64_t OrigLowPc,
     const FunctionIntervals::const_iterator &FuncRange,
     const std::vector<DWARFDebugRangeList::RangeListEntry> &Entries,
void DwarfLinker::loadClangModule(StringRef Filename, StringRef ModulePath,
                                   StringRef ModuleName, uint64_t DwoId,
                                   DebugMap &ModuleMap, unsigned Indent) {
   SmallString<80> Path(Options.PrependPath);
   if (sys::path::is_relative(Filename))
     sys::path::append(Path, ModulePath, Filename);
   else
     sys::path::append(Path, Filename);
   BinaryHolder ObjHolder(Options.Verbose);
-  auto &Obj =
-      ModuleMap.addDebugMapObject(Path, sys::TimePoint<std::chrono::seconds>());
+  auto &Obj = ModuleMap.addDebugMapObject(
+      Path, sys::TimePoint<std::chrono::seconds>(), MachO::N_OSO);
   auto ErrOrObj = loadObject(ObjHolder, Obj, ModuleMap);
   if (!ErrOrObj) {
     // Try and emit more helpful warnings by applying some heuristics.
     StringRef ObjFile = CurrentDebugObject->getObjectFilename();
     bool isClangModule = sys::path::extension(Filename).equals(".pcm");
     bool isArchive = ObjFile.endswith(")");
     if (isClangModule) {
       StringRef ModuleCacheDir = sys::path::parent_path(Path);
bool DwarfLinker::link(const DebugMap &Map) {
   UnitID = 0;
   DebugMap ModuleMap(Map.getTriple(), Map.getBinaryPath());
 
   for (const auto &Obj : Map.objects()) {
     CurrentDebugObject = Obj.get();
 
     if (Options.Verbose)
       outs() << "DEBUG MAP OBJECT: " << Obj->getObjectFilename() << "\n";
+
+    // N_AST objects (swiftmodule files) should get dumped directly into the
+    // appropriate DWARF section.
+    if (Obj->getType() == MachO::N_AST) {
+      auto ErrOrMemBufferRefs = BinHolder.GetMemoryBuffersForFile(
+          Obj->getObjectFilename(), Obj->getTimestamp());
+      if (ErrOrMemBufferRefs.getError())
+        continue;
+      if (!Options.NoOutput)
+        Streamer->emitSwiftAST(ErrOrMemBufferRefs.get());
+      continue;
+    }
+
     auto ErrOrObj = loadObject(BinHolder, *Obj, Map);
     if (!ErrOrObj)
       continue;
 
     // Look for relocations that correspond to debug map entries.
     RelocationManager RelocMgr(*this);
     if (!RelocMgr.findValidRelocsInDebugInfo(*ErrOrObj, *Obj)) {
       if (Options.Verbose)

llvm/trunk/tools/dsymutil/MachODebugMapParser.cpp

void MachODebugMapParser::switchToNewDebugMapObject(
 
   auto ErrOrAchObj =
       CurrentObjectHolder.GetAs<MachOObjectFile>(Result->getTriple());
   if (auto Err = ErrOrAchObj.getError()) {
     return Warning(Twine("cannot open debug object \"") + Path.str() + "\": " +
                    Err.message() + "\n");
   }
 
-  CurrentDebugMapObject = &Result->addDebugMapObject(Path, Timestamp);
+  CurrentDebugMapObject =
+      &Result->addDebugMapObject(Path, Timestamp, MachO::N_OSO);
   loadCurrentObjectFileSymbols(*ErrOrAchObj);
 }
 
 static std::string getArchName(const object::MachOObjectFile &Obj) {
   Triple T = Obj.getArchTriple();
   return T.getArchName();
 }
 
if (!(Type & MachO::N_STAB))
     return;
 
   const char *Name = &MainBinaryStrings.data()[StringIndex];
 
   // An N_OSO entry represents the start of a new object file description.
   if (Type == MachO::N_OSO)
     return switchToNewDebugMapObject(Name, sys::toTimePoint(Value));
 
+  if (Type == MachO::N_AST) {
+    SmallString<80> Path(PathPrefix);
+    sys::path::append(Path, Name);
+    Result->addDebugMapObject(Path, sys::toTimePoint(Value), Type);
+    return;
+  }
+
   // If the last N_OSO object file wasn't found,
   // CurrentDebugMapObject will be null. Do not update anything
   // until we find the next valid N_OSO entry.
   if (!CurrentDebugMapObject)
     return;
 
   uint32_t Size = 0;
   switch (Type) {