[DebugInfo]Support for debug_macinfo.dwo section in llvm and llvm-dwarfdump.

[SouraVX]

Summary

This patch adds support for debug_macinfo.dwo section[pre-standardized] to llvm and llvm-dwarfdump.

This is the first patch in direction of debug info WRT to macros.

• This was necessary step forward in direction of debug info WRT to macros. macros section could be huge, so transferring this completely to dwo file reduce the size of executable + GDB also supports this style[loading macinfo.dwo section from dwo file].
• This contains redundancies and require a lot cleanup, So I would like to have some initial thoughts on this.
• Furthermore, once both macinfo /macinfo.dwo are set up, will try construct DWARF5 macro/macro.dwo on top of this infra, perhaps share/Reuse some common code.

[merge_guards_bot]

Build result: fail - 60307 tests passed, 1 failed and 732 were skipped.

failed: LLVM.DebugInfo/debugmacinfo-dwo.test

Log files: console-log.txt, CMakeCache.txt

[SouraVX]

In D70705#1759783, @merge_guards_bot wrote:

Build result: fail - 60307 tests passed, 1 failed and 732 were skipped.

failed: LLVM.DebugInfo/debugmacinfo-dwo.test

Log files: console-log.txt, CMakeCache.txt

Log -- from BOT

Is this failure could be spurious,By any chance ??
Log shows the binary is not found by filecheck. -- Running fine on my System. Passing as well, no warning

 'RUN: at line 1';   /mnt/disks/ssd0/agent/workspace/amd64_debian_testing_clang8/build/bin/llvm-dwarfdump -debug-macro /mnt/disks/ssd0/agent/workspace/amd64_debian_testing_clang8/llvm/test/DebugInfo/Inputs/dwarfdump-macro.dwo    | /mnt/disks/ssd0/agent/workspace/amd64_debian_testing_clang8/build/bin/FileCheck /mnt/disks/ssd0/agent/workspace/amd64_debian_testing_clang8/llvm/test/DebugInfo/debugmacinfo-dwo.test -check-prefix TEST_MACINFODWO
--
Exit Code: 2

Command Output (stderr):
--
error: /mnt/disks/ssd0/agent/workspace/amd64_debian_testing_clang8/llvm/test/DebugInfo/Inputs/dwarfdump-macro.dwo: The file was not recognized as a valid object file
FileCheck error: '-' is empty.
FileCheck command line:  /mnt/disks/ssd0/agent/workspace/amd64_debian_testing_clang8/build/bin/FileCheck /mnt/disks/ssd0/agent/workspace/amd64_debian_testing_clang8/llvm/test/DebugInfo/debugmacinfo-dwo.test -check-prefix TEST_MACINFODWO

[aprantl]

Seems reasonable.

llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp

1237

that comment looks misplaced

[SouraVX]

Thanks @aprantl for reviewing this!
Hi @dblaikie , @probinson , would you guys also want to share some final thoughts before I move forward and commit this.

[dblaikie]

This looks to be missing test coverage - this one patch adds both parsing and emission support for macinfo.dwo, but only tests the parsing support (there's no llc test that tests the emission)

Please separate patches like this into, first, a parsing patch (adds libDebugInfoDWARF functionality and test(s) like the one you have here) and then in a separate patch the emission support & tests for that.

(to be fair, I'm sort of OK (& have done myself in the past) with /only/ testing the emission, and getting test coverage for the parsing as a consequence of the emission testing, without separate parsing test coverage - but the opposite (only testing the parsing, without the emission) isn't sufficient)

[dblaikie]

This seems like the wrong direction since the macinfo section was replaced (like loc/ranges) with a new format in DWARFv5: "Replace the .debug_macinfo macro information representation with with a .debug_macro representation that can potentially be much more compact."

Seems the thing to do would be to correct the DWARFv5 non-split-DWARF support by using the new format there, then expand that support to cover the split DWARF case as well.

llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp

2771–2807

Please refactor these functions to use a common implementation.

eg:

void DwarfDebug::emitDebugMacinfoDWO() {
  emitDebugMacinfoImpl(Asm->getObjFileLowering().getDwarfMacinfoDWOSection());
}
void DwarfDebug::emitDebugMacinfo() {
  emitDebugMacinfoImpl(Asm->getObjFileLowering().getDwarfMacinfoSection());
}
void DwarfDebug::emitDebugMacinfoImpl(MCSection *Section) {
  for (const auto &P : CUMap) {
    auto &TheCU = *P.second;
    auto *SkCU = TheCU.getSkeleton();
    DwarfCompileUnit &U = SkCU ? *SkCU : TheCU;
    auto *CUNode = cast<DICompileUnit>(P.first);
    DIMacroNodeArray Macros = CUNode->getMacros();
    if (Macros.empty())
      continue;
    Asm->OutStreamer->SwitchSection(Section);
    Asm->OutStreamer->EmitLabel(U.getMacroLabelBegin());
    handleMacroNodes(Macros, U);
    Asm->OutStreamer->AddComment("End Of Macro List Mark");
    Asm->emitInt8(0);
  }
}

[dblaikie]

(you can ignore my other feedback, since this is just not the right direction in general - but the feedback might be useful for ideas on future patches for this and other features)

[SouraVX]

In D70705#1767133, @dblaikie wrote:

This looks to be missing test coverage - this one patch adds both parsing and emission support for macinfo.dwo, but only tests the parsing support (there's no llc test that tests the emission)

Please separate patches like this into, first, a parsing patch (adds libDebugInfoDWARF functionality and test(s) like the one you have here) and then in a separate patch the emission support & tests for that.

(to be fair, I'm sort of OK (& have done myself in the past) with /only/ testing the emission, and getting test coverage for the parsing as a consequence of the emission testing, without separate parsing test coverage - but the opposite (only testing the parsing, without the emission) isn't sufficient)

(you can ignore my other feedback, since this is just not the right direction in general - but the feedback might be useful for ideas on future patches for this and other features)

Thanks David, I understand a lot cleanup has to be done for unification and to build DWARFv5 macro on top of this. I'll add emission test case soon. This macinfo.dwo emission and loclist parsing test case were in my TODO list, but I was carried away by other stuff. Will do those in separate patches/reviews.
Thanks again for sharing the implementation feedback also -- will work your inputs.

llvm/include/llvm/DebugInfo/DWARF/DWARFContext.h

class DWARFContext : public DIContext {
   std::unique_ptr<DWARFDebugNames> Names;
   std::unique_ptr<AppleAcceleratorTable> AppleNames;
   std::unique_ptr<AppleAcceleratorTable> AppleTypes;
   std::unique_ptr<AppleAcceleratorTable> AppleNamespaces;
   std::unique_ptr<AppleAcceleratorTable> AppleObjC;
 
   DWARFUnitVector DWOUnits;
   std::unique_ptr<DWARFDebugAbbrev> AbbrevDWO;
+  std::unique_ptr<DWARFDebugMacro> MacroDWO;
 
   /// The maximum DWARF version of all units.
   unsigned MaxVersion = 0;
 
   struct DWOFile {
     object::OwningBinary<object::ObjectFile> File;
     std::unique_ptr<DWARFContext> Context;
   };
public:
   const DWARFDebugFrame *getDebugFrame();
 
   /// Get a pointer to the parsed eh frame information object.
   const DWARFDebugFrame *getEHFrame();
 
   /// Get a pointer to the parsed DebugMacro object.
   const DWARFDebugMacro *getDebugMacro();
 
+  /// Get a pointer to the parsed DebugMacroDWO object.
+  const DWARFDebugMacro *getDebugMacroDWO();
+
   /// Get a reference to the parsed accelerator table object.
   const DWARFDebugNames &getDebugNames();
 
   /// Get a reference to the parsed accelerator table object.
   const AppleAcceleratorTable &getAppleNames();
 
   /// Get a reference to the parsed accelerator table object.
   const AppleAcceleratorTable &getAppleTypes();

llvm/include/llvm/DebugInfo/DWARF/DWARFObject.h

public:
   virtual const DWARFSection &getFrameSection() const { return Dummy; }
   virtual const DWARFSection &getEHFrameSection() const { return Dummy; }
   virtual const DWARFSection &getLineSection() const { return Dummy; }
   virtual StringRef getLineStrSection() const { return ""; }
   virtual StringRef getStrSection() const { return ""; }
   virtual const DWARFSection &getRangesSection() const { return Dummy; }
   virtual const DWARFSection &getRnglistsSection() const { return Dummy; }
   virtual StringRef getMacinfoSection() const { return ""; }
+  virtual StringRef getMacinfoDWOSection() const { return ""; }
   virtual const DWARFSection &getPubnamesSection() const { return Dummy; }
   virtual const DWARFSection &getPubtypesSection() const { return Dummy; }
   virtual const DWARFSection &getGnuPubnamesSection() const { return Dummy; }
   virtual const DWARFSection &getGnuPubtypesSection() const { return Dummy; }
   virtual const DWARFSection &getStrOffsetsSection() const { return Dummy; }
   virtual void
   forEachInfoDWOSections(function_ref<void(const DWARFSection &)> F) const {}
   virtual void

llvm/include/llvm/MC/MCObjectFileInfo.h

protected:
   // These are used for the Fission separate debug information files.
   MCSection *DwarfInfoDWOSection = nullptr;
   MCSection *DwarfTypesDWOSection = nullptr;
   MCSection *DwarfAbbrevDWOSection = nullptr;
   MCSection *DwarfStrDWOSection = nullptr;
   MCSection *DwarfLineDWOSection = nullptr;
   MCSection *DwarfLocDWOSection = nullptr;
   MCSection *DwarfStrOffDWOSection = nullptr;
+  MCSection *DwarfMacinfoDWOSection = nullptr;
 
   /// The DWARF v5 string offset and address table sections.
   MCSection *DwarfStrOffSection = nullptr;
   MCSection *DwarfAddrSection = nullptr;
   /// The DWARF v5 range list section.
   MCSection *DwarfRnglistsSection = nullptr;
   /// The DWARF v5 locations list section.
   MCSection *DwarfLoclistsSection = nullptr;
public:
   MCSection *getDwarfStrOffSection() const { return DwarfStrOffSection; }
   MCSection *getDwarfAddrSection() const { return DwarfAddrSection; }
   MCSection *getDwarfRnglistsDWOSection() const {
     return DwarfRnglistsDWOSection;
   }
   MCSection *getDwarfLoclistsDWOSection() const {
     return DwarfLoclistsDWOSection;
   }
+  MCSection *getDwarfMacinfoDWOSection() const {
+    return DwarfMacinfoDWOSection;
+  }
   MCSection *getDwarfCUIndexSection() const { return DwarfCUIndexSection; }
   MCSection *getDwarfTUIndexSection() const { return DwarfTUIndexSection; }
   MCSection *getDwarfSwiftASTSection() const { return DwarfSwiftASTSection; }
 
   MCSection *getCOFFDebugSymbolsSection() const {
     return COFFDebugSymbolsSection;
   }
   MCSection *getCOFFDebugTypesSection() const {

llvm/lib/CodeGen/AsmPrinter/DwarfDebug.h

class DwarfDebug : public DebugHandlerBase {
 
   /// Emit address ranges into a debug ranges section.
   void emitDebugRanges();
   void emitDebugRangesDWO();
   void emitDebugRangesImpl(const DwarfFile &Holder, MCSection *Section);
 
   /// Emit macros into a debug macinfo section.
   void emitDebugMacinfo();
+  /// Emit macros into a debug macinfo.dwo section.
+  void emitDebugMacinfoDWO();
   void emitMacro(DIMacro &M);
   void emitMacroFile(DIMacroFile &F, DwarfCompileUnit &U);
   void handleMacroNodes(DIMacroNodeArray Nodes, DwarfCompileUnit &U);
 
   /// DWARF 5 Experimental Split Dwarf Emitters
 
   /// Initialize common features of skeleton units.
   void initSkeletonUnit(const DwarfUnit &U, DIE &Die,

llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp

if (getDwarfVersion() >= 5) {
           U.addSectionLabel(U.getUnitDie(), dwarf::DW_AT_loclists_base,
                             DebugLocs.getSym(),
                             TLOF.getDwarfLoclistsSection()->getBeginSymbol());
       }
     }
 
     auto *CUNode = cast<DICompileUnit>(P.first);
     // If compile Unit has macros, emit "DW_AT_macro_info" attribute.
-    if (CUNode->getMacros())
+    if (CUNode->getMacros() && !useSplitDwarf())
       U.addSectionLabel(U.getUnitDie(), dwarf::DW_AT_macro_info,
                         U.getMacroLabelBegin(),
                         TLOF.getDwarfMacinfoSection()->getBeginSymbol());
   }
 
   // Emit all frontend-produced Skeleton CUs, i.e., Clang modules.
   for (auto *CUNode : MMI->getModule()->debug_compile_units())
     if (CUNode->getDWOId())
void DwarfDebug::endModule() {
 
   // Emit info into a debug aranges section.
   if (GenerateARangeSection)
     emitDebugARanges();
 
   // Emit info into a debug ranges section.
   emitDebugRanges();
 
+  if (useSplitDwarf())
+  // Emit info into a debug macinfo.dwo section.
+    emitDebugMacinfoDWO();
+  else
   // Emit info into a debug macinfo section.
     emitDebugMacinfo();
 
   if (useSplitDwarf()) {
     emitDebugStrDWO();
     emitDebugInfoDWO();
     emitDebugAbbrevDWO();
     emitDebugLineDWO();
     emitDebugRangesDWO();
   }
 
   emitDebugAddr();
void DwarfDebug::emitMacroFile(DIMacroFile &F, DwarfCompileUnit &U) {
   assert(F.getMacinfoType() == dwarf::DW_MACINFO_start_file);
   Asm->EmitULEB128(dwarf::DW_MACINFO_start_file);
   Asm->EmitULEB128(F.getLine());
   Asm->EmitULEB128(U.getOrCreateSourceID(F.getFile()));
   handleMacroNodes(F.getElements(), U);
   Asm->EmitULEB128(dwarf::DW_MACINFO_end_file);
 }
 
 /// Emit macros into a debug macinfo section.
 void DwarfDebug::emitDebugMacinfo() {
   for (const auto &P : CUMap) {
     auto &TheCU = *P.second;
     auto *SkCU = TheCU.getSkeleton();
     DwarfCompileUnit &U = SkCU ? *SkCU : TheCU;
     auto *CUNode = cast<DICompileUnit>(P.first);
     DIMacroNodeArray Macros = CUNode->getMacros();
     if (Macros.empty())
       continue;
     Asm->OutStreamer->SwitchSection(
         Asm->getObjFileLowering().getDwarfMacinfoSection());
     Asm->OutStreamer->EmitLabel(U.getMacroLabelBegin());
     handleMacroNodes(Macros, U);
     Asm->OutStreamer->AddComment("End Of Macro List Mark");
     Asm->emitInt8(0);
   }
 }
 
+void DwarfDebug::emitDebugMacinfoDWO() {
+  for (const auto &P : CUMap) {
+    auto &TheCU = *P.second;
+    auto *SkCU = TheCU.getSkeleton();
+    DwarfCompileUnit &U = SkCU ? *SkCU : TheCU;
+    auto *CUNode = cast<DICompileUnit>(P.first);
+    DIMacroNodeArray Macros = CUNode->getMacros();
+    if (Macros.empty())
+      continue;
+    Asm->OutStreamer->SwitchSection(
+        Asm->getObjFileLowering().getDwarfMacinfoDWOSection());
+    Asm->OutStreamer->EmitLabel(U.getMacroLabelBegin());
+    handleMacroNodes(Macros, U);
+    Asm->OutStreamer->AddComment("End Of Macro List Mark");
+    Asm->emitInt8(0);
+  }
+}
+
 // DWARF5 Experimental Separate Dwarf emitters.
 
 void DwarfDebug::initSkeletonUnit(const DwarfUnit &U, DIE &Die,
                                   std::unique_ptr<DwarfCompileUnit> NewU) {
 
   if (!CompilationDir.empty())
     NewU->addString(Die, dwarf::DW_AT_comp_dir, CompilationDir);
 

llvm/lib/DebugInfo/DWARF/DWARFContext.cpp

void DWARFContext::dump(
   if (const auto *Off = shouldDump(Explicit, ".eh_frame", DIDT_ID_DebugFrame,
                                    DObj->getEHFrameSection().Data))
     getEHFrame()->dump(OS, getRegisterInfo(), *Off);
 
   if (DumpType & DIDT_DebugMacro) {
     if (Explicit || !getDebugMacro()->empty()) {
       OS << "\n.debug_macinfo contents:\n";
       getDebugMacro()->dump(OS);
+    } else if (ExplicitDWO || !getDebugMacroDWO()->empty()) {
+      OS << "\n.debug_macinfo.dwo contents:\n";
+      getDebugMacroDWO()->dump(OS);
     }
   }
 
   if (shouldDump(Explicit, ".debug_aranges", DIDT_ID_DebugAranges,
                  DObj->getArangesSection())) {
     uint64_t offset = 0;
     DataExtractor arangesData(DObj->getArangesSection(), isLittleEndian(), 0);
     DWARFDebugArangeSet set;
const DWARFDebugFrame *DWARFContext::getEHFrame() {
 
   DWARFDataExtractor debugFrameData(*DObj, DObj->getEHFrameSection(),
                                     isLittleEndian(), DObj->getAddressSize());
   DebugFrame.reset(new DWARFDebugFrame(getArch(), true /* IsEH */));
   DebugFrame->parse(debugFrameData);
   return DebugFrame.get();
 }
 
+const DWARFDebugMacro *DWARFContext::getDebugMacroDWO() {
+  if (MacroDWO)
+    return MacroDWO.get();
+
+  DataExtractor MacinfoDWOData(DObj->getMacinfoDWOSection(), isLittleEndian(),
+                               0);
+  MacroDWO.reset(new DWARFDebugMacro());
+  MacroDWO->parse(MacinfoDWOData);
+  return MacroDWO.get();
+}
+
 const DWARFDebugMacro *DWARFContext::getDebugMacro() {
   if (Macro)
     return Macro.get();
 
   DataExtractor MacinfoData(DObj->getMacinfoSection(), isLittleEndian(), 0);
   Macro.reset(new DWARFDebugMacro());
   Macro->parse(MacinfoData);
   return Macro.get();
return StringSwitch<DWARFSectionMap *>(Name)
         .Case("apple_objc", &AppleObjCSection)
         .Default(nullptr);
   }
 
   StringRef AbbrevSection;
   StringRef ArangesSection;
   StringRef StrSection;
   StringRef MacinfoSection;
+  StringRef MacinfoDWOSection;
   StringRef AbbrevDWOSection;
   StringRef StrDWOSection;
   StringRef CUIndexSection;
   StringRef GdbIndexSection;
   StringRef TUIndexSection;
   StringRef LineStrSection;
 
   // A deque holding section data whose iterators are not invalidated when
   // new decompressed sections are inserted at the end.
   std::deque<SmallString<0>> UncompressedSections;
 
   StringRef *mapSectionToMember(StringRef Name) {
     if (DWARFSection *Sec = mapNameToDWARFSection(Name))
       return &Sec->Data;
     return StringSwitch<StringRef *>(Name)
         .Case("debug_abbrev", &AbbrevSection)
         .Case("debug_aranges", &ArangesSection)
         .Case("debug_str", &StrSection)
         .Case("debug_macinfo", &MacinfoSection)
+        .Case("debug_macinfo.dwo", &MacinfoDWOSection)
         .Case("debug_abbrev.dwo", &AbbrevDWOSection)
         .Case("debug_str.dwo", &StrDWOSection)
         .Case("debug_cu_index", &CUIndexSection)
         .Case("debug_tu_index", &TUIndexSection)
         .Case("gdb_index", &GdbIndexSection)
         .Case("debug_line_str", &LineStrSection)
         // Any more debug info sections go here.
         .Default(nullptr);
public:
   }
   const DWARFSection &getLineSection() const override { return LineSection; }
   StringRef getStrSection() const override { return StrSection; }
   const DWARFSection &getRangesSection() const override { return RangesSection; }
   const DWARFSection &getRnglistsSection() const override {
     return RnglistsSection;
   }
   StringRef getMacinfoSection() const override { return MacinfoSection; }
+  StringRef getMacinfoDWOSection() const override { return MacinfoDWOSection; }
   const DWARFSection &getPubnamesSection() const override { return PubnamesSection; }
   const DWARFSection &getPubtypesSection() const override { return PubtypesSection; }
   const DWARFSection &getGnuPubnamesSection() const override {
     return GnuPubnamesSection;
   }
   const DWARFSection &getGnuPubtypesSection() const override {
     return GnuPubtypesSection;
   }

llvm/lib/MC/MCObjectFileInfo.cpp

void MCObjectFileInfo::initELFMCObjectFileInfo(const Triple &T, bool Large) {
   DwarfLineDWOSection =
       Ctx->getELFSection(".debug_line.dwo", DebugSecType, ELF::SHF_EXCLUDE);
   DwarfLocDWOSection =
       Ctx->getELFSection(".debug_loc.dwo", DebugSecType, ELF::SHF_EXCLUDE);
   DwarfStrOffDWOSection = Ctx->getELFSection(".debug_str_offsets.dwo",
                                              DebugSecType, ELF::SHF_EXCLUDE);
   DwarfRnglistsDWOSection =
       Ctx->getELFSection(".debug_rnglists.dwo", DebugSecType, ELF::SHF_EXCLUDE);
+  DwarfMacinfoDWOSection =
+      Ctx->getELFSection(".debug_macinfo.dwo", DebugSecType, ELF::SHF_EXCLUDE);
 
   DwarfLoclistsDWOSection =
       Ctx->getELFSection(".debug_loclists.dwo", DebugSecType, ELF::SHF_EXCLUDE);
 
   // DWP Sections
   DwarfCUIndexSection =
       Ctx->getELFSection(".debug_cu_index", DebugSecType, 0);
   DwarfTUIndexSection =

llvm/test/DebugInfo/debugmacinfo-dwo.test

+RUN: llvm-dwarfdump -debug-macro %p/Inputs/dwarfdump-macro.dwo \
+RUN:   | FileCheck %s -check-prefix TEST_MACINFODWO
+
+; This test verifies that llvm-dwarfdump tools know how to read .debug_macinfo.dwo
+; section.
+; dwarfdump-macro.dwo has been generated from Inputs/dwarfdump-macro.cc
+; clang++ -c -O0 -DM3=Value3 -include dwarfdump-macro-cmd.h dwarfdump-macro.cc -fdebug-macro -gsplit-dwarf
+
+TEST_MACINFODWO: .debug_macinfo.dwo contents:
+TEST_MACINFODWO: DW_MACINFO_start_file - lineno: 0 filenum: 1
+TEST_MACINFODWO:   DW_MACINFO_start_file - lineno: 0 filenum: 2
+TEST_MACINFODWO:     DW_MACINFO_define - lineno: 1 macro: M4 Value4
+TEST_MACINFODWO:   DW_MACINFO_end_file
+TEST_MACINFODWO:   DW_MACINFO_define - lineno: 1 macro: M1 Value1
+TEST_MACINFODWO:   DW_MACINFO_start_file - lineno: 2 filenum: 3
+TEST_MACINFODWO:     DW_MACINFO_undef - lineno: 4 macro: M1
+TEST_MACINFODWO:     DW_MACINFO_define - lineno: 5 macro: M1 NewValue1
+TEST_MACINFODWO:   DW_MACINFO_end_file
+TEST_MACINFODWO:   DW_MACINFO_define - lineno: 3 macro: M2(x,y) ((x)+(y)* Value2)
+TEST_MACINFODWO: DW_MACINFO_end_file