[DWARF] [ObjectYAML] Adding APIs for unittesting

[beanz]

Summary

This patch adds some new APIs to enable using the YAML DWARF representation in unit tests. The most basic new API is DWARFYAML::EmitDebugSections which converts a YAML string into a series of owned MemoryBuffer objects stored in a StringMap. The string map can then be used to construct a DWARFContext for parsing in place of an ObjectFile.

[clayborg]

Looks fine to me. I'll let David chime in though just to be sure.

[dblaikie]

Looks neat - few ideas/questions.

lib/DebugInfo/DWARF/DWARFContext.cpp

819–851

Looks like a duplicate of the StringSwitch in the other ctor - could be refactored to a common utility function?

lib/ObjectYAML/DWARFEmitter.cpp

356–358

Should this perhaps return ErrorOr<StringMap<...>> & so the error can be propagated/reported?

360–369

These are the supported sections at the moment - but I guess it'll grow support over time? Any way to refactor this so it goes along with the support as it's added? (is there similar code elsewhere already doing something like this to generate all the outputs in yaml2obj?)

unittests/DebugInfo/DWARF/DWARFDebugInfoTest.cpp

1163–1177

Are raw string literals an option here/yet? (are they compatible with all LLVM's supported compilers)

[beanz]

lib/DebugInfo/DWARF/DWARFContext.cpp

819–851

Yep, will do.

lib/ObjectYAML/DWARFEmitter.cpp

356–358

Will do.

360–369

Yes, I expect this to grow over time. Because the order and way in which debug sections are emitted can vary between object files there is no duplication of this in yaml2obj that would make sense to remove.

[beanz]

unittests/DebugInfo/DWARF/DWARFDebugInfoTest.cpp

1163–1177

I looked at this. Using multi-line raw syntax is, IMO, undesirable for encoded yaml because yaml is whitespace-sensitive. Using the non-raw syntax the strings in the code can be formatted with clang-format and still look like well-formed yaml. With the raw syntax the indentation gets screwy.

[beanz]

Updates based on feedback from dblaikie.

[dblaikie]

Looks good - I might experiment with raw string literals, see how it looks/feels personally at some point. Reckon it'd be easier to maintain, at least. (not having to wrap every line in "" and newlines, etc.

lib/DebugInfo/DWARF/DWARFContext.cpp

652–653

You can roll the variable declaration into the 'if' here, if you like

819–820

And here

unittests/DebugInfo/DWARF/DWARFDebugInfoTest.cpp

1180

Probably ASSERT_TRUE - if there is an error and you continue after this, it'll assert later from the Error not being handled which will probably be confusing.

@lhames might have some ideas about how to handle this sort of thing - I think it came up before/in other bits of Greg's work, so might be nice to have a solid idiom of "expect this to not Error" and log useful stuff about the error if it does fail, etc.

1182

Can write this as '*ErrOrSections' if you like

[dblaikie]

lib/DebugInfo/DWARF/DWARFContext.cpp

652–685

Did this regress when you committed it (if I'm reading the patch/changes/Phabricator correctly) - looks like the StringSwitch came back, rather than using the common utility function, MapSectionToMember.

unittests/DebugInfo/DWARF/DWARFDebugInfoTest.cpp

1185

This should probably be an ASSERT - otherwise it'll crash/UB on the next line anyway.

Did you chat to Lang about the fact that this would produce a follow-on assert if it failed? (since the code doesn't handle the error, if there is one)

Perhaps something like HandleExpectedError would be suitable/necessary here.

1186

Could write this as **Obj, if you like.

[beanz]

lib/DebugInfo/DWARF/DWARFContext.cpp

652–685

I think the diff just shows up strange. There is only one copy of the StringSwitch in the code I see checked in.

unittests/DebugInfo/DWARF/DWARFDebugInfoTest.cpp

1185

Sorry, forgot to update this. Lang is OOO and not easily reachable. I'll chat with him when he gets back in the meantime I'll update to ASSERT and commit.

include/llvm/DebugInfo/DWARF/DWARFContext.h

 #ifndef LLVM_LIB_DEBUGINFO_DWARFCONTEXT_H
 #define LLVM_LIB_DEBUGINFO_DWARFCONTEXT_H
 
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/iterator_range.h"
 #include "llvm/ADT/MapVector.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringMap.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/DebugInfo/DIContext.h"
 #include "llvm/DebugInfo/DWARF/DWARFCompileUnit.h"
 #include "llvm/DebugInfo/DWARF/DWARFDebugAbbrev.h"
 #include "llvm/DebugInfo/DWARF/DWARFDebugAranges.h"
 #include "llvm/DebugInfo/DWARF/DWARFDebugFrame.h"
 #include "llvm/DebugInfo/DWARF/DWARFDebugLine.h"
 #include "llvm/DebugInfo/DWARF/DWARFDebugLoc.h"
class DWARFContextInMemory : public DWARFContext {
   StringRef TUIndexSection;
 
   SmallVector<SmallString<32>, 4> UncompressedSections;
 
 public:
   DWARFContextInMemory(const object::ObjectFile &Obj,
     const LoadedObjectInfo *L = nullptr);
 
+  DWARFContextInMemory(const StringMap<std::unique_ptr<MemoryBuffer>> &Sections,
+                       uint8_t AddrSize,
+                       bool isLittleEndian = sys::IsLittleEndianHost);
+
   bool isLittleEndian() const override { return IsLittleEndian; }
   uint8_t getAddressSize() const override { return AddressSize; }
   const DWARFSection &getInfoSection() override { return InfoSection; }
   const TypeSectionMap &getTypesSections() override { return TypesSections; }
   StringRef getAbbrevSection() override { return AbbrevSection; }
   const DWARFSection &getLocSection() override { return LocSection; }
   StringRef getARangeSection() override { return ARangeSection; }
   StringRef getDebugFrameSection() override { return DebugFrameSection; }

include/llvm/ObjectYAML/DWARFEmitter.h

 //===--- DWARFEmitter.h - -------------------------------------------*- C++
 //-*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 /// \file
 /// \brief Common declarations for yaml2obj
 //===----------------------------------------------------------------------===//
 #ifndef LLVM_OBJECTYAML_DWARFEMITTER_H
 #define LLVM_OBJECTYAML_DWARFEMITTER_H
 
+#include "llvm/ADT/StringMap.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/Host.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include <memory>
+#include <vector>
+
 namespace llvm {
 class raw_ostream;
 
 namespace DWARFYAML {
 struct Data;
 struct PubSection;
 
 void EmitDebugAbbrev(llvm::raw_ostream &OS, const llvm::DWARFYAML::Data &DI);
 void EmitDebugStr(llvm::raw_ostream &OS, const llvm::DWARFYAML::Data &DI);
 
 void EmitDebugAranges(llvm::raw_ostream &OS, const llvm::DWARFYAML::Data &DI);
 void EmitPubSection(llvm::raw_ostream &OS,
                     const llvm::DWARFYAML::PubSection &Sect,
                     bool IsLittleEndian);
 void EmitDebugInfo(llvm::raw_ostream &OS, const llvm::DWARFYAML::Data &DI);
 void EmitDebugLine(llvm::raw_ostream &OS, const llvm::DWARFYAML::Data &DI);
 
+StringMap<std::unique_ptr<MemoryBuffer>>
+EmitDebugSections(StringRef YAMLString,
+                  bool IsLittleEndian = sys::IsLittleEndianHost);
+
 } // namespace DWARFYAML
 } // namespace llvm
 
 #endif

lib/DebugInfo/DWARF/DWARFContext.cpp

StringRef *SectionData =
             .Case("apple_types", &AppleTypesSection.Data)
             .Case("apple_namespaces", &AppleNamespacesSection.Data)
             .Case("apple_namespac", &AppleNamespacesSection.Data)
             .Case("apple_objc", &AppleObjCSection.Data)
             .Case("debug_cu_index", &CUIndexSection)
             .Case("debug_tu_index", &TUIndexSection)
             .Case("gdb_index", &GdbIndexSection)
             // Any more debug info sections go here.
             .Default(nullptr);
     if (SectionData) {
       *SectionData = data;
       if (name == "debug_ranges") {
         // FIXME: Use the other dwo range section when we emit it.
         RangeDWOSection = data;
       }
     } else if (name == "debug_types") {
       // Find debug_types data by section rather than name as there are
       // multiple, comdat grouped, debug_types sections.
       TypesSections[Section].Data = data;
     } else if (name == "debug_types.dwo") {
       TypesDWOSections[Section].Data = data;
     }
 
     if (RelocatedSection == Obj.section_end())
       continue;
 
     StringRef RelSecName;
     StringRef RelSecData;
     RelocatedSection->getName(RelSecName);
 
     // If the section we're relocating was relocated already by the JIT,
     // then we used the relocated version above, so we do not need to process
     // relocations for it now.
     if (L && L->getLoadedSectionContents(*RelocatedSection,RelSecData))
       continue;
 
     // In Mach-o files, the relocations do not need to be applied if
     // there is no load offset to apply. The value read at the
     // relocation point already factors in the section address
     // (actually applying the relocations will produce wrong results
     // as the section address will be added twice).
     if (!L && isa<MachOObjectFile>(&Obj))
       continue;
 
     RelSecName = RelSecName.substr(
         RelSecName.find_first_not_of("._")); // Skip . and _ prefixes.
 
     // TODO: Add support for relocations in other sections as needed.
     // Record relocations for the debug_info and debug_line sections.
     RelocAddrMap *Map = StringSwitch<RelocAddrMap*>(RelSecName)
//                        << ", Section: " << Section.getRawDataRefImpl() << "\n";
                      << " with width " << format("%d", R.Width)
                      << "\n");
         Map->insert(std::make_pair(Address, std::make_pair(R.Width, R.Value)));
       }
     }
   }
 }
 
+DWARFContextInMemory::DWARFContextInMemory(
+    const StringMap<std::unique_ptr<MemoryBuffer>> &Sections, uint8_t AddrSize,
+    bool isLittleEndian)
+    : IsLittleEndian(isLittleEndian), AddressSize(AddrSize) {
+  for (const auto &SecIt : Sections) {
+    StringRef *SectionData =
+        StringSwitch<StringRef *>(SecIt.first())
+            .Case("debug_info", &InfoSection.Data)
+            .Case("debug_abbrev", &AbbrevSection)
+            .Case("debug_loc", &LocSection.Data)
+            .Case("debug_line", &LineSection.Data)
+            .Case("debug_aranges", &ARangeSection)
+            .Case("debug_frame", &DebugFrameSection)
+            .Case("eh_frame", &EHFrameSection)
+            .Case("debug_str", &StringSection)
+            .Case("debug_ranges", &RangeSection)
+            .Case("debug_macinfo", &MacinfoSection)
+            .Case("debug_pubnames", &PubNamesSection)
+            .Case("debug_pubtypes", &PubTypesSection)
+            .Case("debug_gnu_pubnames", &GnuPubNamesSection)
+            .Case("debug_gnu_pubtypes", &GnuPubTypesSection)
+            .Case("debug_info.dwo", &InfoDWOSection.Data)
+            .Case("debug_abbrev.dwo", &AbbrevDWOSection)
+            .Case("debug_loc.dwo", &LocDWOSection.Data)
+            .Case("debug_line.dwo", &LineDWOSection.Data)
+            .Case("debug_str.dwo", &StringDWOSection)
+            .Case("debug_str_offsets.dwo", &StringOffsetDWOSection)
+            .Case("debug_addr", &AddrSection)
+            .Case("apple_names", &AppleNamesSection.Data)
+            .Case("apple_types", &AppleTypesSection.Data)
+            .Case("apple_namespaces", &AppleNamespacesSection.Data)
+            .Case("apple_namespac", &AppleNamespacesSection.Data)
+            .Case("apple_objc", &AppleObjCSection.Data)
+            .Case("debug_cu_index", &CUIndexSection)
+            .Case("debug_tu_index", &TUIndexSection)
+            .Case("gdb_index", &GdbIndexSection)
+            // Any more debug info sections go here.
+            .Default(nullptr);
+    if (SectionData)
+      *SectionData = SecIt.second->getBuffer();
+  }
+}
+
 void DWARFContextInMemory::anchor() { }

lib/ObjectYAML/DWARFEmitter.cpp

for (auto Op : LineTable.Opcodes) {
           for (auto OpData : Op.StandardOpcodeData) {
             encodeULEB128(OpData, OS);
           }
         }
       }
     }
   }
 }
+
+typedef void (*EmitFuncType)(raw_ostream &, const DWARFYAML::Data &);
+
+void EmitDebugSectionImpl(
+    const DWARFYAML::Data &DI, EmitFuncType EmitFunc, StringRef Sec,
+    StringMap<std::unique_ptr<MemoryBuffer>> &OutputBuffers) {
+  std::string Data;
+  raw_string_ostream DebugInfoStream(Data);
+  EmitFunc(DebugInfoStream, DI);
+  DebugInfoStream.flush();
+  if (!Data.empty())
+    OutputBuffers[Sec] = MemoryBuffer::getMemBufferCopy(Data);
+}
+
+StringMap<std::unique_ptr<MemoryBuffer>>
+DWARFYAML::EmitDebugSections(StringRef YAMLString,
+                             bool IsLittleEndian) {
+  StringMap<std::unique_ptr<MemoryBuffer>> DebugSections;
+
+  yaml::Input YIn(YAMLString);
+
+  DWARFYAML::Data DI;
+  DI.IsLittleEndian = IsLittleEndian;
+  YIn >> DI;
+  if (YIn.error())
+    return DebugSections;
+
+  EmitDebugSectionImpl(DI, &DWARFYAML::EmitDebugInfo, "debug_info",
+                       DebugSections);
+  EmitDebugSectionImpl(DI, &DWARFYAML::EmitDebugLine, "debug_line",
+                       DebugSections);
+  EmitDebugSectionImpl(DI, &DWARFYAML::EmitDebugStr, "debug_str",
+                       DebugSections);
+  EmitDebugSectionImpl(DI, &DWARFYAML::EmitDebugAbbrev, "debug_abbrev",
+                       DebugSections);
+  EmitDebugSectionImpl(DI, &DWARFYAML::EmitDebugAranges, "debug_aranges",
+                       DebugSections);
+  return DebugSections;
+}

unittests/DebugInfo/DWARF/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   ${LLVM_TARGETS_TO_BUILD}
   AsmPrinter
   DebugInfoDWARF
   MC
   Object
+  ObjectYAML
   Support
   )
 
 set(DebugInfoSources
   DwarfGenerator.cpp
   DWARFDebugInfoTest.cpp
   DWARFFormValueTest.cpp
   )
 
 add_llvm_unittest(DebugInfoDWARFTests
   ${DebugInfoSources}
   )

unittests/DebugInfo/DWARF/DWARFDebugInfoTest.cpp

 //===- llvm/unittest/DebugInfo/DWARFFormValueTest.cpp ---------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "DwarfGenerator.h"
 #include "llvm/DebugInfo/DWARF/DWARFAbbreviationDeclaration.h"
 #include "llvm/DebugInfo/DWARF/DWARFContext.h"
 #include "llvm/DebugInfo/DWARF/DWARFDie.h"
 #include "llvm/DebugInfo/DWARF/DWARFFormValue.h"
 #include "llvm/DebugInfo/DWARF/DWARFUnit.h"
+#include "llvm/ObjectYAML/DWARFYAML.h"
+#include "llvm/ObjectYAML/DWARFEmitter.h"
 #include "llvm/Support/Dwarf.h"
 #include "llvm/Support/Host.h"
 #include "llvm/Support/TargetSelect.h"
 #include "gtest/gtest.h"
 #include <climits>
 
 using namespace llvm;
 using namespace dwarf;
TEST(DWARFDebugInfo, TestChildIteratorsOnInvalidDie) {
   DWARFDie Invalid;
   auto begin = Invalid.begin();
   auto end = Invalid.end();
   EXPECT_FALSE(begin->isValid());
   EXPECT_FALSE(end->isValid());
   EXPECT_EQ(begin, end);
 }
 
-  
 TEST(DWARFDebugInfo, TestEmptyChildren) {
-  // Test a DIE that says it has children in the abbreviation, but actually
-  // doesn't have any attributes, will not return anything during iteration.
-  // We do this by making sure the begin and end iterators are equal.
-  uint16_t Version = 4;
-  
-  const uint8_t AddrSize = sizeof(void *);
-  initLLVMIfNeeded();
-  Triple Triple = getHostTripleForAddrSize(AddrSize);
-  auto ExpectedDG = dwarfgen::Generator::create(Triple, Version);
-  if (HandleExpectedError(ExpectedDG))
-    return;
-  dwarfgen::Generator *DG = ExpectedDG.get().get();
-  dwarfgen::CompileUnit &CU = DG->addCompileUnit();
-  
-  // Scope to allow us to re-use the same DIE names
-  {
-    // Create a compile unit DIE that has an abbreviation that says it has
-    // children, but doesn't have any actual attributes. This helps us test
-    // a DIE that has only one child: a NULL DIE.
-    auto CUDie = CU.getUnitDIE();
-    CUDie.setForceChildren();
-  }
-  
-  MemoryBufferRef FileBuffer(DG->generate(), "dwarf");
-  auto Obj = object::ObjectFile::createObjectFile(FileBuffer);
-  EXPECT_TRUE((bool)Obj);
-  DWARFContextInMemory DwarfContext(*Obj.get());
-  
+  const char *yamldata = "debug_abbrev:\n"
+                         "  - Code:            0x00000001\n"
+                         "    Tag:             DW_TAG_compile_unit\n"
+                         "    Children:        DW_CHILDREN_yes\n"
+                         "    Attributes:\n"
+                         "debug_info:\n"
+                         "  - Length:          9\n"
+                         "    Version:         4\n"
+                         "    AbbrOffset:      0\n"
+                         "    AddrSize:        8\n"
+                         "    Entries:\n"
+                         "      - AbbrCode:        0x00000001\n"
+                         "        Values:\n"
+                         "      - AbbrCode:        0x00000000\n"
+                         "        Values:\n";
+  StringMap<std::unique_ptr<MemoryBuffer>> DebugSections =
+      DWARFYAML::EmitDebugSections(StringRef(yamldata));
+  DWARFContextInMemory DwarfContext(DebugSections, 8);
   // Verify the number of compile units is correct.
   uint32_t NumCUs = DwarfContext.getNumCompileUnits();
   EXPECT_EQ(NumCUs, 1u);
   DWARFCompileUnit *U = DwarfContext.getCompileUnitAtIndex(0);
-  
   // Get the compile unit DIE is valid.
   auto CUDie = U->getUnitDIE(false);
   EXPECT_TRUE(CUDie.isValid());
   
   // Verify that the CU Die that says it has children, but doesn't, actually
   // has begin and end iterators that are equal. We want to make sure we don't
   // see the Null DIEs during iteration.
   EXPECT_EQ(CUDie.begin(), CUDie.end());