[DWARF] [ObjectYAML] Adding APIs for unittesting

[beanz]

Summary

This patch adds some new APIs to enable using the YAML DWARF representation in unit tests. The most basic new API is DWARFYAML::EmitDebugSections which converts a YAML string into a series of owned MemoryBuffer objects stored in a StringMap. The string map can then be used to construct a DWARFContext for parsing in place of an ObjectFile.

[clayborg]

Looks fine to me. I'll let David chime in though just to be sure.

[dblaikie]

Looks neat - few ideas/questions.

lib/DebugInfo/DWARF/DWARFContext.cpp

819–851 ↗

(On Diff #84747)

Looks like a duplicate of the StringSwitch in the other ctor - could be refactored to a common utility function?

lib/ObjectYAML/DWARFEmitter.cpp

356–358 ↗

(On Diff #84747)

Should this perhaps return ErrorOr<StringMap<...>> & so the error can be propagated/reported?

360–369 ↗

(On Diff #84747)

These are the supported sections at the moment - but I guess it'll grow support over time? Any way to refactor this so it goes along with the support as it's added? (is there similar code elsewhere already doing something like this to generate all the outputs in yaml2obj?)

unittests/DebugInfo/DWARF/DWARFDebugInfoTest.cpp

1163–1177 ↗

(On Diff #84747)

Are raw string literals an option here/yet? (are they compatible with all LLVM's supported compilers)

[beanz]

lib/DebugInfo/DWARF/DWARFContext.cpp

819–851 ↗

(On Diff #84747)

Yep, will do.

lib/ObjectYAML/DWARFEmitter.cpp

356–358 ↗

(On Diff #84747)

Will do.

360–369 ↗

(On Diff #84747)

Yes, I expect this to grow over time. Because the order and way in which debug sections are emitted can vary between object files there is no duplication of this in yaml2obj that would make sense to remove.

[beanz]

unittests/DebugInfo/DWARF/DWARFDebugInfoTest.cpp

1163–1177 ↗

(On Diff #84747)

I looked at this. Using multi-line raw syntax is, IMO, undesirable for encoded yaml because yaml is whitespace-sensitive. Using the non-raw syntax the strings in the code can be formatted with clang-format and still look like well-formed yaml. With the raw syntax the indentation gets screwy.

[beanz]

Updates based on feedback from dblaikie.

[dblaikie]

Looks good - I might experiment with raw string literals, see how it looks/feels personally at some point. Reckon it'd be easier to maintain, at least. (not having to wrap every line in "" and newlines, etc.

lib/DebugInfo/DWARF/DWARFContext.cpp

620–621 ↗

(On Diff #84893)

You can roll the variable declaration into the 'if' here, if you like

787–788 ↗

(On Diff #84893)

And here

unittests/DebugInfo/DWARF/DWARFDebugInfoTest.cpp

1180 ↗

(On Diff #84893)

Probably ASSERT_TRUE - if there is an error and you continue after this, it'll assert later from the Error not being handled which will probably be confusing.

@lhames might have some ideas about how to handle this sort of thing - I think it came up before/in other bits of Greg's work, so might be nice to have a solid idiom of "expect this to not Error" and log useful stuff about the error if it does fail, etc.

1182 ↗

(On Diff #84893)

Can write this as '*ErrOrSections' if you like

[dblaikie]

lib/DebugInfo/DWARF/DWARFContext.cpp

620–653 ↗

(On Diff #84893)

Did this regress when you committed it (if I'm reading the patch/changes/Phabricator correctly) - looks like the StringSwitch came back, rather than using the common utility function, MapSectionToMember.

unittests/DebugInfo/DWARF/DWARFDebugInfoTest.cpp

1187 ↗

(On Diff #84893)

This should probably be an ASSERT - otherwise it'll crash/UB on the next line anyway.

Did you chat to Lang about the fact that this would produce a follow-on assert if it failed? (since the code doesn't handle the error, if there is one)

Perhaps something like HandleExpectedError would be suitable/necessary here.

1188 ↗

(On Diff #84893)

Could write this as **Obj, if you like.

[beanz]

lib/DebugInfo/DWARF/DWARFContext.cpp

620–653 ↗

(On Diff #84893)

I think the diff just shows up strange. There is only one copy of the StringSwitch in the code I see checked in.

unittests/DebugInfo/DWARF/DWARFDebugInfoTest.cpp

1187 ↗

(On Diff #84893)

Sorry, forgot to update this. Lang is OOO and not easily reachable. I'll chat with him when he gets back in the meantime I'll update to ASSERT and commit.

llvm/trunk/include/llvm/DebugInfo/DWARF/DWARFContext.h

 #ifndef LLVM_LIB_DEBUGINFO_DWARFCONTEXT_H
 #define LLVM_LIB_DEBUGINFO_DWARFCONTEXT_H
 
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/iterator_range.h"
 #include "llvm/ADT/MapVector.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringMap.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/DebugInfo/DIContext.h"
 #include "llvm/DebugInfo/DWARF/DWARFCompileUnit.h"
 #include "llvm/DebugInfo/DWARF/DWARFDebugAbbrev.h"
 #include "llvm/DebugInfo/DWARF/DWARFDebugAranges.h"
 #include "llvm/DebugInfo/DWARF/DWARFDebugFrame.h"
 #include "llvm/DebugInfo/DWARF/DWARFDebugLine.h"
 #include "llvm/DebugInfo/DWARF/DWARFDebugLoc.h"
class DWARFContextInMemory : public DWARFContext {
   DWARFSection AppleNamespacesSection;
   DWARFSection AppleObjCSection;
   StringRef CUIndexSection;
   StringRef GdbIndexSection;
   StringRef TUIndexSection;
 
   SmallVector<SmallString<32>, 4> UncompressedSections;
 
+  StringRef *MapSectionToMember(StringRef Name);
+
 public:
   DWARFContextInMemory(const object::ObjectFile &Obj,
     const LoadedObjectInfo *L = nullptr);
 
+  DWARFContextInMemory(const StringMap<std::unique_ptr<MemoryBuffer>> &Sections,
+                       uint8_t AddrSize,
+                       bool isLittleEndian = sys::IsLittleEndianHost);
+
   bool isLittleEndian() const override { return IsLittleEndian; }
   uint8_t getAddressSize() const override { return AddressSize; }
   const DWARFSection &getInfoSection() override { return InfoSection; }
   const TypeSectionMap &getTypesSections() override { return TypesSections; }
   StringRef getAbbrevSection() override { return AbbrevSection; }
   const DWARFSection &getLocSection() override { return LocSection; }
   StringRef getARangeSection() override { return ARangeSection; }
   StringRef getDebugFrameSection() override { return DebugFrameSection; }

llvm/trunk/include/llvm/ObjectYAML/DWARFEmitter.h

 //===--- DWARFEmitter.h - -------------------------------------------*- C++
 //-*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 /// \file
 /// \brief Common declarations for yaml2obj
 //===----------------------------------------------------------------------===//
 #ifndef LLVM_OBJECTYAML_DWARFEMITTER_H
 #define LLVM_OBJECTYAML_DWARFEMITTER_H
 
+#include "llvm/ADT/StringMap.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/Host.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include <memory>
+#include <vector>
+
 namespace llvm {
 class raw_ostream;
 
 namespace DWARFYAML {
 struct Data;
 struct PubSection;
 
 void EmitDebugAbbrev(llvm::raw_ostream &OS, const llvm::DWARFYAML::Data &DI);
 void EmitDebugStr(llvm::raw_ostream &OS, const llvm::DWARFYAML::Data &DI);
 
 void EmitDebugAranges(llvm::raw_ostream &OS, const llvm::DWARFYAML::Data &DI);
 void EmitPubSection(llvm::raw_ostream &OS,
                     const llvm::DWARFYAML::PubSection &Sect,
                     bool IsLittleEndian);
 void EmitDebugInfo(llvm::raw_ostream &OS, const llvm::DWARFYAML::Data &DI);
 void EmitDebugLine(llvm::raw_ostream &OS, const llvm::DWARFYAML::Data &DI);
 
+Expected<StringMap<std::unique_ptr<MemoryBuffer>>>
+EmitDebugSections(StringRef YAMLString,
+                  bool IsLittleEndian = sys::IsLittleEndianHost);
+
 } // namespace DWARFYAML
 } // namespace llvm
 
 #endif

llvm/trunk/lib/DebugInfo/DWARF/DWARFContext.cpp

if (Decompressor::isCompressed(Section)) {
       data = UncompressedSections.back();
     }
 
     // Compressed sections names in GNU style starts from ".z",
     // at this point section is decompressed and we drop compression prefix.
     name = name.substr(
         name.find_first_not_of("._z")); // Skip ".", "z" and "_" prefixes.
 
-    StringRef *SectionData =
-        StringSwitch<StringRef *>(name)
-            .Case("debug_info", &InfoSection.Data)
-            .Case("debug_abbrev", &AbbrevSection)
-            .Case("debug_loc", &LocSection.Data)
-            .Case("debug_line", &LineSection.Data)
-            .Case("debug_aranges", &ARangeSection)
-            .Case("debug_frame", &DebugFrameSection)
-            .Case("eh_frame", &EHFrameSection)
-            .Case("debug_str", &StringSection)
-            .Case("debug_ranges", &RangeSection)
-            .Case("debug_macinfo", &MacinfoSection)
-            .Case("debug_pubnames", &PubNamesSection)
-            .Case("debug_pubtypes", &PubTypesSection)
-            .Case("debug_gnu_pubnames", &GnuPubNamesSection)
-            .Case("debug_gnu_pubtypes", &GnuPubTypesSection)
-            .Case("debug_info.dwo", &InfoDWOSection.Data)
-            .Case("debug_abbrev.dwo", &AbbrevDWOSection)
-            .Case("debug_loc.dwo", &LocDWOSection.Data)
-            .Case("debug_line.dwo", &LineDWOSection.Data)
-            .Case("debug_str.dwo", &StringDWOSection)
-            .Case("debug_str_offsets.dwo", &StringOffsetDWOSection)
-            .Case("debug_addr", &AddrSection)
-            .Case("apple_names", &AppleNamesSection.Data)
-            .Case("apple_types", &AppleTypesSection.Data)
-            .Case("apple_namespaces", &AppleNamespacesSection.Data)
-            .Case("apple_namespac", &AppleNamespacesSection.Data)
-            .Case("apple_objc", &AppleObjCSection.Data)
-            .Case("debug_cu_index", &CUIndexSection)
-            .Case("debug_tu_index", &TUIndexSection)
-            .Case("gdb_index", &GdbIndexSection)
-            // Any more debug info sections go here.
-            .Default(nullptr);
-    if (SectionData) {
+    if (StringRef *SectionData = MapSectionToMember(name)) {
       *SectionData = data;
       if (name == "debug_ranges") {
         // FIXME: Use the other dwo range section when we emit it.
         RangeDWOSection = data;
       }
     } else if (name == "debug_types") {
       // Find debug_types data by section rather than name as there are
       // multiple, comdat grouped, debug_types sections.
//                        << ", Section: " << Section.getRawDataRefImpl() << "\n";
                      << " with width " << format("%d", R.Width)
                      << "\n");
         Map->insert(std::make_pair(Address, std::make_pair(R.Width, R.Value)));
       }
     }
   }
 }
 
+DWARFContextInMemory::DWARFContextInMemory(
+    const StringMap<std::unique_ptr<MemoryBuffer>> &Sections, uint8_t AddrSize,
+    bool isLittleEndian)
+    : IsLittleEndian(isLittleEndian), AddressSize(AddrSize) {
+  for (const auto &SecIt : Sections) {
+    if (StringRef *SectionData = MapSectionToMember(SecIt.first()))
+      *SectionData = SecIt.second->getBuffer();
+  }
+}
+
+StringRef *DWARFContextInMemory::MapSectionToMember(StringRef Name) {
+  return StringSwitch<StringRef *>(Name)
+      .Case("debug_info", &InfoSection.Data)
+      .Case("debug_abbrev", &AbbrevSection)
+      .Case("debug_loc", &LocSection.Data)
+      .Case("debug_line", &LineSection.Data)
+      .Case("debug_aranges", &ARangeSection)
+      .Case("debug_frame", &DebugFrameSection)
+      .Case("eh_frame", &EHFrameSection)
+      .Case("debug_str", &StringSection)
+      .Case("debug_ranges", &RangeSection)
+      .Case("debug_macinfo", &MacinfoSection)
+      .Case("debug_pubnames", &PubNamesSection)
+      .Case("debug_pubtypes", &PubTypesSection)
+      .Case("debug_gnu_pubnames", &GnuPubNamesSection)
+      .Case("debug_gnu_pubtypes", &GnuPubTypesSection)
+      .Case("debug_info.dwo", &InfoDWOSection.Data)
+      .Case("debug_abbrev.dwo", &AbbrevDWOSection)
+      .Case("debug_loc.dwo", &LocDWOSection.Data)
+      .Case("debug_line.dwo", &LineDWOSection.Data)
+      .Case("debug_str.dwo", &StringDWOSection)
+      .Case("debug_str_offsets.dwo", &StringOffsetDWOSection)
+      .Case("debug_addr", &AddrSection)
+      .Case("apple_names", &AppleNamesSection.Data)
+      .Case("apple_types", &AppleTypesSection.Data)
+      .Case("apple_namespaces", &AppleNamespacesSection.Data)
+      .Case("apple_namespac", &AppleNamespacesSection.Data)
+      .Case("apple_objc", &AppleObjCSection.Data)
+      .Case("debug_cu_index", &CUIndexSection)
+      .Case("debug_tu_index", &TUIndexSection)
+      .Case("gdb_index", &GdbIndexSection)
+      // Any more debug info sections go here.
+      .Default(nullptr);
+}
+
 void DWARFContextInMemory::anchor() { }

llvm/trunk/lib/ObjectYAML/DWARFEmitter.cpp

for (auto Op : LineTable.Opcodes) {
           for (auto OpData : Op.StandardOpcodeData) {
             encodeULEB128(OpData, OS);
           }
         }
       }
     }
   }
 }
+
+typedef void (*EmitFuncType)(raw_ostream &, const DWARFYAML::Data &);
+
+void EmitDebugSectionImpl(
+    const DWARFYAML::Data &DI, EmitFuncType EmitFunc, StringRef Sec,
+    StringMap<std::unique_ptr<MemoryBuffer>> &OutputBuffers) {
+  std::string Data;
+  raw_string_ostream DebugInfoStream(Data);
+  EmitFunc(DebugInfoStream, DI);
+  DebugInfoStream.flush();
+  if (!Data.empty())
+    OutputBuffers[Sec] = MemoryBuffer::getMemBufferCopy(Data);
+}
+
+Expected<StringMap<std::unique_ptr<MemoryBuffer>>>
+DWARFYAML::EmitDebugSections(StringRef YAMLString,
+                             bool IsLittleEndian) {
+  StringMap<std::unique_ptr<MemoryBuffer>> DebugSections;
+
+  yaml::Input YIn(YAMLString);
+
+  DWARFYAML::Data DI;
+  DI.IsLittleEndian = IsLittleEndian;
+  YIn >> DI;
+  if (YIn.error())
+    return errorCodeToError(YIn.error());
+
+  EmitDebugSectionImpl(DI, &DWARFYAML::EmitDebugInfo, "debug_info",
+                       DebugSections);
+  EmitDebugSectionImpl(DI, &DWARFYAML::EmitDebugLine, "debug_line",
+                       DebugSections);
+  EmitDebugSectionImpl(DI, &DWARFYAML::EmitDebugStr, "debug_str",
+                       DebugSections);
+  EmitDebugSectionImpl(DI, &DWARFYAML::EmitDebugAbbrev, "debug_abbrev",
+                       DebugSections);
+  EmitDebugSectionImpl(DI, &DWARFYAML::EmitDebugAranges, "debug_aranges",
+                       DebugSections);
+  return std::move(DebugSections);
+}

llvm/trunk/unittests/DebugInfo/DWARF/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   ${LLVM_TARGETS_TO_BUILD}
   AsmPrinter
   DebugInfoDWARF
   MC
   Object
+  ObjectYAML
   Support
   )
 
 set(DebugInfoSources
   DwarfGenerator.cpp
   DWARFDebugInfoTest.cpp
   DWARFFormValueTest.cpp
   )
 
 add_llvm_unittest(DebugInfoDWARFTests
   ${DebugInfoSources}
   )

llvm/trunk/unittests/DebugInfo/DWARF/DWARFDebugInfoTest.cpp

 //===- llvm/unittest/DebugInfo/DWARFFormValueTest.cpp ---------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "DwarfGenerator.h"
 #include "llvm/DebugInfo/DWARF/DWARFAbbreviationDeclaration.h"
 #include "llvm/DebugInfo/DWARF/DWARFContext.h"
 #include "llvm/DebugInfo/DWARF/DWARFDie.h"
 #include "llvm/DebugInfo/DWARF/DWARFFormValue.h"
 #include "llvm/DebugInfo/DWARF/DWARFUnit.h"
+#include "llvm/ObjectYAML/DWARFYAML.h"
+#include "llvm/ObjectYAML/DWARFEmitter.h"
 #include "llvm/Support/Dwarf.h"
 #include "llvm/Support/Host.h"
 #include "llvm/Support/TargetSelect.h"
 #include "gtest/gtest.h"
 #include <climits>
 
 using namespace llvm;
 using namespace dwarf;
TEST(DWARFDebugInfo, TestChildIteratorsOnInvalidDie) {
   DWARFDie Invalid;
   auto begin = Invalid.begin();
   auto end = Invalid.end();
   EXPECT_FALSE(begin->isValid());
   EXPECT_FALSE(end->isValid());
   EXPECT_EQ(begin, end);
 }
 
-  
 TEST(DWARFDebugInfo, TestEmptyChildren) {
-  // Test a DIE that says it has children in the abbreviation, but actually
-  // doesn't have any attributes, will not return anything during iteration.
-  // We do this by making sure the begin and end iterators are equal.
-  uint16_t Version = 4;
-  
-  const uint8_t AddrSize = sizeof(void *);
-  initLLVMIfNeeded();
-  Triple Triple = getHostTripleForAddrSize(AddrSize);
-  auto ExpectedDG = dwarfgen::Generator::create(Triple, Version);
-  if (HandleExpectedError(ExpectedDG))
-    return;
-  dwarfgen::Generator *DG = ExpectedDG.get().get();
-  dwarfgen::CompileUnit &CU = DG->addCompileUnit();
-  
-  // Scope to allow us to re-use the same DIE names
-  {
-    // Create a compile unit DIE that has an abbreviation that says it has
-    // children, but doesn't have any actual attributes. This helps us test
-    // a DIE that has only one child: a NULL DIE.
-    auto CUDie = CU.getUnitDIE();
-    CUDie.setForceChildren();
-  }
-  
-  MemoryBufferRef FileBuffer(DG->generate(), "dwarf");
-  auto Obj = object::ObjectFile::createObjectFile(FileBuffer);
-  EXPECT_TRUE((bool)Obj);
-  DWARFContextInMemory DwarfContext(*Obj.get());
-  
+  const char *yamldata = "debug_abbrev:\n"
+                         "  - Code:            0x00000001\n"
+                         "    Tag:             DW_TAG_compile_unit\n"
+                         "    Children:        DW_CHILDREN_yes\n"
+                         "    Attributes:\n"
+                         "debug_info:\n"
+                         "  - Length:          9\n"
+                         "    Version:         4\n"
+                         "    AbbrOffset:      0\n"
+                         "    AddrSize:        8\n"
+                         "    Entries:\n"
+                         "      - AbbrCode:        0x00000001\n"
+                         "        Values:\n"
+                         "      - AbbrCode:        0x00000000\n"
+                         "        Values:\n";
+  auto ErrOrSections = DWARFYAML::EmitDebugSections(StringRef(yamldata));
+  EXPECT_TRUE((bool)ErrOrSections);
+  auto &DebugSections = *ErrOrSections;
+  DWARFContextInMemory DwarfContext(DebugSections, 8);
   // Verify the number of compile units is correct.
   uint32_t NumCUs = DwarfContext.getNumCompileUnits();
   EXPECT_EQ(NumCUs, 1u);
   DWARFCompileUnit *U = DwarfContext.getCompileUnitAtIndex(0);
-  
   // Get the compile unit DIE is valid.
   auto CUDie = U->getUnitDIE(false);
   EXPECT_TRUE(CUDie.isValid());
   
   // Verify that the CU Die that says it has children, but doesn't, actually
   // has begin and end iterators that are equal. We want to make sure we don't
   // see the Null DIEs during iteration.
   EXPECT_EQ(CUDie.begin(), CUDie.end());