[DWARF] Add an api to get "interpreted" location lists

[labath]

Summary

This patch adds DWARFDie::getLocations, which returns the location
expressions for a given attribute (typically DW_AT_location). It handles
both "inline" locations and references to the external location list
sections (currently only of the DW_FORM_sec_offset type). It is
implemented on top of DWARFUnit::findLoclistFromOffset, which is also
added in this patch. I tried to make their signatures similar to the
equivalent range list functionality.

The actual location list interpretation logic is in
DWARFLocationTable::visitAbsoluteLocationList. This part is not
equivalent to the range list code, but this deviation is motivated by a
desire to reuse the same location list parsing code within lldb.

The functionality is tested via a c++ unit test of the DWARFDie API.

[labath]

llvm/include/llvm/DebugInfo/DWARF/DWARFAddressRange.h

59 ↗

(On Diff #229832)

Ideally I'd have both this and the operator< above also compare the section index, but I did not want to do that in the same patch.

llvm/include/llvm/DebugInfo/DWARF/DWARFLocationExpression.h

32–37

These, and other random operators the patch is adding, are to make sure one can make sense of the gtest output when the tests fail.

llvm/lib/ObjectYAML/DWARFEmitter.cpp

318

This doesn't take type units into account, but then the whole of DWARFEmitter does not handle type units AFAICT.

llvm/test/tools/llvm-dwarfdump/X86/statistics-dwo.test

83–84 ↗

(On Diff #229832)

I haven't checked if these numbers are fully correct, but they should definitely be *more* correct then before, as the code previously did not handle split dwarf location lists at all.

[dblaikie]

llvm/include/llvm/DebugInfo/DWARF/DWARFAddressRange.h

52–61 ↗

(On Diff #229832)

Welcome to/probably best to commit changes like this by themselves and separate from this commit - makes the review easier without these orthogonal fixes, etc.

llvm/include/llvm/DebugInfo/DWARF/DWARFLocationExpression.h

32–38

Maybe introducing these separately with unit tests of their own would be good?

llvm/include/llvm/Object/ObjectFile.h

158–159 ↗

(On Diff #229832)

Separate+unit test, perhaps? (wouldn't bother sending this & the other for review - I imagine they'd be simple enough for review-after-commit)

llvm/lib/ObjectYAML/DWARFEmitter.cpp

318

Might be worth a comment or rephrasing to make the 8/7 values clear as to what they relate to/come from?

llvm/test/tools/llvm-dwarfdump/X86/statistics-dwo.test

83–84 ↗

(On Diff #229832)

Might be worth a manual check of the clang binary to see that the numbers remain the same in the cases where it's expected & perhaps that the numbers are the same for split and non-split builds if this change is expected to fix a bug where they were divergent?

[labath]

• committed operator<<s separately
• statistics stuff moved to a separate patch
• add DWARFLocationExpressionsVector typedef for more range list symmetry

llvm/test/tools/llvm-dwarfdump/X86/statistics-dwo.test

83–84 ↗

(On Diff #229832)

Closer examination of this code revealed more problems, so I am going to split the statistics stuff into a separate patch too.

[dblaikie]

llvm/unittests/DebugInfo/DWARF/DWARFDieTest.cpp

57–68

Maybe break this down a bit further by field? (0, 0, 0, 0 length \ 5 version, etc... including for the location expressions? Otherwise the strings of zeros especially (& random 2s?) are a bit opaque)

67

"intentionally missing end of list entry to test error handling" (maybe that's too verbose - but as it stands, just "no end of list" is a bit confusing to me - it's not clear why that's the case/being highlighted)

llvm/include/llvm/DebugInfo/DWARF/DWARFDebugLoc.h

public:
   /// iff it has successfully reched the end of the list. This means that one
   /// can attempt to parse another list after the current one (\p Offset will be
   /// updated to point past the end of the current list).
   bool dumpLocationList(uint64_t *Offset, raw_ostream &OS,
                         Optional<object::SectionedAddress> BaseAddr,
                         const MCRegisterInfo *MRI, DWARFUnit *U,
                         DIDumpOptions DumpOpts, unsigned Indent) const;
 
+  Error visitAbsoluteLocationList(
+      uint64_t Offset, Optional<object::SectionedAddress> BaseAddr,
+      std::function<Optional<object::SectionedAddress>(uint32_t)> LookupAddr,
+      function_ref<bool(Expected<DWARFLocationExpression>)> Callback) const;
+
 protected:
   DWARFDataExtractor Data;
 
   virtual void dumpRawEntry(const DWARFLocationEntry &Entry, raw_ostream &OS,
                             unsigned Indent) const = 0;
 };
 
 class DWARFDebugLoc final : public DWARFLocationTable {

llvm/include/llvm/DebugInfo/DWARF/DWARFDie.h

 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/iterator.h"
 #include "llvm/ADT/iterator_range.h"
 #include "llvm/BinaryFormat/Dwarf.h"
 #include "llvm/DebugInfo/DIContext.h"
 #include "llvm/DebugInfo/DWARF/DWARFAddressRange.h"
 #include "llvm/DebugInfo/DWARF/DWARFAttribute.h"
 #include "llvm/DebugInfo/DWARF/DWARFDebugInfoEntry.h"
+#include "llvm/DebugInfo/DWARF/DWARFDebugLoc.h"
 #include <cassert>
 #include <cstdint>
 #include <iterator>
 
 namespace llvm {
 
 class DWARFUnit;
 class raw_ostream;
public:
   /// non-contiguous address ranges from the DW_AT_ranges attribute for this DIE
   /// and all children.
   ///
   /// \param Ranges the addres range vector to fill in.
   void collectChildrenAddressRanges(DWARFAddressRangesVector &Ranges) const;
 
   bool addressRangeContainsAddress(const uint64_t Address) const;
 
+  Expected<DWARFLocationExpressionsVector>
+  getLocations(dwarf::Attribute Attr) const;
+
   /// If a DIE represents a subprogram (or inlined subroutine), returns its
   /// mangled name (or short name, if mangled is missing). This name may be
   /// fetched from specification or abstract origin for this subprogram.
   /// Returns null if no name is found.
   const char *getSubroutineName(DINameKind Kind) const;
 
   /// Return the DIE name resolving DW_AT_sepcification or DW_AT_abstract_origin
   /// references if necessary. Returns null if no name is found.

llvm/include/llvm/DebugInfo/DWARF/DWARFLocationExpression.h

struct DWARFLocationExpression {
   /// default entry which is valid in addresses not covered by other location
   /// expressions, or everywhere if there are no other expressions.
   Optional<DWARFAddressRange> Range;
 
   /// The expression itself.
   SmallVector<uint8_t, 4> Expr;
 };
 
 inline bool operator==(const DWARFLocationExpression &L,
                        const DWARFLocationExpression &R) {
   return L.Range == R.Range && L.Expr == R.Expr;
 }
 
 inline bool operator!=(const DWARFLocationExpression &L,
                        const DWARFLocationExpression &R) {
   return !(L == R);
 }
 
 raw_ostream &operator<<(raw_ostream &OS, const DWARFLocationExpression &Loc);
 
+/// Represents a set of absolute location expressions.
+using DWARFLocationExpressionsVector = std::vector<DWARFLocationExpression>;
+
 } // end namespace llvm
 
 #endif // LLVM_DEBUGINFO_DWARF_DWARFLOCATIONEXPRESSION_H

llvm/include/llvm/DebugInfo/DWARF/DWARFUnit.h

public:
 
   Optional<uint64_t> getLoclistOffset(uint32_t Index) {
     if (LoclistTableHeader)
       return LoclistTableHeader->getOffsetEntry(Index);
     return None;
   }
   Expected<DWARFAddressRangesVector> collectAddressRanges();
 
+  Expected<DWARFLocationExpressionsVector>
+  findLoclistFromOffset(uint64_t Offset);
+
   /// Returns subprogram DIE with address range encompassing the provided
   /// address. The pointer is alive as long as parsed compile unit DIEs are not
   /// cleared.
   DWARFDie getSubroutineForAddress(uint64_t Address);
 
   /// getInlinedChainForAddress - fetches inlined chain for a given address.
   /// Returns empty chain if there is no subprogram containing address. The
   /// chain is valid as long as parsed compile unit DIEs are not cleared.

llvm/lib/DebugInfo/DWARF/DWARFDebugLoc.cpp

if (E) {
     OS << "\n";
     OS.indent(Indent);
     OS << "error: " << toString(std::move(E));
     return false;
   }
   return true;
 }
 
+Error DWARFLocationTable::visitAbsoluteLocationList(
+    uint64_t Offset, Optional<SectionedAddress> BaseAddr,
+    std::function<Optional<SectionedAddress>(uint32_t)> LookupAddr,
+    function_ref<bool(Expected<DWARFLocationExpression>)> Callback) const {
+  DWARFLocationInterpreter Interp(BaseAddr, std::move(LookupAddr));
+  return visitLocationList(&Offset, [&](const DWARFLocationEntry &E) {
+    Expected<Optional<DWARFLocationExpression>> Loc = Interp.Interpret(E);
+    if (!Loc)
+      return Callback(Loc.takeError());
+    if (*Loc)
+      return Callback(**Loc);
+    return true;
+  });
+}
+
 DWARFDebugLoc::LocationList const *
 DWARFDebugLoc::getLocationListAtOffset(uint64_t Offset) const {
   auto It = partition_point(
       Locations, [=](const LocationList &L) { return L.Offset < Offset; });
   if (It != Locations.end() && It->Offset == Offset)
     return &(*It);
   return nullptr;
 }

llvm/lib/DebugInfo/DWARF/DWARFDie.cpp

bool DWARFDie::addressRangeContainsAddress(const uint64_t Address) const {
   }
 
   for (const auto &R : RangesOrError.get())
     if (R.LowPC <= Address && Address < R.HighPC)
       return true;
   return false;
 }
 
+Expected<DWARFLocationExpressionsVector>
+DWARFDie::getLocations(dwarf::Attribute Attr) const {
+  Optional<DWARFFormValue> Location = find(Attr);
+  if (!Location)
+    return createStringError(inconvertibleErrorCode(), "No %s",
+                             dwarf::AttributeString(Attr).data());
+
+  if (Optional<uint64_t> Off = Location->getAsSectionOffset())
+    return U->findLoclistFromOffset(*Off);
+
+  if (Optional<ArrayRef<uint8_t>> Expr = Location->getAsBlock()) {
+    return DWARFLocationExpressionsVector{
+        DWARFLocationExpression{None, to_vector<4>(*Expr)}};
+  }
+
+  return createStringError(
+      inconvertibleErrorCode(), "Unsupported %s encoding: %s",
+      dwarf::AttributeString(Attr).data(),
+      dwarf::FormEncodingString(Location->getForm()).data());
+}
+
 const char *DWARFDie::getSubroutineName(DINameKind Kind) const {
   if (!isSubroutineDIE())
     return nullptr;
   return getName(Kind);
 }
 
 const char *DWARFDie::getName(DINameKind Kind) const {
   if (!isValid() || Kind == DINameKind::None)

llvm/lib/DebugInfo/DWARF/DWARFUnit.cpp

Expected<DWARFAddressRangesVector> DWARFUnit::collectAddressRanges() {
   auto CUDIERangesOrError = UnitDie.getAddressRanges();
   if (!CUDIERangesOrError)
     return createStringError(errc::invalid_argument,
                              "decoding address ranges: %s",
                              toString(CUDIERangesOrError.takeError()).c_str());
   return *CUDIERangesOrError;
 }
 
+Expected<DWARFLocationExpressionsVector>
+DWARFUnit::findLoclistFromOffset(uint64_t Offset) {
+  DWARFLocationExpressionsVector Result;
+
+  Error InterpretationError = Error::success();
+
+  Error ParseError = getLocationTable().visitAbsoluteLocationList(
+      Offset, getBaseAddress(),
+      [this](uint32_t Index) { return getAddrOffsetSectionItem(Index); },
+      [&](Expected<DWARFLocationExpression> L) {
+        if (L)
+          Result.push_back(std::move(*L));
+        else
+          InterpretationError =
+              joinErrors(L.takeError(), std::move(InterpretationError));
+        return !InterpretationError;
+      });
+
+  if (ParseError || InterpretationError)
+    return joinErrors(std::move(ParseError), std::move(InterpretationError));
+
+  return Result;
+}
+
 void DWARFUnit::updateAddressDieMap(DWARFDie Die) {
   if (Die.isSubroutineDIE()) {
     auto DIERangesOrError = Die.getAddressRanges();
     if (DIERangesOrError) {
       for (const auto &R : DIERangesOrError.get()) {
         // Ignore 0-sized ranges.
         if (R.LowPC == R.HighPC)
           continue;

llvm/lib/ObjectYAML/DWARFEmitter.cpp

 namespace {
 class DIEFixupVisitor : public DWARFYAML::Visitor {
   uint64_t Length;
 
 public:
   DIEFixupVisitor(DWARFYAML::Data &DI) : DWARFYAML::Visitor(DI){};
 
 private:
-  virtual void onStartCompileUnit(DWARFYAML::Unit &CU) { Length = 7; }
+  virtual void onStartCompileUnit(DWARFYAML::Unit &CU) {
+    // Size of the unit header, excluding the length field itself.
+    Length = CU.Version >= 5 ? 8 : 7;
+  }
 
   virtual void onEndCompileUnit(DWARFYAML::Unit &CU) {
     CU.Length.setLength(Length);
   }
 
   virtual void onStartDIE(DWARFYAML::Unit &CU, DWARFYAML::Entry &DIE) {
     Length += getULEB128Size(DIE.AbbrCode);
   }

llvm/unittests/DebugInfo/DWARF/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   ${LLVM_TARGETS_TO_BUILD}
   AsmPrinter
   DebugInfoDWARF
   MC
   Object
   ObjectYAML
   Support
   )
 
 add_llvm_unittest(DebugInfoDWARFTests
   DwarfGenerator.cpp
   DwarfUtils.cpp
   DWARFDebugInfoTest.cpp
   DWARFDebugLineTest.cpp
+  DWARFDieTest.cpp
   DWARFFormValueTest.cpp
   DWARFLocationExpressionTest.cpp
   )
 
 target_link_libraries(DebugInfoDWARFTests PRIVATE LLVMTestingSupport)

llvm/unittests/DebugInfo/DWARF/DWARFDieTest.cpp

+//===- llvm/unittest/DebugInfo/DWARFDieTest.cpp ---------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/BinaryFormat/Dwarf.h"
+#include "llvm/DebugInfo/DWARF/DWARFContext.h"
+#include "llvm/ObjectYAML/DWARFEmitter.h"
+#include "llvm/Testing/Support/Error.h"
+#include "gtest/gtest.h"
+
+using namespace llvm;
+using namespace llvm::dwarf;
+using object::SectionedAddress;
+
+namespace {
+
+TEST(DWARFLocationTable, getLocations) {
+  const char *yamldata = R"(
+    debug_abbrev:
+      - Code:            0x00000001
+        Tag:             DW_TAG_compile_unit
+        Children:        DW_CHILDREN_no
+        Attributes:
+          - Attribute:       DW_AT_location
+            Form:            DW_FORM_sec_offset
+          - Attribute:       DW_AT_data_member_location
+            Form:            DW_FORM_exprloc
+          - Attribute:       DW_AT_vtable_elem_location
+            Form:            DW_FORM_sec_offset
+          - Attribute:       DW_AT_call_data_location
+            Form:            DW_FORM_sec_offset
+    debug_info:
+      - Length:
+          TotalLength:     0
+        Version:         5
+        UnitType:        DW_UT_compile
+        AbbrOffset:      0
+        AddrSize:        4
+        Entries:
+          - AbbrCode:        0x00000001
+            Values:
+              - Value:           12
+              - Value:           0x0000000000000001
+                BlockData:       [ 0x47 ]
+              - Value:           20
+              - Value:           25
+  )";
+  Expected<StringMap<std::unique_ptr<MemoryBuffer>>> Sections =
+      DWARFYAML::EmitDebugSections(StringRef(yamldata),
+                                   /*IsLittleEndian=*/true);
+  ASSERT_THAT_EXPECTED(Sections, Succeeded());
+  std::vector<uint8_t> Loclists{
+      // Header
+      0, 0, 0, 0, // Length
+      5, 0,       // Version
+      4,          // Address size
+      0,          // Segment selector size
+      0, 0, 0, 0, // Offset entry count
+      // First location list.
+      DW_LLE_start_length, // First entry
+      1, 0, 0, 0,          // Start offset
+      2,                   // Length
+      0,                   // Expression length
+      DW_LLE_end_of_list,
+      // Second location list.
+      DW_LLE_startx_length, // First entry
+      1,                    // Start index
+      2,                    // Length
+      0,                    // Expression length
+      DW_LLE_end_of_list,
+      // Third location list.
+      DW_LLE_start_length, // First entry
+      1, 0, 0, 0,          // Start offset
+      2,                   // Length
+      0,                   // Expression length
+                           // end_of_list intentionally missing
+  };
+  Loclists[0] = Loclists.size() - 4;
+  Sections->try_emplace(
+      "debug_loclists",
+      MemoryBuffer::getMemBuffer(toStringRef(Loclists), "debug_loclists",
+                                 /*RequiresNullTerminator=*/false));
+  std::unique_ptr<DWARFContext> Ctx = DWARFContext::create(*Sections, 8);
+  DWARFCompileUnit *CU = Ctx->getCompileUnitForOffset(0);
+  ASSERT_NE(nullptr, CU);
+  DWARFDie Die = CU->getUnitDIE();
+  ASSERT_TRUE(Die.isValid());
+
+  EXPECT_THAT_EXPECTED(Die.getLocations(DW_AT_location),
+                       HasValue(testing::ElementsAre(DWARFLocationExpression{
+                           DWARFAddressRange{1, 3}, {}})));
+
+  EXPECT_THAT_EXPECTED(
+      Die.getLocations(DW_AT_data_member_location),
+      HasValue(testing::ElementsAre(DWARFLocationExpression{None, {0x47}})));
+
+  EXPECT_THAT_EXPECTED(
+      Die.getLocations(DW_AT_vtable_elem_location),
+      Failed<ErrorInfoBase>(testing::Property(
+          &ErrorInfoBase::message,
+          "Unable to resolve indirect address 1 for: DW_LLE_startx_length")));
+
+  EXPECT_THAT_EXPECTED(Die.getLocations(DW_AT_call_data_location),
+                       Failed<ErrorInfoBase>(testing::Property(
+                           &ErrorInfoBase::message, "unexpected end of data")));
+
+  EXPECT_THAT_EXPECTED(
+      Die.getLocations(DW_AT_call_data_value),
+      Failed<ErrorInfoBase>(testing::Property(&ErrorInfoBase::message,
+                                              "No DW_AT_call_data_value")));
+}
+
+} // end anonymous namespace