[LLDB] - Support the single file split DWARF.

[grimar]

Summary

DWARF5 spec describes single file split dwarf case (when .dwo sections are in the .o files).

Problem is that LLDB does not work correctly in that case. The issue is that, for example, both .debug_info and .debug_info.dwo
has the same type: eSectionTypeDWARFDebugInfo. And when code searches section by type it might find the regular debug section
and not the .dwo one. It seems can be easily fixed though, patch shows the fix I am suggesting.

Note that currently, clang is unable to produce such output, the patch is on review atm: D52296.
It is still would be possible to use assembler to produce such files though.
But the test case refers to the flag from the review mentioned.

[clayborg]

So the main questions is: do we need a new section enum called eSectionTypeDWARFDebugInfoDWO? If so, then this patch changes. I think I would prefer adding a new enum.

source/Plugins/SymbolFile/DWARF/SymbolFileDWARFDwo.cpp

68

If dwo has a unique section name, just name a new enum for the .debug_info.dwo.

[labath]

I believe you should be able to use lldb-test to write this kind of a test. I am thinking of a sequence like:

yaml2obj %p/Inputs/test.yaml > %t/test.yaml
yaml2obj %p/Inputs/test.o.yaml > %t/test.o.yaml
lldb-test breakpoint %t/test.yaml %s | FileCheck %s
b main
CHECK-LABEL: b main
CHECK: Address: {{.*}}main + 13 at test.cpp:2:3

b foo
CHECK-LABEL: b foo
CHECK: Address: {{.*}}foo() + 4 at test.cpp:6:1

You can look at existing tests in lit/Breakpoints for details.

[grimar]

Thanks for all the comments!

In D52403#1243127, @clayborg wrote:

So the main questions is: do we need a new section enum called eSectionTypeDWARFDebugInfoDWO? If so, then this patch changes. I think I would prefer adding a new enum.

Yeah, that is what I thought at the first place too. I found that when code looks for some section, it seems sometimes assumes it can find .dwo section by regular section enum value.
I thought that time that much more changes might be required (for example, introducing enum values for each .dwo section and perhaps fixing callers)
and so selected the simplest solution (this patch), because original code already used filtering by section name "dwo" suffix.

I'll try to investigate it again, want to check a few ideas I have in mind.

[grimar]

• Addressed review comments.

[clayborg]

Just a few fixes. Looking good.

include/lldb/lldb-enumerations.h

643–660 ↗

(On Diff #167287)

Add all of these to the end of this enum for API stability since this is a public header used in the API. If an older binary runs against a newer liblldb.so, all of these enums will be off.

source/Symbol/ObjectFile.cpp

347 ↗

(On Diff #167287)

Check for other ObjectFile subclasses that override this function. I believe ObjectFileMachO does.

[grimar]

• Addressed review comments.

include/lldb/lldb-enumerations.h

643–660 ↗

(On Diff #167287)

Done.

source/Symbol/ObjectFile.cpp

347 ↗

(On Diff #167287)

Yes, and my patch already has the change for ObjectFileMachO :)
All other classes are fine too I think.

[clayborg]

Looks good!

[grimar]

Thanks for the review! It still depends on -gsingle-file-split-dwarf (D52403), I'll wait for it some time.
It should be possible to rewrite the comment in the test case to avoid mentioning the flag, but I would prefer to use it.

[grimar]

Committed as rLLDB346848.
The commit message has a wrong review link by mistake.

include/lldb/Core/Section.h

public:
 
   void Dump(Stream *s, Target *target, bool show_header, uint32_t depth) const;
 
   lldb::SectionSP FindSectionByName(const ConstString &section_dstr) const;
 
   lldb::SectionSP FindSectionByID(lldb::user_id_t sect_id) const;
 
   lldb::SectionSP FindSectionByType(lldb::SectionType sect_type,
-                                    bool check_children,
-                                    size_t start_idx = 0) const;
+                                    bool check_children, size_t start_idx = 0,
+                                    bool onlyDwo = false) const;
 
   lldb::SectionSP
   FindSectionContainingFileAddress(lldb::addr_t addr,
                                    uint32_t depth = UINT32_MAX) const;
 
   // Get the number of sections in this list only
   size_t GetSize() const { return m_sections.size(); }
 

packages/Python/lldbsuite/test/functionalities/single-file-split-dwarf/TestSingleFileSplitDwarf.py

+import lldb
+from lldbsuite.test.lldbtest import *
+from lldbsuite.test.decorators import *
+
+# This test checks that source code location is shown correctly
+# when single split file debug information is used (when .dwo sections are in .o files).
+#
+# test.yaml and test.o.yaml are reduced yaml files produces from the
+# object file and the corresponding executable.
+#
+# The following code was used initially:
+# int main() {
+#   return 0;
+# }
+# 
+# void foo() {
+# }
+#
+# Invocation used to produce .o file was:
+# clang test.cpp -gdwarf-5 -gsplit-dwarf -c -o test.o -v -gsingle-file-split-dwarf
+#
+# Invocation used to produce the executable was:
+# "/usr/local/bin/ld" -z relro --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -dynamic-linker
+#  /lib64/ld-linux-x86-64.so.2 -o test /usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../x86_64-linux-gnu/crt1.o
+#  /usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/7.3.0/crtbegin.o
+#  -L/usr/lib/gcc/x86_64-linux-gnu/7.3.0 -L/usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../x86_64-linux-gnu
+#  -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../..
+#  -L/home/umb/LLVM/build/bin/../lib -L/lib -L/usr/lib test.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc
+#  --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/x86_64-linux-gnu/7.3.0/crtend.o
+#  /usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../x86_64-linux-gnu/crtn.o
+
+class TestSingleFileSplitDwarf(TestBase):
+
+    mydir = TestBase.compute_mydir(__file__)
+
+    def test_main(self):
+        yaml_path = os.path.join(self.getSourceDir(), "test.yaml")
+        yaml_base, ext = os.path.splitext(yaml_path)
+        obj_path = self.getBuildArtifact(yaml_base)
+        self.yaml2obj(yaml_path, obj_path)
+
+        yaml_o_path = os.path.join(self.getSourceDir(), "test.o.yaml")
+        yaml_o_base, ext_o = os.path.splitext(yaml_o_path)
+        obj_o_path = self.getBuildArtifact(yaml_o_base)
+        self.yaml2obj(yaml_o_path, obj_o_path)
+
+        def cleanup():
+            if os.path.exists(obj_path):
+                os.unlink(obj_path)
+            if os.path.exists(obj_o_path):
+                os.unlink(obj_o_path)
+
+        # Execute the cleanup function during test case tear down.
+        self.addTearDownHook(cleanup)
+
+        # Create a target with the object file we just created from YAML
+        target = self.dbg.CreateTarget(obj_path)
+
+        # Check we are able to show the locations properly.
+        self.expect("b main", VALID_BREAKPOINT_LOCATION,
+                    substrs=['main + 13 at test.cpp:2:3, address = 0x000000000040052d'])
+
+        self.expect("b foo", VALID_BREAKPOINT_LOCATION,
+                    substrs=['foo() + 4 at test.cpp:6:1, address = 0x0000000000400534'])

packages/Python/lldbsuite/test/functionalities/single-file-split-dwarf/test.o.yaml

+--- !ELF
+FileHeader:      
+  Class:           ELFCLASS64
+  Data:            ELFDATA2LSB
+  Type:            ET_REL
+  Machine:         EM_X86_64
+Sections:        
+  - Name:            .text
+    Type:            SHT_PROGBITS
+    Flags:           [ SHF_ALLOC, SHF_EXECINSTR ]
+    AddressAlign:    0x0000000000000010
+    Content:         554889E531C0C745FC000000005DC390554889E55DC3
+  - Name:            .debug_str_offsets
+    Type:            SHT_PROGBITS
+    AddressAlign:    0x0000000000000001
+    Content:         0C000000050000000000000000000000
+  - Name:            .debug_str
+    Type:            SHT_PROGBITS
+    Flags:           [ SHF_MERGE, SHF_STRINGS ]
+    AddressAlign:    0x0000000000000001
+    Content:         746573742E6F002F686F6D652F756D622F74657374735F323031382F39355F6C6C64622F726570726F2F6477617266355F73706C69745F73696E676C655F66696C652F707265706172655F73616D706C65006D61696E00666F6F005F5A33666F6F7600696E7400
+  - Name:            .debug_loc.dwo
+    Type:            SHT_PROGBITS
+    Flags:           [ SHF_EXCLUDE ]
+    AddressAlign:    0x0000000000000001
+    Content:         ''
+  - Name:            .debug_abbrev
+    Type:            SHT_PROGBITS
+    AddressAlign:    0x0000000000000001
+    Content:         01110010177217B042251B25B44219B3421711011206000000
+  - Name:            .debug_info
+    Type:            SHT_PROGBITS
+    AddressAlign:    0x0000000000000001
+    Content:         2B00000005000408000000003F4B7684A29835B9010000000000000000000100000000000000000000000016000000
+  - Name:            .debug_macinfo
+    Type:            SHT_PROGBITS
+    AddressAlign:    0x0000000000000001
+    Content:         '00'
+  - Name:            .debug_str_offsets.dwo
+    Type:            SHT_PROGBITS
+    Flags:           [ SHF_EXCLUDE ]
+    AddressAlign:    0x0000000000000001
+    Content:         200000000500000000000000070000002A00000033000000380000003C00000044000000
+  - Name:            .debug_str.dwo
+    Type:            SHT_PROGBITS
+    Flags:           [ SHF_EXCLUDE, SHF_MERGE, SHF_STRINGS ]
+    AddressAlign:    0x0000000000000001
+    Content:         746573742E6F00636C616E672076657273696F6E20382E302E3020287472756E6B203334323731382900746573742E637070006D61696E00696E74005F5A33666F6F7600666F6F00
+  - Name:            .debug_info.dwo
+    Type:            SHT_PROGBITS
+    Flags:           [ SHF_EXCLUDE ]
+    AddressAlign:    0x0000000000000001
+    Content:         3600000005000508000000003F4B7684A29835B901000104000202000F0000000156030101350000000301060000000156050601050404050400
+  - Name:            .debug_abbrev.dwo
+    Type:            SHT_PROGBITS
+    Flags:           [ SHF_EXCLUDE ]
+    AddressAlign:    0x0000000000000001
+    Content:         011101B042252525130503250000022E0011813E1206401803253A0B3B0B49133F190000032E0011813E120640186E2503253A0B3B0B3F19000004240003253E0B0B0B000000
+  - Name:            .debug_addr
+    Type:            SHT_PROGBITS
+    AddressAlign:    0x0000000000000001
+    Content:         '140000000500080000000000000000000000000000000000'
+  - Name:            .debug_names
+    Type:            SHT_PROGBITS
+    AddressAlign:    0x0000000000000004
+    Content:         A00000000500000001000000000000000000000004000000040000000D000000080000004C4C564D3037303000000000010000000200000003000000040000003080880B8973880B6A7F9A7C0B3D06B5000000000000000000000000000000000000000009000000120000001B0000002E2E03130000242403130000002435000000000000002E29000000000000002E1A000000000000002E2900000000000000000000
+  - Name:            .debug_gnu_pubnames
+    Type:            SHT_PROGBITS
+    AddressAlign:    0x0000000000000001
+    Content:         210000000200000000002F0000002900000030666F6F001A000000306D61696E0000000000
+  - Name:            .debug_gnu_pubtypes
+    Type:            SHT_PROGBITS
+    AddressAlign:    0x0000000000000001
+    Content:         170000000200000000002F0000003500000090696E740000000000
+  - Name:            .debug_line
+    Type:            SHT_PROGBITS
+    AddressAlign:    0x0000000000000001
+    Content:         70000000050008004C000000010101FB0E0D00010101010000000100000101011F010000000003011F020F051E020000000000FD7C0F2E46BA561F7BDA351B04E677090000000000FD7C0F2E46BA561F7BDA351B04E6770900090200000000000000000105030AC905003F05010A4B0202000101
+  - Name:            .debug_line_str
+    Type:            SHT_PROGBITS
+    Flags:           [ SHF_MERGE, SHF_STRINGS ]
+    AddressAlign:    0x0000000000000001
+    Content:         2F686F6D652F756D622F74657374735F323031382F39355F6C6C64622F726570726F2F6477617266355F73706C69745F73696E676C655F66696C652F707265706172655F73616D706C6500746573742E63707000
+...

packages/Python/lldbsuite/test/functionalities/single-file-split-dwarf/test.yaml

+--- !ELF
+FileHeader:      
+  Class:           ELFCLASS64
+  Data:            ELFDATA2LSB
+  Type:            ET_EXEC
+  Machine:         EM_X86_64
+  Entry:           0x0000000000400440
+Sections:        
+  - Name:            .text
+    Type:            SHT_PROGBITS
+    Flags:           [ SHF_ALLOC, SHF_EXECINSTR ]
+    Address:         0x0000000000400440
+    AddressAlign:    0x0000000000000010
+    Content:         31ED4989D15E4889E24883E4F0505449C7C0B005400048C7C14005400048C7C720054000E8B7FFFFFFF4660F1F44000055B820204000483D202040004889E57417B8000000004885C0740D5DBF20204000FFE00F1F4400005DC3660F1F440000BE20204000554881EE202040004889E548C1FE034889F048C1E83F4801C648D1FE7415B8000000004885C0740B5DBF20204000FFE00F1F005DC3660F1F440000803D391B0000007517554889E5E87EFFFFFFC605271B0000015DC30F1F440000F3C30F1F4000662E0F1F840000000000554889E55DEB89660F1F840000000000554889E531C0C745FC000000005DC390554889E55DC3662E0F1F840000000000415741564189FF415541544C8D25B618000055488D2DB6180000534989F64989D54C29E54883EC0848C1FD03E87FFEFFFF4885ED742031DB0F1F8400000000004C89EA4C89F64489FF41FF14DC4883C3014839EB75EA4883C4085B5D415C415D415E415FC390662E0F1F840000000000F3C3
+  - Name:            .debug_str_offsets
+    Type:            SHT_PROGBITS
+    AddressAlign:    0x0000000000000001
+    Content:         0C000000050000000000000007000000
+  - Name:            .debug_str
+    Type:            SHT_PROGBITS
+    Flags:           [ SHF_MERGE, SHF_STRINGS ]
+    AddressAlign:    0x0000000000000001
+    Content:         746573742E6F002F686F6D652F756D622F74657374735F323031382F39355F6C6C64622F726570726F2F6477617266355F73706C69745F73696E676C655F66696C652F707265706172655F73616D706C65006D61696E00666F6F005F5A33666F6F7600696E7400
+  - Name:            .debug_abbrev
+    Type:            SHT_PROGBITS
+    AddressAlign:    0x0000000000000001
+    Content:         01110010177217B042251B25B44219B3421711011206000000
+  - Name:            .debug_info
+    Type:            SHT_PROGBITS
+    AddressAlign:    0x0000000000000001
+    Content:         2B00000005000408000000003F4B7684A29835B9010000000008000000000108000000200540000000000016000000
+  - Name:            .debug_macinfo
+    Type:            SHT_PROGBITS
+    AddressAlign:    0x0000000000000001
+    Content:         '00'
+  - Name:            .debug_addr
+    Type:            SHT_PROGBITS
+    AddressAlign:    0x0000000000000001
+    Content:         '140000000500080020054000000000003005400000000000'
+  - Name:            .debug_names
+    Type:            SHT_PROGBITS
+    AddressAlign:    0x0000000000000004
+    Content:         A00000000500000001000000000000000000000004000000040000000D000000080000004C4C564D3037303000000000010000000200000003000000040000003080880B8973880B6A7F9A7C0B3D06B56300000057000000520000005B0000000000000009000000120000001B0000002E2E03130000242403130000002435000000000000002E29000000000000002E1A000000000000002E2900000000000000000000
+  - Name:            .debug_gnu_pubnames
+    Type:            SHT_PROGBITS
+    AddressAlign:    0x0000000000000001
+    Content:         210000000200000000002F0000002900000030666F6F001A000000306D61696E0000000000
+  - Name:            .debug_gnu_pubtypes
+    Type:            SHT_PROGBITS
+    AddressAlign:    0x0000000000000001
+    Content:         170000000200000000002F0000003500000090696E740000000000
+  - Name:            .debug_line
+    Type:            SHT_PROGBITS
+    AddressAlign:    0x0000000000000001
+    Content:         70000000050008004C000000010101FB0E0D00010101010000000100000101011F010000000003011F020F051E024B00000000FD7C0F2E46BA561F7BDA351B04E677094B00000000FD7C0F2E46BA561F7BDA351B04E6770900090220054000000000000105030AC905003F05010A4B0202000101
+  - Name:            .debug_line_str
+    Type:            SHT_PROGBITS
+    Flags:           [ SHF_MERGE, SHF_STRINGS ]
+    AddressAlign:    0x0000000000000001
+    Content:         2F686F6D652F756D622F74657374735F323031382F39355F6C6C64622F726570726F2F6477617266355F73706C69745F73696E676C655F66696C652F707265706172655F73616D706C6500746573742E63707000
+...

source/Core/Section.cpp

for (sect_iter = m_sections.begin();
         sect_sp = (*sect_iter)->GetChildren().FindSectionByID(sect_id);
       }
     }
   }
   return sect_sp;
 }
 
 SectionSP SectionList::FindSectionByType(SectionType sect_type,
-                                         bool check_children,
-                                         size_t start_idx) const {
+                                         bool check_children, size_t start_idx,
+                                         bool onlyDwo) const {
   SectionSP sect_sp;
   size_t num_sections = m_sections.size();
   for (size_t idx = start_idx; idx < num_sections; ++idx) {
     if (m_sections[idx]->GetType() == sect_type) {
+      if (onlyDwo && !m_sections[idx]->GetName().GetStringRef().endswith("dwo"))
+        continue;
+
       sect_sp = m_sections[idx];
       break;
     } else if (check_children) {
       sect_sp = m_sections[idx]->GetChildren().FindSectionByType(
           sect_type, check_children, 0);
       if (sect_sp)
         break;
     }

source/Plugins/SymbolFile/DWARF/SymbolFileDWARFDwo.cpp

SymbolFileDWARFDwo::SymbolFileDWARFDwo(ObjectFileSP objfile,
   SetID(((lldb::user_id_t)dwarf_cu->GetOffset()) << 32);
 }
 
 void SymbolFileDWARFDwo::LoadSectionData(lldb::SectionType sect_type,
                                          DWARFDataExtractor &data) {
   const SectionList *section_list =
       m_obj_file->GetSectionList(false /* update_module_section_list */);
   if (section_list) {
-    SectionSP section_sp(section_list->FindSectionByType(sect_type, true));
+    SectionSP section_sp(section_list->FindSectionByType(
+        sect_type, true, 0 /*start_idx*/, true /*onlyDwo*/));
     if (section_sp) {
       // See if we memory mapped the DWARF segment?
       if (m_dwarf_data.GetByteSize()) {
         data.SetData(m_dwarf_data, section_sp->GetOffset(),
                      section_sp->GetFileSize());
         return;
       }
 
assert(GetCompileUnit() == dwarf_cu && "SymbolFileDWARFDwo::ParseCompileUnit "
                                          "called with incompatible compile "
                                          "unit");
   return GetBaseSymbolFile()->ParseCompileUnit(m_base_dwarf_cu, UINT32_MAX);
 }
 
 DWARFUnit *SymbolFileDWARFDwo::GetCompileUnit() {
   // A clang module is found via a skeleton CU, but is not a proper DWO.
   // Clang modules have a .debug_info section instead of the *_dwo variant.
-  if (auto *section_list = m_obj_file->GetSectionList(false))
-    if (auto section_sp =
-            section_list->FindSectionByType(eSectionTypeDWARFDebugInfo, true))
-      if (!section_sp->GetName().GetStringRef().endswith("dwo"))
-        return nullptr;
+  if (auto *section_list = m_obj_file->GetSectionList(false)) {
+    auto section_sp = section_list->FindSectionByType(
+        eSectionTypeDWARFDebugInfo, true, 0 /*start_idx*/, true /*onlyDwo*/);
+    if (!section_sp)
+  }
 
   // Only dwo files with 1 compile unit is supported
   if (GetNumCompileUnits() == 1)
     return DebugInfo()->GetCompileUnitAtIndex(0);
   else
     return nullptr;
 }