Add support for type units (.debug_types) to LLDB in a way that is compatible with DWARF 5

[clayborg]

Summary

This patch adds support for type units and parsing the .debug_types section which allows LLDB to debug binaries that were built with "-gdwarf-4 -fdebug-types-section". This patch takes into account how DWARF 5 will represent type units: all compile units will contain extra fields and all type units are in the .debug_info section.

Normally this change would be very invasive because we would need to make a way for DIEs in the .debug_types section to have unique IDs (lldb::user_id_t). This patch takes advantage of the fact that debug info in the .debug_types section can't point to any other DWARF directly. There are only CU relative references and type signatures that might point to other debug info, but indirected through the type signature. This means the offset of the DIEs in the ,.debug_types section are not important. Given all of this, there are the important takes from this patch:

• Modify DWARFCompileUnit to contain extra fields: type signature and type offset. This is the way DWARF5 is doing it, so we will need to do that anyway in the near future.
• Slide the offset of all DIEs in .debug_types by the size of the .debug_info section. This allows every DIE to have unique offset and allows this patch to "just work" with the existing DWARF parser
• Anyone extracting attribute values must use the DWARFDIE::GetData() or DWARFCompileUnit::GetData() to make sure they are getting the right info (.debug_info or .debug_types) to extract from
• LLDB testing adds a new "_dwarf_debug_types" variant (like we add the _dwarf, _dwarf_dsym, _dwo) on platforms that support the .debug_types

I would love to get this in. Anything is better that what LLDB does now when it debugs binaries with .debug_types: crash a lot. So we should get this in and iterate on it.

[labath]

The new tests which are not passing need to be disabled -- we cannot have a red tree while this is being developed.

[clayborg]

Yeah, I was having trouble getting a real read on what was failing due to a non Ubuntu setup at work for me. Is there any way to run a patch on one of the buildbots? If not, can you email me the list of what is failing?

[labath]

In D32167#729508, @clayborg wrote:

Yeah, I was having trouble getting a real read on what was failing due to a non Ubuntu setup at work for me. Is there any way to run a patch on one of the buildbots? If not, can you email me the list of what is failing?

I can send you the list tomorrow, but I am not sure how helpful will it be... When I tried to run it today, it looked like the list was quite large.

What is your development setup nowadays? I am sure you will get some failures on non-ubuntu machines, but I wouldn't expect it to be that many of them. In any case, I think we should work towards eliminating these environment dependencies, particularly if you're going to be doing linux-based development now...

[aprantl]

packages/Python/lldbsuite/test/make/Makefile.rules

197

This shouldn't be necessary on any platform LLDB cares about. DWARF 4 should be the default everywhere.

packages/Python/lldbsuite/test/plugins/builder_base.py

204

Why does the type unit configuration care about whether there are dsyms or not? Shouldn't this be orthogonal?

packages/Python/lldbsuite/test/test_categories.py

27

This is conflating two things (-fdebug-types-section and type units) DWARF5 doesn't have a debug_types section but it still offers type units. Clang doesn't implement this yet, but GCC might, I haven't checked.

source/Plugins/SymbolFile/DWARF/DWARFCompileUnit.cpp

1109

if (auto cu = ...)

1116

if (auto cu = ...)

source/Plugins/SymbolFile/DWARF/DWARFCompileUnit.h

169

/// everywhere?

source/Plugins/SymbolFile/DWARF/DWARFDataExtractor.h

51

I think the //--- line might confuse Doxygen, but I'm not sure.

source/Plugins/SymbolFile/DWARF/DWARFDebugInfo.cpp

121

Is that check common LLDB style?

[jasonmolenda]

Looks fine to me, I don't know the debug_types sections that you're supporting but I read over the code and don't have any problems.

source/Plugins/SymbolFile/DWARF/DWARFDIE.h

92 ↗

(On Diff #95572)

DWARFDIE::GetData() handles the difference between debug_info / debug_types so the callers of this method don't need to worry about that, correct? Maybe the comment could say

// Get the data that contains the attribute values for this DIE. Support
// for .debug_types means that any DIE can have its data either in the
// .debug_info or the .debug_types section; this method will return the
// correct section data.

or something.

[labath]

This is the list I am getting when using ToT clang targetting x86_64 linux.

Note that there are a couple of non-type-units test broken by this. This needs to be addressed first.

=============
Issue Details
=============
FAIL: test_SBType_template_aspects_dwarf_type_units (lang/cpp/stl/TestSTL.py)
FAIL: test_char_type_dwarf_type_units (types/TestIntegerTypes.py)
FAIL: test_char_type_dwarf_type_units (types/TestIntegerTypesExpr.py)
FAIL: test_double_type_dwarf_type_units (types/TestFloatTypes.py)
FAIL: test_double_type_dwarf_type_units (types/TestFloatTypesExpr.py)
FAIL: test_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libcxx/atomic/TestLibCxxAtomic.py)
FAIL: test_dwarf_type_units (python_api/type/TestTypeList.py)
FAIL: test_dwarf_type_units (python_api/value/empty_class/TestValueAPIEmptyClass.py)
FAIL: test_dwarf_type_units (lang/cpp/wchar_t/TestCxxWCharT.py)
FAIL: test_dwarf_type_units (expression_command/call-function/TestCallStopAndContinue.py)
FAIL: test_dwarf_type_units (expression_command/formatters/TestFormatters.py)
FAIL: test_dwarf_type_units (python_api/sbvalue_persist/TestSBValuePersist.py)
FAIL: test_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libcxx/function/TestLibCxxFunction.py)
FAIL: test_enum_args_dwarf_type_units (lang/cpp/template/TestTemplateArgs.py)
FAIL: test_eval_index_variable_dwarf_type_units (functionalities/non-overlapping-index-variable-i/TestIndexVariable.py)
FAIL: test_float_type_dwarf_type_units (types/TestFloatTypesExpr.py)
FAIL: test_float_type_dwarf_type_units (types/TestFloatTypes.py)
FAIL: test_force_synth_off_dwarf_type_units (python_api/formatters/TestFormattersSBAPI.py)
FAIL: test_formatters_api_dwarf_type_units (python_api/formatters/TestFormattersSBAPI.py)
FAIL: test_get_dynamic_vals_dwarf_type_units (lang/cpp/dynamic-value/TestDynamicValue.py)
FAIL: test_int_type_dwarf_type_units (types/TestIntegerTypesExpr.py)
FAIL: test_int_type_dwarf_type_units (types/TestIntegerTypes.py)
FAIL: test_integer_args_dwarf_type_units (lang/cpp/template/TestTemplateArgs.py)
FAIL: test_long_long_type_dwarf_type_units (types/TestIntegerTypesExpr.py)
FAIL: test_long_long_type_dwarf_type_units (types/TestIntegerTypes.py)
FAIL: test_long_type_dwarf_type_units (types/TestIntegerTypes.py)
FAIL: test_long_type_dwarf_type_units (types/TestIntegerTypesExpr.py)
FAIL: test_operator_new_dwarf_type_units (lang/cpp/global_operators/TestCppGlobalOperators.py)
FAIL: test_rdar10960550_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-python-synth/TestDataFormatterPythonSynth.py)
FAIL: test_recursive_unique_ptr_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libstdcpp/unique_ptr/TestDataFormatterStdUniquePtr.py)
FAIL: test_short_type_dwarf_type_units (types/TestIntegerTypesExpr.py)
FAIL: test_short_type_dwarf_type_units (types/TestIntegerTypes.py)
FAIL: test_unsigned_char_type_dwarf_type_units (types/TestIntegerTypesExpr.py)
FAIL: test_unsigned_char_type_dwarf_type_units (types/TestIntegerTypes.py)
FAIL: test_unsigned_int_type_dwarf_type_units (types/TestIntegerTypesExpr.py)
FAIL: test_unsigned_int_type_dwarf_type_units (types/TestIntegerTypes.py)
FAIL: test_unsigned_long_long_type_dwarf_type_units (types/TestIntegerTypes.py)
FAIL: test_unsigned_long_long_type_dwarf_type_units (types/TestIntegerTypesExpr.py)
FAIL: test_unsigned_long_type_dwarf_type_units (types/TestIntegerTypesExpr.py)
FAIL: test_unsigned_long_type_dwarf_type_units (types/TestIntegerTypes.py)
FAIL: test_unsigned_short_type_dwarf_type_units (types/TestIntegerTypes.py)
FAIL: test_unsigned_short_type_dwarf_type_units (types/TestIntegerTypesExpr.py)
FAIL: test_value_cast_with_regular_inheritance_dwarf_type_units (lang/cpp/dynamic-value/TestCppValueCast.py)
FAIL: test_virtual_madness_dwarf_type_units (lang/cpp/virtual/TestVirtual.py)
FAIL: test_with_dwarf_type_units (expression_command/call-function/TestCallStdStringFunction.py)
FAIL: test_with_expr_parser_dwarf_type_units (lang/cpp/class_types/TestClassTypes.py)
FAIL: test_with_gmodules (functionalities/data-formatter/stringprinter/TestStringPrinter.py)
FAIL: test_with_python_api_dwarf_type_units (lang/cpp/class_static/TestStaticVariables.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libstdcpp/vbool/TestDataFormatterStdVBool.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libcxx/vbool/TestDataFormatterLibcxxVBool.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/format-propagation/TestFormatPropagation.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-synthtype/TestDataFormatterSynthType.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/breakpoint/inlined_breakpoints/TestInlinedBreakpoints.py)
FAIL: test_with_run_command_dwarf_type_units (lang/cpp/class_types/TestClassTypes.py)
FAIL: test_with_run_command_dwarf_type_units (lang/cpp/static_methods/TestCPPStaticMethods.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libcxx/vector/TestDataFormatterLibcxxVector.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-python-synth/TestDataFormatterPythonSynth.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/language_category_updates/TestDataFormatterLanguageCategoryUpdates.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libstdcpp/map/TestDataFormatterStdMap.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-smart-array/TestDataFormatterSmartArray.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libstdcpp/list/TestDataFormatterStdList.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/user-format-vs-summary/TestUserFormatVsSummary.py)
FAIL: test_with_run_command_dwarf_type_units (lang/cpp/diamond/TestDiamond.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libstdcpp/iterator/TestDataFormatterStdIterator.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-cpp/TestDataFormatterCpp.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libcxx/string/TestDataFormatterLibcxxString.py)
FAIL: test_with_run_command_dwarf_type_units (python_api/sbdata/TestSBData.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libstdcpp/vector/TestDataFormatterStdVector.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/refpointer-recursion/TestDataFormatterRefPtrRecursion.py)
FAIL: test_with_run_command_dwarf_type_units (lang/cpp/overloaded-functions/TestOverloadedFunctions.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/synthcapping/TestSyntheticCapping.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-enum-format/TestDataFormatterEnumFormat.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-synthval/TestDataFormatterSynthVal.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-categories/TestDataFormatterCategories.py)
FAIL: test_with_run_command_dwarf_type_units (lang/cpp/auto/TestCPPAuto.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libcxx/multimap/TestDataFormatterLibccMultiMap.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/var-in-aggregate-misuse/TestVarInAggregateMisuse.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-disabling/TestDataFormatterDisabling.py)
FAIL: test_with_run_command_dwarf_type_units (lang/cpp/this/TestCPPThis.py)
FAIL: test_with_run_command_dwarf_type_units (lang/cpp/chained-calls/TestCppChainedCalls.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libcxx/unordered/TestDataFormatterUnordered.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libstdcpp/string/TestDataFormatterStdString.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-globals/TestDataFormatterGlobals.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/summary-string-onfail/Test-rdar-9974002.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libcxx/iterator/TestDataFormatterLibccIterator.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/varscript_formatting/TestDataFormatterVarScriptFormatting.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-synth/TestDataFormatterSynth.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/hexcaps/TestDataFormatterHexCaps.py)
FAIL: test_with_run_command_dwarf_type_units (lang/cpp/class_static/TestStaticVariables.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libstdcpp/smart_ptr/TestDataFormatterStdSmartPtr.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libstdcpp/tuple/TestDataFormatterStdTuple.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/type_completion/TestTypeCompletion.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/recursion/TestValueObjectRecursion.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libcxx/list/TestDataFormatterLibcxxList.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libstdcpp/unique_ptr/TestDataFormatterStdUniquePtr.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-script/TestDataFormatterScript.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-named-summaries/TestDataFormatterNamedSummaries.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libcxx/map/TestDataFormatterLibccMap.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-skip-summary/TestDataFormatterSkipSummary.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libcxx/multiset/TestDataFormatterLibcxxMultiSet.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-advanced/TestDataFormatterAdv.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/frameformat_smallstruct/TestFrameFormatSmallStruct.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libcxx/list/loop/TestDataFormatterLibcxxListLoop.py)
ERROR: test_dwarf (functionalities/archives/TestBSDArchives.py)
ERROR: test_dwarf_type_units (functionalities/archives/TestBSDArchives.py)
ERROR: test_dwarf_type_units (python_api/value/linked_list/TestValueAPILinkedList.py)
ERROR: test_dwo (functionalities/archives/TestBSDArchives.py)
ERROR: test_expr_dwarf (lang/c/shared_lib_stripped_symbols/TestSharedLibStrippedSymbols.py)
ERROR: test_expr_dwarf_type_units (lang/c/shared_lib_stripped_symbols/TestSharedLibStrippedSymbols.py)
ERROR: test_expr_dwo (lang/c/shared_lib_stripped_symbols/TestSharedLibStrippedSymbols.py)
ERROR: test_expr_gmodules (lang/c/shared_lib_stripped_symbols/TestSharedLibStrippedSymbols.py)
ERROR: test_gmodules (functionalities/archives/TestBSDArchives.py)
UNEXPECTED SUCCESS: test_dwarf (functionalities/thread/exit_during_break/TestExitDuringBreak.py)
UNEXPECTED SUCCESS: test_dwarf_type_units (functionalities/thread/exit_during_break/TestExitDuringBreak.py)
UNEXPECTED SUCCESS: test_dwo (functionalities/thread/exit_during_break/TestExitDuringBreak.py)
UNEXPECTED SUCCESS: test_gmodules (functionalities/thread/exit_during_break/TestExitDuringBreak.py)
UNEXPECTED SUCCESS: test_lldbmi_gdb_set_target_async_off (tools/lldb-mi/TestMiGdbSetShow.py)
UNEXPECTED SUCCESS: test_lldbmi_process_output (tools/lldb-mi/syntax/TestMiSyntax.py)
UNEXPECTED SUCCESS: test_lldbmi_settings_set_target_run_args_after (tools/lldb-mi/interpreter/TestMiInterpreterExec.py)
UNEXPECTED SUCCESS: test_process_interrupt_dwarf_type_units (functionalities/thread/state/TestThreadStates.py)
UNEXPECTED SUCCESS: test_process_interrupt_dwo (functionalities/thread/state/TestThreadStates.py)
UNEXPECTED SUCCESS: test_process_interrupt_gmodules (functionalities/thread/state/TestThreadStates.py)
UNEXPECTED SUCCESS: test_restart_bug_dwarf (functionalities/signal/raise/TestRaise.py)
UNEXPECTED SUCCESS: test_restart_bug_dwarf_type_units (functionalities/signal/raise/TestRaise.py)
UNEXPECTED SUCCESS: test_sb_api_listener_resume_dwarf (api/multithreaded/TestMultithreaded.py)
UNEXPECTED SUCCESS: test_sb_api_listener_resume_dwarf_type_units (api/multithreaded/TestMultithreaded.py)
UNEXPECTED SUCCESS: test_sb_api_listener_resume_dwo (api/multithreaded/TestMultithreaded.py)
UNEXPECTED SUCCESS: test_sb_api_listener_resume_gmodules (api/multithreaded/TestMultithreaded.py)
UNEXPECTED SUCCESS: test_with_dwarf (lang/cpp/printf/TestPrintf.py)
UNEXPECTED SUCCESS: test_with_dwo (lang/cpp/printf/TestPrintf.py)
UNEXPECTED SUCCESS: test_with_gmodules (lang/cpp/printf/TestPrintf.py)

===================
Test Result Summary
===================
Test Methods:       3220
Reruns:                0
Success:            2031
Expected Failure:    187
Failure:             103
Error:                 9
Exceptional Exit:      0
Unexpected Success:   19
Skip:                871
Timeout:               0
Expected Timeout:      0

[clayborg]

Updated to top of tree sources and verified it passes all tests on linux.

[davide]

This commit has no tests. It should have many. It's very big, so it could be split in several pieces.
I'd really appreciate if you can take the time to do so. For now, some comments.

include/lldb/lldb-enumerations.h

659 ↗

(On Diff #135913)

this needs a comment.

packages/Python/lldbsuite/test/lldbtest.py

704

spurious whitespaces, please revert.

1765–1766

ditto.

source/Plugins/SymbolFile/DWARF/DIERef.cpp

51–68

Why? This needs to be explained in a comment.

source/Plugins/SymbolFile/DWARF/DWARFDebugInfo.h

66

any reason why you can't use LLVM adt?

[clayborg]

In D32167#1019399, @davide wrote:

This commit has no tests. It should have many.

I has full testing. Please read the commit and notice there is a new debug info flavor that is added. So of course it is tested.

It's very big, so it could be split in several pieces.

This is one thing that goes together. It isn't that big. I adds support for .debug_types into the ObjectFile and parses the information. How would you propose breaking this up?

I'd really appreciate if you can take the time to do so. For now, some comments.

I believe I did take the time to test this properly. Let me know if you still think otherwise.

packages/Python/lldbsuite/test/lldbtest.py

704

It is removing spurious whitespaces and there are other changes in this file. Most editors these days will remove spurious spaces and we should strive to remove them where possibly no?

1765–1766

Again, removing spurious whitespaces?

packages/Python/lldbsuite/test/make/Makefile.rules

197

Ok, I can remove this.

packages/Python/lldbsuite/test/plugins/builder_base.py

204

All other "def buildXXX" functions do this, so I am following the convention.

packages/Python/lldbsuite/test/test_categories.py

27

So what is the suggestion here?

source/Plugins/SymbolFile/DWARF/DIERef.cpp

51–68

Will do

source/Plugins/SymbolFile/DWARF/DWARFDataExtractor.h

51

It doesn't. I checked the docs that show up in the doxygen in http://lldb.llvm.org/cpp_reference/html/index.html and these trailing comments don't hose it up.

source/Plugins/SymbolFile/DWARF/DWARFDebugInfo.cpp

121

I am copying the first loop.

source/Plugins/SymbolFile/DWARF/DWARFDebugInfo.h

66

No reason. IMHO STL is fine unless there is a real reason llvm adt is better?

[labath]

So, I tried this out on my machine, and got just one failure:

======================================================================
FAIL: test_with_run_command_dwarf_type_units (TestInlinedBreakpoints.InlinedBreakpointsTestCase)
   Test 'b basic_types.cpp:176' does break (where int.cpp includes basic_type.cpp).
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1827, in dwarf_type_units_test_method
    return attrvalue(self)
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/functionalities/breakpoint/inlined_breakpoints/TestInlinedBreakpoints.py", line 24, in test_with_run_command
    self.inlined_breakpoints()
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/functionalities/breakpoint/inlined_breakpoints/TestInlinedBreakpoints.py", line 46, in inlined_breakpoints
    self, "basic_type.cpp", self.line, num_expected_locations=0)
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 376, in run_break_set_by_file_and_line
    num_locations=num_expected_locations)
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 572, in check_breakpoint_result
    out_num_locations))
AssertionError: False is not True : Expecting 0 locations, got 1.

However, I am not so sure about the proliferation of debug info variants that we seem to be having. Right now we have two outstanding patches wanting to add a debug variant, which would bring the total amount of variants per test to 6. I'm not sure this is a tractable way forward.
IIUC, clang only puts "non-trivial" types in type units. I'm not sure how many of our tests even define classes/structs (i.e., for how many tests this debug info variant would actually be testing something new).

source/Plugins/SymbolFile/DWARF/DWARFCompileUnit.h

49

It seems weird to have an IsTypeUnit method on a compile unit, when a dwarf compile unit is clearly not a type unit. Would some of the refactors that Jan is doing (I haven't looked at those too closely as it's not really my area, but I know he is adding a DwarfUnit class) help with this?

[emaste]

packages/Python/lldbsuite/test/make/Makefile.rules

197

FreeBSD still defaults to DWARF2 at present.

[clayborg]

In D32167#1019467, @labath wrote:

So, I tried this out on my machine, and got just one failure:

======================================================================
FAIL: test_with_run_command_dwarf_type_units (TestInlinedBreakpoints.InlinedBreakpointsTestCase)
   Test 'b basic_types.cpp:176' does break (where int.cpp includes basic_type.cpp).
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1827, in dwarf_type_units_test_method
    return attrvalue(self)
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/functionalities/breakpoint/inlined_breakpoints/TestInlinedBreakpoints.py", line 24, in test_with_run_command
    self.inlined_breakpoints()
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/functionalities/breakpoint/inlined_breakpoints/TestInlinedBreakpoints.py", line 46, in inlined_breakpoints
    self, "basic_type.cpp", self.line, num_expected_locations=0)
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 376, in run_break_set_by_file_and_line
    num_locations=num_expected_locations)
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 572, in check_breakpoint_result
    out_num_locations))
AssertionError: False is not True : Expecting 0 locations, got 1.

I saw this too. When I ran _any_ of the variants manually, they all failed to set any breakpoints. Not sure this test is testing what it claims to be testing.

However, I am not so sure about the proliferation of debug info variants that we seem to be having. Right now we have two outstanding patches wanting to add a debug variant, which would bring the total amount of variants per test to 6. I'm not sure this is a tractable way forward.
IIUC, clang only puts "non-trivial" types in type units. I'm not sure how many of our tests even define classes/structs (i.e., for how many tests this debug info variant would actually be testing something new).

We could check after the build if the binary for the test even has a .debug_types sections and if not abort? That was we don't have to mark up the test in any way (like "this tests has structures or enums"), but we could ensure we test with .debug_types when needed.

[clayborg]

In D32167#1019504, @clayborg wrote:

In D32167#1019467, @labath wrote:

So, I tried this out on my machine, and got just one failure:

======================================================================
FAIL: test_with_run_command_dwarf_type_units (TestInlinedBreakpoints.InlinedBreakpointsTestCase)
   Test 'b basic_types.cpp:176' does break (where int.cpp includes basic_type.cpp).
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1827, in dwarf_type_units_test_method
    return attrvalue(self)
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/functionalities/breakpoint/inlined_breakpoints/TestInlinedBreakpoints.py", line 24, in test_with_run_command
    self.inlined_breakpoints()
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/functionalities/breakpoint/inlined_breakpoints/TestInlinedBreakpoints.py", line 46, in inlined_breakpoints
    self, "basic_type.cpp", self.line, num_expected_locations=0)
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 376, in run_break_set_by_file_and_line
    num_locations=num_expected_locations)
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 572, in check_breakpoint_result
    out_num_locations))
AssertionError: False is not True : Expecting 0 locations, got 1.

I saw this too. When I ran _any_ of the variants manually, they all failed to set any breakpoints. Not sure this test is testing what it claims to be testing.

Actually I can repro this and will look into it

[labath]

In D32167#1019504, @clayborg wrote:

In D32167#1019467, @labath wrote:

However, I am not so sure about the proliferation of debug info variants that we seem to be having. Right now we have two outstanding patches wanting to add a debug variant, which would bring the total amount of variants per test to 6. I'm not sure this is a tractable way forward.
IIUC, clang only puts "non-trivial" types in type units. I'm not sure how many of our tests even define classes/structs (i.e., for how many tests this debug info variant would actually be testing something new).

We could check after the build if the binary for the test even has a .debug_types sections and if not abort? That was we don't have to mark up the test in any way (like "this tests has structures or enums"), but we could ensure we test with .debug_types when needed.

That would help somewhat, but that's still a very "shotgun" approach to testing this. What I would try to do (and I think this is what Davide had in mind as well) is to identify the cases that are interesting from the type unit perspective (a single class in type unit, two classes in type units referencing each other, class referencing an enum, etc. (I'm not really sure about what all that goes into a type unit)) and then write specific tests for that (or reuse existing ones). For example, plenty of tests do breakpoint setting or unwinding and those probably don't have to be tested separately with type units (or maybe one test that checks that setting breakpoint on a method in type unit is sufficient).

[jingham]

Right now, which tests are "run all variants" is represents more history than policy. Opt out of all variants is opt in, and while I at least kick out obvious tests that don't really need to run all variants when I touch them, but I don't think we have done a careful audit. I think the general idea of running variants is good, but we really should reduce the number of tests we run this way to the ones that actually test things that might break that way.

Note, I have (though very very occasionally) seen a dsymutil bug that broke line tables. So it would be good to keep a few "set a breakpoint and run to it" tests just to sanity check this. But most tests that just run somewhere and print a backtrace or view an integer variable or such-like do not need to run all variants.

[davide]

In D32167#1019621, @labath wrote:

In D32167#1019504, @clayborg wrote:

In D32167#1019467, @labath wrote:

However, I am not so sure about the proliferation of debug info variants that we seem to be having. Right now we have two outstanding patches wanting to add a debug variant, which would bring the total amount of variants per test to 6. I'm not sure this is a tractable way forward.
IIUC, clang only puts "non-trivial" types in type units. I'm not sure how many of our tests even define classes/structs (i.e., for how many tests this debug info variant would actually be testing something new).

We could check after the build if the binary for the test even has a .debug_types sections and if not abort? That was we don't have to mark up the test in any way (like "this tests has structures or enums"), but we could ensure we test with .debug_types when needed.

That would help somewhat, but that's still a very "shotgun" approach to testing this. What I would try to do (and I think this is what Davide had in mind as well) is to identify the cases that are interesting from the type unit perspective (a single class in type unit, two classes in type units referencing each other, class referencing an enum, etc. (I'm not really sure about what all that goes into a type unit)) and then write specific tests for that (or reuse existing ones). For example, plenty of tests do breakpoint setting or unwinding and those probably don't have to be tested separately with type units (or maybe one test that checks that setting breakpoint on a method in type unit is sufficient).

Precisely. I'm afraid what's proposed here is too coarse to be useful.

[labath]

In D32167#1019635, @jingham wrote:

Note, I have (though very very occasionally) seen a dsymutil bug that broke line tables. So it would be good to keep a few "set a breakpoint and run to it" tests just to sanity check this. But most tests that just run somewhere and print a backtrace or view an integer variable or such-like do not need to run all variants.

The thing here is that I'm not sure everything can be captured by a single "debug info variant" dimension. The current dimensions kinda make sense, as in they alter the way pretty much all of debug info is accessed (we can think of "dwarf" as the basic one, "dsym" and "dwo" are similar in that they place everything in a separate file, I'm not sure about gmodules, but that's because I have no idea on how it works). However, I'm not sure the same goes for type units. For one they are not completely mutually exclusive with the existing variants, so you can have type-units+regular dwarf and type units+dwo (maybe even type units+dsym?). But obviously we can't add another "type unit" dimension and do a cartesian product....

[clayborg]

I am afraid of the opposite: we test what we think we need to test and our simple tests cases don't adequately test the feature we are adding. I can certainly limit the testing to very simple test cases with a one test for a class and one for a enum, but that wouldn't have caught the issues I ran into with static variables in classes that I fixed before submitting this. My point is it is hard to figure out what a compiler, or debugger might hose up without testing all possible cases. Please chime in with opinions as I will go with the flow on this.

[clayborg]

In D32167#1019701, @labath wrote:

In D32167#1019635, @jingham wrote:

Note, I have (though very very occasionally) seen a dsymutil bug that broke line tables. So it would be good to keep a few "set a breakpoint and run to it" tests just to sanity check this. But most tests that just run somewhere and print a backtrace or view an integer variable or such-like do not need to run all variants.

The thing here is that I'm not sure everything can be captured by a single "debug info variant" dimension. The current dimensions kinda make sense, as in they alter the way pretty much all of debug info is accessed (we can think of "dwarf" as the basic one, "dsym" and "dwo" are similar in that they place everything in a separate file, I'm not sure about gmodules, but that's because I have no idea on how it works). However, I'm not sure the same goes for type units. For one they are not completely mutually exclusive with the existing variants, so you can have type-units+regular dwarf and type units+dwo (maybe even type units+dsym?). But obviously we can't add another "type unit" dimension and do a cartesian product....

if you see how horrible type units are in their implementation, you might change your mind. If one has a positive view of type units, you might think "I get one type and one type only and that can only mean good things". I reality this feature's implementation is really messy and there are so many areas that can screw up. For instance if you have a class that has static members, you will get a nice full type in the type unit that defines everything in the class. But with type units you can't refer to anything within the type unit itself other than the top level type itself (the struct/union/class or enum). So you now need to declare a DW_TAG_variable that points to the static member of the class so you can give it an address... What happens? The type is duplicated in each compile unit that uses it and a skeleton incomplete version of the class is represented in the compile unit DWARF (not in a type unit). The top level DW_TAG_structure or DW_TAG_class has a DW_AT_signature that points to the type in the type unit, but the incomplete struct/class has _just_ the static members inside of it (again, in the compile unit, not in the type unit). Then you have a DW_TAG_variable that points to the DW_TAG_member within the incomplete struct or class because there is no way to point to anything within the type if the type is in a type unit. So I believe that we do need to fully test types and run the .debug_types as a separate variant. Each compiler might also do things a bit differently as there is nothing in the .debug_types specification that says that this is how things need to be. And that is just one thing I found while debugging .debug_types... I am sure there are more skeletons in the closet. If we want to support this feature we need to test it. I come to this conclusion after years of dealing with many different "this DWARF does this cool new way to do things" where it has bit us in the past and things break.

[vsk]

I'm concerned about the time it takes to run the test suite. If we can get decent test coverage of this patch without adding a new debug info flavor, that'd be ideal.

packages/Python/lldbsuite/test/lldbtest.py

704

No, we generally avoid making changes that are unrelated to the patch, as they make patch review and subsequent merges harder.

source/Plugins/SymbolFile/DWARF/DWARFDataExtractor.h

58

Please clang-format your diffs.

[clayborg]

clang format
fixed comments
removed white space
fixed functionalities/breakpoint/inlined_breakpoints/TestInlinedBreakpoints.py failures

[aprantl]

packages/Python/lldbsuite/test/test_categories.py

27

I would mention type units in the comment rather than -fdebug-types-section.

source/Plugins/SymbolFile/DWARF/DWARFDebugInfo.cpp

97–98

We probably should convert this to an early exit.

103–104

Perhaps add a comment explaining what is being done here?

118

I find hard to follow. What do you think about:

while (debug_types_data.ValidOffset(offset)) {
     cu_sp = DWARFCompileUnit::Extract(m_dwarf2Data, debug_types_data,
                                            &offset, true)));
     if (!cu_sp)
        return;
     m_type_sig_to_cu_index[cu_sp->GetTypeSignature()] =
         m_compile_units.size();
     m_compile_units.push_back(cu_sp);
     offset = cu_sp->GetNextCompileUnitOffset();
   }

?

Also, shouldn't the error be propagated?

[clayborg]

Address Adrian's comments.

[labath]

In D32167#1019702, @clayborg wrote:

I am afraid of the opposite: we test what we think we need to test and our simple tests cases don't adequately test the feature we are adding. I can certainly limit the testing to very simple test cases with a one test for a class and one for a enum, but that wouldn't have caught the issues I ran into with static variables in classes that I fixed before submitting this. My point is it is hard to figure out what a compiler, or debugger might hose up without testing all possible cases. Please chime in with opinions as I will go with the flow on this.

I don't think anyone here is suggesting to test less. The question we are asking is if running all 1000 or so tests doesn't just giv a false sense of security (and makes the testing process longer). A specially crafted test can trigger more corner cases and make it easier to debug future failures that a generic test. If the case of a static variable in a class is interesting, then maybe a test where you have two static variables from the same class defined in two different compilation units (and maybe referenced from a third one) is interesting as well. I'm pretty sure we don't have a test like that right now. Another interesting case which would not be tested in the "separate flavour" mode is the mixed-flags case: have part of your module compiled with type units, part without (and maybe a third part with type units and fission)..

Running the entire dotest test suite in -fdebug-types-section is certainly a good way to catch problems, but it's not the way to write regression tests. FWIW, the way I plan to test the new accelerator tables when I get to the lldb part is to run dotest with the new flags locally during development, use the failures to identify interesting test vectors, and then write regular tests to trigger these.

[probinson]

In D32167#1020032, @labath wrote:

Running the entire dotest test suite in -fdebug-types-section is certainly a good way to catch problems, but it's not the way to write regression tests.

Is there testing in place that is more serious/thorough than the normal regression testing? It might be reasonable to do the full cartesian set at a slower pace. For example LLVM has the concept of "expensive checks" and there are a small number of bots out there that enable them. But it's not something anybody runs normally.

[clayborg]

In D32167#1020032, @labath wrote:

In D32167#1019702, @clayborg wrote:

I am afraid of the opposite: we test what we think we need to test and our simple tests cases don't adequately test the feature we are adding. I can certainly limit the testing to very simple test cases with a one test for a class and one for a enum, but that wouldn't have caught the issues I ran into with static variables in classes that I fixed before submitting this. My point is it is hard to figure out what a compiler, or debugger might hose up without testing all possible cases. Please chime in with opinions as I will go with the flow on this.

I don't think anyone here is suggesting to test less. The question we are asking is if running all 1000 or so tests doesn't just giv a false sense of security (and makes the testing process longer). A specially crafted test can trigger more corner cases and make it easier to debug future failures that a generic test. If the case of a static variable in a class is interesting, then maybe a test where you have two static variables from the same class defined in two different compilation units (and maybe referenced from a third one) is interesting as well. I'm pretty sure we don't have a test like that right now. Another interesting case which would not be tested in the "separate flavour" mode is the mixed-flags case: have part of your module compiled with type units, part without (and maybe a third part with type units and fission)..

Running the entire dotest test suite in -fdebug-types-section is certainly a good way to catch problems, but it's not the way to write regression tests. FWIW, the way I plan to test the new accelerator tables when I get to the lldb part is to run dotest with the new flags locally during development, use the failures to identify interesting test vectors, and then write regular tests to trigger these.

For the accelerator tables you could always just manually index the DWARF and compare the results to the new accelerator tables as a unit test. No further testing needed?

[clayborg]

Would everyone be ok with a solution where we leave the .debug_types tests in as a flavor, but we check the binary after creating it to ensure it has a .debug_types section. If it doesn't, then we skip the test (or pass it?), and if it does, then we go ahead and test it? The only overhead that is added then is some building. Many of the tests don't have types that go into the .debug_types section, so this would be an easy way for us to test .debug_types when it is present and just build and skip if it doesn't. No maintenance needed on any tests, we will just detect and enable if needed?

[clayborg]

Ping for my last comment

[labath]

Sorry, I've been waiting to give others (Davide?) a chance to express their opinion. I personally don't think having a new debug info flavour is a good idea. Tests written specifically to test this functionality will be easier to maintain and debug when they break. And keeping adding new flavours for every compiler option is not a viable long term strategy.

[davide]

As already pointed out, I think this feature should be thought again & have more focused testing. We can have a meeting/discussion about this, and I need to think about it more.
But what you're proposing is basically a sledgehammer and I don't feel confident about having the patch committed in its current state.

[jasonmolenda]

I think we're being a little hasty here. Greg's last suggestion is worth investigation -- how many tests would actually be run for this new variant?

Greg, maybe you could remove the make clean targets from packages/Python/lldbsuite/test/plugins/builder_base.py (I think that'll do it), do a testsuite run to create all the test binaries, and then find out how how many tests are actually going to be run here?

Pavel is right that the explosion of debug variations will quickly become untenable, even with bots, and I'm not sure what a good solution to that is. Running the debug information parsers against the wide variety of programs we include in the testsuite is a good way to test them. This is a problem we should discuss, but if the practical impact of adding the extra testing for .debug_types is that we run 50 tests through an additional variant, I think it is premature to draw the line here and say this must be solved now.

[jankratochvil]

In D32167#1024546, @labath wrote:

I personally don't think having a new debug info flavour is a good idea. Tests written specifically to test this functionality will be easier to maintain and debug when they break. And keeping adding new flavours for every compiler option is not a viable long term strategy.

Here I disagree, from GDB testsuite experience I find more wider testing always better as it will better catch bugs even in unrelated new functionalities being implemented in the future. One can never guess all the combinations in advance - if one could then one needs no regression testing at all. There are more than enough CPU cycles available for testing. I find rather a serious problem racy==unreliable test results which waste human efforts investigating the results and with more parallel testing (make -j) and virtualized infrastructure the racy results happen even more often. As an example one racy testcase I idenitifed is:

packages/Python/lldbsuite/test/functionalities/signal/handle-segv/TestHandleSegv.py
line 33: AssertionError: 4 != 5 (eStateLaunching != eStateStopped)

I think at least these test are racy:

BreakpointAfterJoinTestCase-test
CreateDuringStepTestCase-test_step_inst_with
HandleSegvTestCase-test_inferior_handle_sigsegv
ReturnValueTestCase-test_vector_values
TestTargetSymbolsBuildidSymlink-test_target_symbols_buildid
ThreadSpecificBreakPlusConditionTestCase-test_python
ThreadSpecificBreakTestCase-test_python
ThreadStateTestCase-test_process_interrupt

[davide]

In D32167#1026762, @jankratochvil wrote:

In D32167#1024546, @labath wrote:

I personally don't think having a new debug info flavour is a good idea. Tests written specifically to test this functionality will be easier to maintain and debug when they break. And keeping adding new flavours for every compiler option is not a viable long term strategy.

Here I disagree, from GDB testsuite experience I find more wider testing always better as it will better catch bugs even in unrelated new functionalities being implemented in the future. One can never guess all the combinations in advance - if one could then one needs no regression testing at all. There are more than enough CPU cycles available for testing. I find rather a serious problem racy==unreliable test results which waste human efforts investigating the results and with more parallel testing (make -j) and virtualized infrastructure the racy results happen even more often. As an example one racy testcase I idenitifed is:

packages/Python/lldbsuite/test/functionalities/signal/handle-segv/TestHandleSegv.py
line 33: AssertionError: 4 != 5 (eStateLaunching != eStateStopped)

I think at least these test are racy:

BreakpointAfterJoinTestCase-test
CreateDuringStepTestCase-test_step_inst_with
HandleSegvTestCase-test_inferior_handle_sigsegv
ReturnValueTestCase-test_vector_values
TestTargetSymbolsBuildidSymlink-test_target_symbols_buildid
ThreadSpecificBreakPlusConditionTestCase-test_python
ThreadSpecificBreakTestCase-test_python
ThreadStateTestCase-test_process_interrupt

I'm not necessarily convinced having another column in the matrix is something that can substitute targeted testing.
If you're talking about something complementary, I think we might consider it, as a follow up.
Also, thanks for bringing the problem of the racy tests up. Adding another row in the matrix increases the chances of test failures. I think we might consider addressing the unreliability before making such a big change. Do you have a way of reproducing? [or even better, investigating]

Our bots have been green for a while (we tried to investigate known issues http://lab.llvm.org:8080/green/view/LLDB/) but we can only address what we know :)

[jankratochvil]

In D32167#1026779, @davide wrote:

Do you have a way of reproducing?

It just happens for me each time - on Fedora 27 x86_64 on 16-core (32HT) 2-node NUMA machine having env var MAKEFLAGS=-j32 (and tuned-adm throughput-performance if it matters). Comparing multiple runs makes some tests {Expected,Unexpected,}{Success,Failure,Error} without changing anything.

[or even better, investigating]

When the testcase is then ran separately it sure succeeds as the machine then has no load that time. Maybe one could make some race reproducer helper (such as was GDB read1) but I should analyse more LLDB testsuite races to know the common racy issues in LLDB testsuite.

[davide]

In D32167#1026780, @jankratochvil wrote:

In D32167#1026779, @davide wrote:

Do you have a way of reproducing?

It just happens for me each time - on Fedora 27 x86_64 on 16-core (32HT) 2-node NUMA machine having env var MAKEFLAGS=-j32 (and tuned-adm throughput-performance if it matters). Comparing multiple runs makes some tests {Expected,Unexpected,}{Success,Failure,Error} without changing anything.

[or even better, investigating]

When the testcase is then ran separately it sure succeeds as the machine then has no load that time. Maybe one could make some race reproducer helper (such as was GDB read1) but I should analyse more LLDB testsuite races to know the common racy issues in LLDB testsuite.

That sounds a great helper. Any chance you're interested in working on that? I don't have access to a machine beefy enough to repro the failures, I'm afraid.

[jankratochvil]

In D32167#1026781, @davide wrote:

Any chance you're interested in working on that?

Yes, it is in the shortterm or rather midterm plan.

[clayborg]

In D32167#1026779, @davide wrote:

In D32167#1026762, @jankratochvil wrote:

In D32167#1024546, @labath wrote:

I personally don't think having a new debug info flavour is a good idea. Tests written specifically to test this functionality will be easier to maintain and debug when they break. And keeping adding new flavours for every compiler option is not a viable long term strategy.

Here I disagree, from GDB testsuite experience I find more wider testing always better as it will better catch bugs even in unrelated new functionalities being implemented in the future. One can never guess all the combinations in advance - if one could then one needs no regression testing at all. There are more than enough CPU cycles available for testing. I find rather a serious problem racy==unreliable test results which waste human efforts investigating the results and with more parallel testing (make -j) and virtualized infrastructure the racy results happen even more often. As an example one racy testcase I idenitifed is:

packages/Python/lldbsuite/test/functionalities/signal/handle-segv/TestHandleSegv.py
line 33: AssertionError: 4 != 5 (eStateLaunching != eStateStopped)

I think at least these test are racy:

BreakpointAfterJoinTestCase-test
CreateDuringStepTestCase-test_step_inst_with
HandleSegvTestCase-test_inferior_handle_sigsegv
ReturnValueTestCase-test_vector_values
TestTargetSymbolsBuildidSymlink-test_target_symbols_buildid
ThreadSpecificBreakPlusConditionTestCase-test_python
ThreadSpecificBreakTestCase-test_python
ThreadStateTestCase-test_process_interrupt

I'm not necessarily convinced having another column in the matrix is something that can substitute targeted testing.

If I would have done targeted testing I wouldn't have caught the breakpoint issue that came up with this patch. So I agree with jankratochvil that it is best to test this all the way by adding another row. There is no way we can guess what certain compilers will do and and all the different permutations this can cause. Not sure where ".debug_types" lands in terms of popularity, but I would guess it comes in above other DWARF flavors (dwo, dwz). Can I at least land this and then we can work on future improvements to the rows required. This is only currently enabled on linux where we tend to have fast buildbots. Darwin compilers don't support this with mach-o due to the 255 section limit that mach-o suffers from.

[jankratochvil]

source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.cpp

597

I do not like this DWARFDataExtractor::m_start modification, it sort of corrupts the DataExtractor and various operations stop working then - such as DWARFDataExtractor::GetByteSize(). DWZ patch makes from current dw_offset_t a virtual (remapped) offset and introduces new physical file section offset which is looked up for data extraction. The file offset is represented as DWARFFileOffset in D40474, instead of bool m_is_dwz; there could be some enum { DEBUG_INFO, DEBUG_TYPES, DWZ_DEBUG_INFO } m_where; instead.

[clayborg]

Is there a link to some documentation that explains what DWZ does? I found a bunch of blurbs and man pages, but nothing useful. Nothing is found when I look for "DWZ" in either the DWARF 4 or DWARF 5 spec.

source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.cpp

597

This means that this diff doesn't affect all of the other DWARF code. Nothing in .debug_types will refer to anything else (not DW_FORM_ref_addr, or any external references). So this trick allows us to just treat .debug_info as if .debug_types was appended to the end since nothing in .debug_types refers to any DIE outside of its type unit. This also mirrors what will actually happen with DWARF5 when all of the data is contained inside of the .debug_info section. This allows each DIE to have a unique "ID". Any other change requires a lot of changes to the DWARF parser and logic. So I actually like this feature. We can fix the GetByteSize() if needed. Basically every object in DWARf right now must be able to make a unique 64 bit unsigned integer ID in some way that we can get back to that info and partially parse more. These are handed out as lldb::user_id_t values for types, functions and more. Each flavor of DWARF will encode what they want into here. The normal DWARF it is just the absolute offset within the .debug_info. With .debug_types we just add the size of the .debug_info to the ID. For DWARF in .o files on Darwin, we encode the compile unit index into the top 32 bits and the DIE offset into the lower, DWO does something just as DWZ will need to. DWARFFileOffset doesn't mean much if there are multiple files. We have many competing type uniquing/debug info size reduction strategies being employed here. I can't believe we have DWO, DWZ, and debug types... But we have to make them all work. We can't just use the absolute file offset because DWO used external files where the file offsets could be the same in the external .o files... Not sure how this works with DWZ or what the best option is. I will read up on DWZ so I can propose some viable options. But each new flavor of the day that gets added the DWARF parser is adding a bunch of logic and edge cases. If two technologies (DWZ + DWO, DWZ + debug_types, etc) are used together, we need to ensure they can.

[clayborg]

Found the DWZ stuff:
http://www.dwarfstd.org/ShowIssue.php?issue=120604.1&type=open

[clayborg]

For DWZ files, it seems types can be contained in external DWARF files and are referred to using new forms. The right way to fix this in LLDB it to load each of these external files as a separate DWZ file (no changes needed to anything) and then teach the SymbolFileDWARF that refers to these files to import the types. If we have 24 shared libraries that use a single DWZ, we shouldn't be loading it more than once and inlining the DWARF, we should be using the AST importer to import the type from that file into the AST for the current DWARF file. We do have done this with -gmodules already, so DWZ shouldn't be that different. Let me know if I am missing anything.

[jankratochvil]

In D32167#1062618, @clayborg wrote:

we should be using the AST importer to import the type from that file into the AST for the current DWARF file. We do have done this with -gmodules already, so DWZ shouldn't be that different.

Is AST usable for imported declarations of DW_TAG_variable, DW_TAG_subprogram, DW_TAG_enumeration_type+DW_TAG_enumerators etc.? Still imported DW_TAG_partial_unit cannot contain any code or data addresses which may make it easier for AST.

source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.cpp

597

Any other change requires a lot of changes to the DWARF parser and logic. So I actually like this feature.

I agree it is a fine quick&dirty hack. Just if my DWZ support gets accepted later anyway then this .debug_types feature could be implemented by its framework in a clean way (as a regular DIEs remapping which is required for DWZ anyway).

If two technologies (DWZ + DWO, DWZ + debug_types, etc) are used together, we need to ensure they can.

DWZ + DWO do not make sense to me. I haven't tried to use DWZ for DWO but DWZ finds common DWARF subtrees typically across CUs so it would not find much.

DWZ + debug_types is explicitly supported by the DWZ tool although that is IMO for compatibility only, DWZ can make the common type references slightly smaller than debug_types. I will sure need to implement debug_types support into my DWZ-for-LLDB patchset later.

[jankratochvil]

In D32167#1062614, @clayborg wrote:

Found the DWZ stuff:
http://www.dwarfstd.org/ShowIssue.php?issue=120604.1&type=open

BTW the current DWZ tool supports only DWARF-4 and the DWZ tags are non-standard GNU extensions. Therefore DW_FORM_GNU_ref_alt in existing DWZ files is DW_FORM_ref_alt in the proposal you found. I am not aware of any real documentation for this DWARF-4 GNU extension. But DWARF-5 further modified the proposal so it became DW_FORM_ref_sup4+DW_FORM_ref_sup8 in the DWARF-5 standard.

My DWZ-for-LLDB patchset implements the DWARF-4 non-standard GNU extensions as it is used in RHEL-7 (and possibly RHEL-8, not sure yet) which will be supported at least till 2024. It is also present in all recent Fedoras. Currently the DWZ tool has not yet been ported to DWARF-5, Mark Wielaard from Red Hat plans to do so.

[jankratochvil]

source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.cpp

597

For DWARF in .o files on Darwin, we encode the compile unit index into the top 32 bits and the DIE offset into the lower

BTW that prevents implementing DWARF64 so I wanted to prevent using the 32/32 split. Currently Fedora/RHEL already contains files with .debug_info size near the 32-bit limit:

http://ftp.muni.cz/pub/linux/fedora/linux/development/rawhide/Everything/x86_64/debug/tree/Packages/q/qt5-qtwebkit-debuginfo-5.212.0-0.20.alpha2.fc29.x86_64.rpm
/usr/lib/debug/usr/lib64/libQt5WebKit.so.5.212.0-5.212.0-0.20.alpha2.fc29.x86_64.debug
.debug_info size = 0x9bff1b08 = 2.4GiB

Although that is because neither DWZ (as it would run out of memory) nor .debug_types (as it is expected DWZ will handle the duplicities) are applied and so all CUs are separate and even if it was >4GB there would still be no need for 64-bit DW_FORM_ref_addr. Still we are already close to the 32-bit limit, also shifting the split for example to 40/24 would make some limit on number of CUs etc.

[clayborg]

Update patch to work with changes from:

https://reviews.llvm.org/D45170

[probinson]

I am looking at making the LLVM parser handle type units across v4/v5, and the trick of pretending .debug_types is pasted onto the end of .debug_info seems like a pretty convenient fiction. It unifies DIE handling, and still allows the dumper to distinguish which bits are in which section, which is traditionally how the dumper UI thinks about DWARF. So as long as you and Jan are on the same page there, I'm happy to follow suit.

source/Plugins/SymbolFile/DWARF/DIERef.cpp

53

... the type is

[clayborg]

Ping. I would like to get this in. As for testing, the full matrix should be run on .debug_types as it is the main form of type uniquing that is defined by the DWARF spec. Any formats that are fully accepted and written into the DWARF spec should be fully tested to ensure everything works. .debug_types is only enabled on linux right now and our build bots are fast on these systems.

[labath]

Maybe we could start by splitting of the ObjectFile recognition code for the debug_types section into a separate patch. That should be sufficiently non-controversial and it would reduce the number of files touched by this patch a bit.

[clayborg]

In D32167#1087627, @labath wrote:

Maybe we could start by splitting of the ObjectFile recognition code for the debug_types section into a separate patch. That should be sufficiently non-controversial and it would reduce the number of files touched by this patch a bit.

Sure thing. Anything to get this moving.

[clayborg]

In D32167#1087627, @labath wrote:

Maybe we could start by splitting of the ObjectFile recognition code for the debug_types section into a separate patch. That should be sufficiently non-controversial and it would reduce the number of files touched by this patch a bit.

I broke out the object file parsing code:

https://reviews.llvm.org/D46529

[labath]

source/Plugins/SymbolFile/DWARF/DWARFDataExtractor.h

35–58

This looks like a thing we could focus on next. Sliding the contents of the data buffer seems like a useful fiction, but the way it is implemented now, it relies on undefined behavior (out-of-range pointers, possibly even wrapping the address space) and it violates the invariants that we have some far been assuming about data extractors (namely, that for a DataExtractor, all offsets from 0 to GetByteSize()-1 are dereferencable). For example, for a "slid" data extractor:

• ValidOffset(0) will return true, even though that is clearly not the case
• Append(data2) will try to access data at *m_start and probably crash
• so will Checksum()

Since there seems to be a consensus (at least, nobody has proposed an alternative solution) that this is the way we should treat type units, I think it makes sense to spend some time to make sure we built it on solid foundations. To me, this sliding seems like a fundamental change in the nature of the data extractors, and so I think it should be done in the base DataExtractor class. So I'd propose to move this into a separate patch, where we can make sure that all other DataExtractor APIs do something sensible even if the extractor has been "slid".

[clayborg]

Broke up with patch further by submitting:

[PATH] D46529: Add support to object files for accessing the .debug_types section
https://reviews.llvm.org/D46529

[PATCH] D46606: General cleanup to minimize the .debug_types patch
https://reviews.llvm.org/D46606

This makes this patch easier to read and focus only on the .debug_types patch

[labath]

The patch looks much better now, but I think we still need to discuss the data extractor sliding issue, as right now that's very hacky.

[probinson]

So, I've been attacking the type-units-in-.debug_info problem in LLVM, and what I've done (so far) in the LLVM parser is to factor out a DWARFUnitHeader class, and made DWARFUnit derive from that. This makes it feasible to parse a unit header before deciding what kind of unit we are looking at, which is necessary when .debug_info contains both type and compile units. I haven't posted that patch yet as it's an NFC prep for more work getting to the point of actually allowing mix-n-match units in .debug_info sections. But I'll post that patch today so people can at least look at it.

Mainly Greg and I should not take diverging approaches when we can converge instead.

[clayborg]

In D32167#1094173, @labath wrote:

The patch looks much better now, but I think we still need to discuss the data extractor sliding issue, as right now that's very hacky.

It makes things just work with .debug_info as is pretends that the .debug_types is just appended to the end of the .debug_info (offset wise). All info in .debug_types has no DW_FORM_ref_addr or any references that are outside of the current type unit so the info in .debug_types is position independent. I would like to keep this in DWARFDataExtractor only if possible as it is hacky and won't work in a general case. Feel free to suggest any other solution that doesn't require checked if the data is from .debug_types and adding an offset all over the code, but I couldn't think of one.

[clayborg]

I really need to get this in as well. It is a very important feature and I am trying to do the right thing and get it into open source so we don't end up keeping the patch locally. So any help on expediting any comments and iterations would be great.

[probinson]

I can't coment on the xcode and python parts. The C++ all looks plausible.
I think the "sliding" trick will be trickier in the LLVM parser, as it has to handle .o files (which might have multiple instances of .debug_types in v4, and multiple instances of .debug_info in v5) but is conceptually nice. The sliding should be kept to DWARFDataExtractor and not pushed down into DataExtractor, IMO, because the problem is (I think) pretty specific to DWARF.

[labath]

The thing here is that I don't see how can be done in a non-hacky way in DWARFDataExtractor while DWARFDataExtractor remains a (subclass of) DataExtractor. And it is a hack because as soon as you slide the extractor, a number of its functions become booby-trapped (explode as soon as you call them) or just return nonsensical results. And I'd like us to implement it in a way that it is not a hack, on any level.

I see several ways of achieving this (in decreasing order of intrusiveness):

1. do this in the DataExtractor class. This would require us to change (extend) the definition of the class. Right now, I would define this class as "a view of a range of bytes with a bunch of utility functions for accessing them". After this change it would become something like "a view of a range of bytes with a bunch of utility functions for accessing them, where the accessor functions automatically apply a bias to all offsets". That's precisely the kind of change you are making now, only the difference would be that we would state this up-front rather than doing it behind the DataExtractor's back and hoping it works. In practice, this would be as simple as adding another offset_t m_bias; field to the class and going through it's methods to make sure the do something reasonable with it. "Something reasonable" would generally mean "subtract it from the input offset", but there are a couple of functions that would need to do something different, (e.g., ValidOffset() would return true for values from m_bias to m_bias + (m_end-m_start) -1 because those are the only valid offsets, etc.). This is my preferred solution because it provides with a consistent DataExtractor interface (and in terms of code churn, I don't think it's harder to implement than any of the other solutions). It's true that dwarf type units will probably be the only user of these "slid" extractors, but I don't think that matters as their functionality can be defined in a clear and self-consistent manner without any references to dwarf.

2. Add the m_bias field to the DWARFDataExtractor class, but stop the class from publicly inheriting from DataExtractor (it can inherit privately, or have it as a member). The result of this is the same as the first option - we get to go through the DataExtractor interface and make sure all its functions do something sensible. The difference is that we may not need to go through that all methods, only those that are used for dwarf parsing; and the sliding trick can then be kept dwarf-specific.

3. Keep DWARFDataExtractor as-is, and add a new SlidingDWARFExtractor class instead (which will then to the m_bias trick and so on); then convert all places that need this to use the new class instead. This is a variation on the second option. I don't see it as particularly interesting, as it the DWARFDataExtractor is already a very small class, but it would still provide us self-consistent APIs everywhere.

I can also see some kind of a solution involving a ConcatenatingDataExtractor, which would contain two or more DataExtractors as it's backing store and then pretend they represent one continuous address space from 0 to sum(sizes) (no sliding necessary). This would achieve what you wanted at a higher level. In a way it would be nicer because there would be a single canonical bytestream for each symbol file and you wouldn't have to ask each DIE for it's own extractor, but I expect it to be a bit harder to implement, and may have a runtime cost.

The next solution which does not require sliding is to modify your previous patch D46606 and insert some kind of offset fixup code there. It seems that in every place you call the new GetData() function, you have an offset around (makes sense, as the reason you are calling GetData is because you want to parse something). We could modify that function to take a virtual (slid) offset as a parameter, and return an offset which is suitable for usage in the returned extractor. So, the usage would become something like std::tie(extractor, real_offset) = die.GetData(virtual_offset);. I'm not sure how nice this will end up in practice, but it's a possibility.

Let me know what you think of these. I'm open to other options as well. The thing I'd like avoid is an implementation that relies on putting objects in an invalid state or invoking undefined behavior.

[probinson]

I haven't given Pavel's comments as much thought as they deserve, but here are some of mine.

It's a worthwhile point that a class ought to represent some useful and reasonably well defined abstraction. DataExtractor has one, and DWARFDataExtractor builds on that (at least in the LLVM parser) by handling relocations; the most reasonable way to handle multiple buffers would be with multiple extractors, building on top of what's there rather than hacking the insides.

One aspect I am looking at is that the LLVM parser needs to handle .o files, which can have multiple .debug_types (in v4) or .debug_info (in v5) sections, because the units are broken up into COMDATs. So, the dumper needs to be able to handle an arbitrary number of either kind of section, where LLDB needs at most one of each section. One way of dealing with all this is to have a vector or deque of sections, each section has its DWARFDataExtractor and knows its own "base offset" which would be the sum of the sizes of the preceding sections. I don't know how awkward it might be to turn this into a meta-extractor that hid details from consumers, but then I think it probably would not be all that hard to have consumers be aware that there can be multiple lumps of DWARF and get the offsets right on their own.

[clayborg]

After reading Pavel's and Paul's comments, I will come up with a solution that allows multiple sections to be combined into one large section. That way it will be all legal and easy to extend. If we have just .debug_info, then we can use a base class that just calls directly through to DWARFDataExtractor. If we have more that one section that need to act as one section, I will make a new class that can put N sections together. I'll post a patch on its own that implements this with tests and we can iterate on that and make it work for all of our needs, then switch this patch over to use it. Sound good?

[probinson]

Go for it. It sounds like an abstraction that hides the multiple-section nature of the beast will particularly help LLDB; on the LLVM side the dumper will want to remain a bit more cognizant of the underlying sections. But we can surely leverage ideas from each other, which will ultimately help with converging on the Grand Unified Parser.

[labath]

Sounds like a plan.

[jankratochvil]

Just a ping, that ConcatenatingDataExtractor is still not coded, even as some draft patch? Just checking to prevent duplicating some existing work.
FYI providing a rebase with few simple conflicts resolved. I had to drop conflicting lldb.xcodeproj/project.pbxproj changes there.

[clayborg]

In D32167#1212783, @jankratochvil wrote:

Just a ping, that ConcatenatingDataExtractor is still not coded, even as some draft patch? Just checking to prevent duplicating some existing work.
FYI providing a rebase with few simple conflicts resolved. I had to drop conflicting lldb.xcodeproj/project.pbxproj changes there.

I have stalled on this due to performance. Anything I do will ruin the performance of 99% of the DWARF consumers for .debug_types which is rarely used by our clients. Right now we have one data extractor with only a virtual destructor.

[jankratochvil]

Just to clarify - I guess DWARFConcatenatingDataExtractor should replace all uses of DWARFDataExtractor in the DWARF/ subdirectory, whenever lldb::offset_t offset is used, right?
As the other option would be to remap just lldb::user_id_t but that would not solve much.
BTW my D40474 patch: DWZ is using DWARF references in both main file .debug_info and in DWZ common file .debug_info so at least one of them always needs to get the read-in DWARF offsets adjusted anyway in DWARFFormValue::Reference(). Then I needed adjustments also in DWARFDebugInfo::GetCompileUnitAranges() as the DWZ common file I had to place from offset 0 so that it can be shared between multiple DWARF files using it, therefore .debug_info of the main files (users of DWZ common file) need to be shifted.
Also the DWZ common file is a separate DWARFDebugInfo so there needs to be some dispatcher (DWARFDebugInfo::DWZRedirect() in my D40474 DWZ patch) for it anyway, which is why I do not find DWARFConcatenatingDataExtractor as much a win for DWZ.
OTOH I do not think the DWARF parsing performance matters too much after .debug_names (or the Apple index which I haven't/cannot test) is used, why it should?

[jankratochvil]

In D32167#1215821, @clayborg wrote:

I have stalled on this due to performance. Anything I do will ruin the performance of 99% of the DWARF consumers for .debug_types which is rarely used by our clients.

I have tried to implement it with acceptable performance as D51578.

[jankratochvil]

BTW I would welcome some direction forward (for our internal yearly team meeting) how to unblock this patch (and assuming DWZ D40474). I can code some other of the solutions described above by Pavel Labath or to performance-improve more D51578 (although then I can come with more DWARF performance improvements, at least re-introduce atomics to aid my mutexes from D40470).

[jankratochvil]

I could not updates this revision so an updated version of this patch is now: D54670

include/lldb/lldb-forward.h

 class DataBuffer;
 class DataEncoder;
 class DataExtractor;
 class Debugger;
 class Declaration;
 class DiagnosticManager;
 class Disassembler;
 class DumpValueObjectOptions;
+class DWARFDataExtractor;
 class DynamicCheckerFunctions;
 class DynamicLoader;
 class Editline;
 class EmulateInstruction;
 class Status;
 class EvaluateExpressionOptions;
 class Event;
 class EventData;

lldb.xcodeproj/project.pbxproj

/* Begin PBXBuildFile section */
     2654A6901E552ED500DA1013 /* VASprintf.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 2654A68F1E552ED500DA1013 /* VASprintf.cpp */; };
     2656BBC31AE0739C00441749 /* libedit.dylib in Frameworks */ = {isa = PBXBuildFile; fileRef = 26F5C32A10F3DFDD009D5894 /* libedit.dylib */; };
     2656BBC41AE073A800441749 /* libncurses.dylib in Frameworks */ = {isa = PBXBuildFile; fileRef = 2670F8111862B44A006B332C /* libncurses.dylib */; };
     2656BBC51AE073AD00441749 /* libpanel.dylib in Frameworks */ = {isa = PBXBuildFile; fileRef = 260157C41885F4FF00F875CF /* libpanel.dylib */; };
     2656BBC61AE073B500441749 /* libz.dylib in Frameworks */ = {isa = PBXBuildFile; fileRef = 966C6B7818E6A56A0093F5EC /* libz.dylib */; };
     2657AFB71B86910100958979 /* CompilerDeclContext.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 2657AFB61B86910100958979 /* CompilerDeclContext.cpp */; };
     2660AAB914622483003A9694 /* LLDBWrapPython.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 26A4EEB511682AAC007A372A /* LLDBWrapPython.cpp */; settings = {COMPILER_FLAGS = "-Dregister="; }; };
     26651A18133BF9E0005B64B7 /* Opcode.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 26651A17133BF9DF005B64B7 /* Opcode.cpp */; };
+    26652E662076BC0A0053A2FC /* DWARFTypeUnit.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 26652E642076BC0A0053A2FC /* DWARFTypeUnit.cpp */; };
+    26652E672076BC0A0053A2FC /* DWARFTypeUnit.h in Headers */ = {isa = PBXBuildFile; fileRef = 26652E652076BC0A0053A2FC /* DWARFTypeUnit.h */; };
     2666ADC61B3CB675001FAFD3 /* DynamicLoaderHexagonDYLD.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 2666ADC11B3CB675001FAFD3 /* DynamicLoaderHexagonDYLD.cpp */; };
     2666ADC81B3CB675001FAFD3 /* HexagonDYLDRendezvous.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 2666ADC31B3CB675001FAFD3 /* HexagonDYLDRendezvous.cpp */; };
     2668020E115FD12C008E1FE4 /* lldb-defines.h in Headers */ = {isa = PBXBuildFile; fileRef = 26BC7C2510F1B3BC00F91463 /* lldb-defines.h */; settings = {ATTRIBUTES = (Public, ); }; };
     2668020F115FD12C008E1FE4 /* lldb-enumerations.h in Headers */ = {isa = PBXBuildFile; fileRef = 26BC7C2610F1B3BC00F91463 /* lldb-enumerations.h */; settings = {ATTRIBUTES = (Public, ); }; };
     26680214115FD12C008E1FE4 /* lldb-types.h in Headers */ = {isa = PBXBuildFile; fileRef = 26BC7C2910F1B3BC00F91463 /* lldb-types.h */; settings = {ATTRIBUTES = (Public, ); }; };
     26680219115FD13D008E1FE4 /* SBBreakpoint.h in Headers */ = {isa = PBXBuildFile; fileRef = 9AF16A9E11402D69007A7B3F /* SBBreakpoint.h */; settings = {ATTRIBUTES = (Public, ); }; };
     2668021A115FD13D008E1FE4 /* SBBreakpointLocation.h in Headers */ = {isa = PBXBuildFile; fileRef = 9AF16CC611408686007A7B3F /* SBBreakpointLocation.h */; settings = {ATTRIBUTES = (Public, ); }; };
     2668021B115FD13D008E1FE4 /* SBBroadcaster.h in Headers */ = {isa = PBXBuildFile; fileRef = 9A9830F31125FC5800A56CB0 /* SBBroadcaster.h */; settings = {ATTRIBUTES = (Public, ); }; };
/* Begin PBXFileReference section */
     2657AFB61B86910100958979 /* CompilerDeclContext.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = CompilerDeclContext.cpp; path = source/Symbol/CompilerDeclContext.cpp; sourceTree = "<group>"; };
     265ABF6210F42EE900531910 /* DebugSymbols.framework */ = {isa = PBXFileReference; lastKnownFileType = wrapper.framework; name = DebugSymbols.framework; path = /System/Library/PrivateFrameworks/DebugSymbols.framework; sourceTree = "<absolute>"; };
     265E9BE1115C2BAA00D0DCCB /* debugserver.xcodeproj */ = {isa = PBXFileReference; lastKnownFileType = "wrapper.pb-project"; name = debugserver.xcodeproj; path = tools/debugserver/debugserver.xcodeproj; sourceTree = "<group>"; };
     2660D9F611922A1300958FBD /* StringExtractor.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = StringExtractor.cpp; path = source/Utility/StringExtractor.cpp; sourceTree = "<group>"; };
     2660D9FE11922A7F00958FBD /* ThreadPlanStepUntil.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = ThreadPlanStepUntil.cpp; path = source/Target/ThreadPlanStepUntil.cpp; sourceTree = "<group>"; };
     26651A14133BEC76005B64B7 /* lldb-public.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; name = "lldb-public.h"; path = "include/lldb/lldb-public.h"; sourceTree = "<group>"; };
     26651A15133BF9CC005B64B7 /* Opcode.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = Opcode.h; path = include/lldb/Core/Opcode.h; sourceTree = "<group>"; };
     26651A17133BF9DF005B64B7 /* Opcode.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = Opcode.cpp; path = source/Core/Opcode.cpp; sourceTree = "<group>"; };
+    26652E642076BC0A0053A2FC /* DWARFTypeUnit.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = DWARFTypeUnit.cpp; sourceTree = "<group>"; };
+    26652E652076BC0A0053A2FC /* DWARFTypeUnit.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = DWARFTypeUnit.h; sourceTree = "<group>"; };
     2665CD0D15080846002C8FAE /* Makefile */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.make; path = Makefile; sourceTree = "<group>"; };
     2666ADC11B3CB675001FAFD3 /* DynamicLoaderHexagonDYLD.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = DynamicLoaderHexagonDYLD.cpp; sourceTree = "<group>"; };
     2666ADC21B3CB675001FAFD3 /* DynamicLoaderHexagonDYLD.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = DynamicLoaderHexagonDYLD.h; sourceTree = "<group>"; };
     2666ADC31B3CB675001FAFD3 /* HexagonDYLDRendezvous.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = HexagonDYLDRendezvous.cpp; sourceTree = "<group>"; };
     2666ADC41B3CB675001FAFD3 /* HexagonDYLDRendezvous.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = HexagonDYLDRendezvous.h; sourceTree = "<group>"; };
     26680207115FD0ED008E1FE4 /* LLDB.framework */ = {isa = PBXFileReference; explicitFileType = wrapper.framework; includeInIndex = 0; path = LLDB.framework; sourceTree = BUILT_PRODUCTS_DIR; };
     2669415B1A6DC2AB0063BE93 /* CMakeLists.txt */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text; name = CMakeLists.txt; path = "tools/lldb-mi/CMakeLists.txt"; sourceTree = SOURCE_ROOT; };
     2669415E1A6DC2AB0063BE93 /* lldb-Info.plist */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text.plist.xml; name = "lldb-Info.plist"; path = "tools/lldb-mi/lldb-Info.plist"; sourceTree = SOURCE_ROOT; };
260C89B210F57C5600BB2B04 /* DWARF */ = {
         260C89CF10F57C5600BB2B04 /* DWARFDefines.cpp */,
         260C89D010F57C5600BB2B04 /* DWARFDefines.h */,
         266E82951B8CE346008FCA06 /* DWARFDIE.h */,
         266E82961B8CE3AC008FCA06 /* DWARFDIE.cpp */,
         260C89D110F57C5600BB2B04 /* DWARFDIECollection.cpp */,
         260C89D210F57C5600BB2B04 /* DWARFDIECollection.h */,
         260C89D310F57C5600BB2B04 /* DWARFFormValue.cpp */,
         260C89D410F57C5600BB2B04 /* DWARFFormValue.h */,
+        26652E642076BC0A0053A2FC /* DWARFTypeUnit.cpp */,
+        26652E652076BC0A0053A2FC /* DWARFTypeUnit.h */,
         7F2AAA5920601BE000A422D8 /* DWARFUnit.cpp */,
         7F2AAA5820601BDF00A422D8 /* DWARFUnit.h */,
         26A0DA4D140F721D006DA411 /* HashedNameToDIE.h */,
         26109B3B1155D70100CC3529 /* LogChannelDWARF.cpp */,
         26109B3C1155D70100CC3529 /* LogChannelDWARF.h */,
         2618D9EA12406FE600F2B8FE /* NameToDIE.cpp */,
         2618D957124056C700F2B8FE /* NameToDIE.h */,
         260C89D910F57C5600BB2B04 /* SymbolFileDWARF.cpp */,
2689FFC813353D7A00698AC0 /* Headers */ = {
         AF8AD62F1BEC28A400150209 /* PlatformAppleTVSimulator.h in Headers */,
         238F2BA91D2C85FA001FF92A /* StructuredDataDarwinLog.h in Headers */,
         AF8AD63A1BEC28C400150209 /* PlatformRemoteAppleWatch.h in Headers */,
         257906651BD5AFD000178368 /* Acceptor.h in Headers */,
         238F2BA21D2C835A001FF92A /* SystemRuntime.h in Headers */,
         260A63171861008E00FECF8E /* IOHandler.h in Headers */,
         267F68581CC02EAE0086832B /* RegisterContextPOSIX_s390x.h in Headers */,
         6D0F614F1C80AB0C00A4ECEE /* JavaLanguageRuntime.h in Headers */,
+        26652E672076BC0A0053A2FC /* DWARFTypeUnit.h in Headers */,
         AF27AD561D3603EA00CF2833 /* DynamicLoaderDarwin.h in Headers */,
         AFCB2BBE1BF577F40018B553 /* PythonExceptionState.h in Headers */,
         267F684B1CC02DED0086832B /* ABISysV_s390x.h in Headers */,
         AF8AD6311BEC28A400150209 /* PlatformAppleWatchSimulator.h in Headers */,
       );
       runOnlyForDeploymentPostprocessing = 0;
     };
 /* End PBXHeadersBuildPhase section */
2689FFC613353D7A00698AC0 /* Sources */ = {
         AFC2DCF01E6E2FD200283714 /* VMRange.cpp in Sources */,
         2689002A13353E0400698AC0 /* Address.cpp in Sources */,
         8CF02AE919DCC01900B14BE0 /* InstrumentationRuntime.cpp in Sources */,
         2689002B13353E0400698AC0 /* AddressRange.cpp in Sources */,
         236124A41986B4E2004EFC37 /* IOObject.cpp in Sources */,
         2689002C13353E0400698AC0 /* AddressResolver.cpp in Sources */,
         945261C01B9A11FC00BF138D /* LibCxx.cpp in Sources */,
         949EEDB11BA7672D008C63CF /* NSDictionary.cpp in Sources */,
+        26652E662076BC0A0053A2FC /* DWARFTypeUnit.cpp in Sources */,
         2689002D13353E0400698AC0 /* AddressResolverFileLine.cpp in Sources */,
         2689002E13353E0400698AC0 /* AddressResolverName.cpp in Sources */,
         250D6AE31A9679440049CC70 /* FileSystem.cpp in Sources */,
         2689003113353E0400698AC0 /* Broadcaster.cpp in Sources */,
         49DCF702170E70120092F75E /* Materializer.cpp in Sources */,
         2689003213353E0400698AC0 /* Communication.cpp in Sources */,
         2689003313353E0400698AC0 /* Connection.cpp in Sources */,
         9404957A1BEC497E00926025 /* NSError.cpp in Sources */,

packages/Python/lldbsuite/test/lldbinline.py

class InlineTest(TestBase):
     @add_test_categories(["gmodules"])
     def __test_with_gmodules(self):
         self.using_dsym = False
         self.BuildMakefile()
         self.build()
         self.do_test()
     __test_with_gmodules.debug_info = "gmodules"
 
+    @add_test_categories(["dwarf_type_units"])
+    def __test_with_dwarf_type_units(self):
+        self.using_dsym = False
+        self.BuildMakefile()
+        self.build()
+        self.do_test()
+    __test_with_dwarf_type_units.debug_info = "dwarf_type_units"
+
     def execute_user_command(self, __command):
         exec(__command, globals(), locals())
 
     def do_test(self):
         exe = self.getBuildArtifact("a.out")
         source_files = [f for f in os.listdir(self.getSourceDir())
                         if source_type(f)]
         target = self.dbg.CreateTarget(exe)
def MakeInlineTest(__file, __globals, decorators=None):
     test.test_with_dsym = ApplyDecoratorsToFunction(
         test._InlineTest__test_with_dsym, decorators)
     test.test_with_dwarf = ApplyDecoratorsToFunction(
         test._InlineTest__test_with_dwarf, decorators)
     test.test_with_dwo = ApplyDecoratorsToFunction(
         test._InlineTest__test_with_dwo, decorators)
     test.test_with_gmodules = ApplyDecoratorsToFunction(
         test._InlineTest__test_with_gmodules, decorators)
+    test.test_with_dwarf_type_units = ApplyDecoratorsToFunction(
+        test._InlineTest__test_with_dwarf_type_units, decorators)
 
     # Add the test case to the globals, and hide InlineTest
     __globals.update({test_name: test})
 
     # Keep track of the original test filename so we report it
     # correctly in test results.
     test.test_filename = __file
     return test

packages/Python/lldbsuite/test/lldbtest.py

class Base(unittest2.TestCase):
     def getBuildDirBasename(self):
         return self.__class__.__module__ + "." + self.testMethodName
 
     def getBuildDir(self):
         """Return the full path to the current test."""
         return os.path.join(os.environ["LLDB_BUILD"], self.mydir,
                             self.getBuildDirBasename())
     
      
     def makeBuildDir(self):
         """Create the test-specific working directory, deleting any previous
         contents."""
         # See also dotest.py which sets up ${LLDB_BUILD}.
         bdir = self.getBuildDir()
         if os.path.isdir(bdir):
             shutil.rmtree(bdir)
         lldbutil.mkdir_p(bdir)
def buildGModules(
             raise Exception("NO_DEBUG_INFO_TESTCASE must build with buildDefault")
 
         module = builder_module()
         dictionary = lldbplatformutil.finalize_build_dictionary(dictionary)
         if not module.buildGModules(self, architecture, compiler,
                                     dictionary, testdir, testname):
             raise Exception("Don't know how to build binary with gmodules")
 
+    def buildDwarfTypeUnits(
+            self,
+            architecture=None,
+            compiler=None,
+            dictionary=None,
+            clean=True):
+        """Platform specific way to build binaries with DWARF type units."""
+        testdir = self.mydir
+        testname = self.getBuildDirBasename()
+        if self.getDebugInfo() != "dwarf_type_units":
+            raise Exception("NO_DEBUG_INFO_TESTCASE must build with buildDefault")
+
+        module = builder_module()
+        dictionary = lldbplatformutil.finalize_build_dictionary(dictionary)
+        if not module.buildDwarfTypeUnits(self, architecture, compiler,
+                                          dictionary, clean, testdir, testname):
+            raise Exception("Don't know how to build binary with DWARF type units")
+
     def buildGo(self):
         """Build the default go binary.
         """
         exe = self.getBuildArtifact("a.out")
         system([[which('go'), 'build -gcflags "-N -l" -o %s main.go' % exe]])
 
     def signBinary(self, binary_path):
         if sys.platform.startswith("darwin"):
def __new__(cls, name, bases, attrs):
                 all_dbginfo_categories = set(test_categories.debug_info_categories)
                 categories = set(
                     getattr(
                         attrvalue,
                         "categories",
                         [])) & all_dbginfo_categories
                 if not categories:
                     categories = all_dbginfo_categories
 
                 for cat in categories:
                     @decorators.add_test_categories([cat])
                     @wraps(attrvalue)
                     def test_method(self, attrvalue=attrvalue):
                         return attrvalue(self)
 
                     method_name = attrname + "_" + cat
                     test_method.__name__ = method_name
                     test_method.debug_info = cat
def build(
         elif self.getDebugInfo() == "dsym":
             return self.buildDsym(architecture, compiler, dictionary)
         elif self.getDebugInfo() == "dwarf":
             return self.buildDwarf(architecture, compiler, dictionary)
         elif self.getDebugInfo() == "dwo":
             return self.buildDwo(architecture, compiler, dictionary)
         elif self.getDebugInfo() == "gmodules":
             return self.buildGModules(architecture, compiler, dictionary)
+        elif self.getDebugInfo() == "dwarf_type_units":
+            return self.buildDwarfTypeUnits(architecture, compiler, dictionary)
         else:
             self.fail("Can't build for debug info: %s" % self.getDebugInfo())
 
     def run_platform_command(self, cmd):
         platform = self.dbg.GetSelectedPlatform()
         shell_command = lldb.SBPlatformShellCommand(cmd)
         err = platform.Run(shell_command)
         return (err, shell_command.GetStatus(), shell_command.GetOutput())

packages/Python/lldbsuite/test/make/Makefile.rules

else
   ifeq "$(ARCH)" "aarch64"
     override ARCH :=
     override ARCHFLAG :=
   endif
   ifeq "$(ARCH)" "arm"
     override ARCH :=
     override ARCHFLAG :=
   endif
   ifeq "$(ARCH)" "s390x"
     override ARCH :=
     override ARCHFLAG :=
   endif
   ifeq "$(findstring mips,$(ARCH))" "mips"
     override ARCHFLAG := -
   endif
 
   ifeq "$(SPLIT_DEBUG_SYMBOLS)" "YES"

 CLANG_MODULE_CACHE_DIR := module-cache
 
 MANDATORY_MODULE_BUILD_CFLAGS := -fmodules -gmodules -fmodules-cache-path=$(CLANG_MODULE_CACHE_DIR)
 
 ifeq "$(MAKE_GMODULES)" "YES"
   CFLAGS += $(MANDATORY_MODULE_BUILD_CFLAGS)
 endif
 
+MANDATORY_DWARF_TYPE_UNITS_CFLAGS := -fdebug-types-section
+
+ifeq "$(DWARF_TYPE_UNITS)" "YES"
+  CFLAGS += $(MANDATORY_DWARF_TYPE_UNITS_CFLAGS)
+endif
+
 CXXFLAGS += -std=c++11 $(CFLAGS)
 LD = $(CC)
 LDFLAGS ?= $(CFLAGS)
 LDFLAGS += $(LD_EXTRAS) $(ARCH_LDFLAGS)
 ifeq (,$(filter $(OS), Windows_NT Android Darwin))
   ifneq (,$(filter YES,$(ENABLE_THREADS)))
     LDFLAGS += -pthread
   endif

packages/Python/lldbsuite/test/plugins/builder_base.py

def buildGModules(
         architecture=None,
         compiler=None,
         dictionary=None,
         testdir=None,
         testname=None):
     """Build the binaries with dwarf debug info."""
     commands = []
     commands.append(getMake(testdir, testname) +
                     ["MAKE_DSYM=NO",
                      "MAKE_GMODULES=YES",
                      getArchSpec(architecture),
                      getCCSpec(compiler),
                      getCmdLine(dictionary)])
 
     lldbtest.system(commands, sender=sender)
     # True signifies that we can handle building with gmodules.
     return True
 
 
+def buildDwarfTypeUnits(
+        sender=None,
+        architecture=None,
+        compiler=None,
+        dictionary=None,
+        clean=True,
+        testdir=None,
+        testname=None):
+    """Build the binaries with type units (type in  a .debug_types section)."""
+    commands = []
+    if clean:
+        commands.append(getMake(testdir, testname) +
+                        ["clean", getCmdLine(dictionary)])
+    commands.append(getMake(testdir, testname) +
+                    ["MAKE_DSYM=NO",
+                     "DWARF_TYPE_UNITS=YES",
+                     getArchSpec(architecture),
+                     getCCSpec(compiler),
+                     getCmdLine(dictionary)])
+
+    lldbtest.system(commands, sender=sender)
+    # True signifies that we can handle building with type units.
+    return True
+
+
 def cleanup(sender=None, dictionary=None):
     """Perform a platform-specific cleanup after the test."""
     return True

packages/Python/lldbsuite/test/test_categories.py

 
 # Third-party modules
 
 # LLDB modules
 from lldbsuite.support import gmodules
 
 
 debug_info_categories = [
-    'dwarf', 'dwo', 'dsym', 'gmodules'
+    'dwarf', 'dwo', 'dsym', 'gmodules', 'dwarf_type_units'
 ]
 
 all_categories = {
     'dataformatters': 'Tests related to the type command and the data formatters subsystem',
     'dwarf': 'Tests that can be run with DWARF debug information',
     'dwo': 'Tests that can be run with DWO debug information',
     'dsym': 'Tests that can be run with DSYM debug information',
     'gmodules': 'Tests that can be run with -gmodules debug information',
+    'dwarf_type_units' : 'Tests that can be run with DWARF type units',
     'expression': 'Tests related to the expression parser',
     'libc++': 'Test for libc++ data formatters',
     'objc': 'Tests related to the Objective-C programming language support',
     'pyapi': 'Tests related to the Python API',
     'basic_process': 'Basic process execution sniff tests.',
     'cmdline': 'Tests related to the LLDB command-line interface',
     'dyntype': 'Tests related to dynamic type support',
     'stresstest': 'Tests related to stressing lldb limits',
if category == "dwo":
         return platform in ["linux", "freebsd"]
     elif category == "dsym":
         return platform in ["darwin", "macosx", "ios", "watchos", "tvos", "bridgeos"]
     elif category == "gmodules":
         # First, check to see if the platform can even support gmodules.
         if platform not in ["linux", "freebsd", "darwin", "macosx", "ios", "watchos", "tvos", "bridgeos"]:
             return False
         return gmodules.is_compiler_clang_with_gmodules(compiler_path)
+    elif category == "dwarf_type_units":
+        return platform in ["linux"]
     return True
 
 
 def validate(categories, exact_match):
     """
     For each category in categories, ensure that it's a valid category (if exact_match is false,
     unique prefixes are also accepted). If a category is invalid, print a message and quit.
        If all categories are valid, return the list of categories. Prefixes are expanded in the

source/Plugins/SymbolFile/DWARF/CMakeLists.txt

add_lldb_library(lldbPluginSymbolFileDWARF PLUGIN
   DWARFDebugMacinfoEntry.cpp
   DWARFDebugRanges.cpp
   DWARFDeclContext.cpp
   DWARFDefines.cpp
   DWARFDIE.cpp
   DWARFDIECollection.cpp
   DWARFFormValue.cpp
   DWARFUnit.cpp
+  DWARFTypeUnit.cpp
   HashedNameToDIE.cpp
   LogChannelDWARF.cpp
   NameToDIE.cpp
   SymbolFileDWARF.cpp
   SymbolFileDWARFDwo.cpp
   SymbolFileDWARFDwoDwp.cpp
   SymbolFileDWARFDwp.cpp
   SymbolFileDWARFDebugMap.cpp

source/Plugins/SymbolFile/DWARF/DIERef.cpp

 //===-- DIERef.cpp ----------------------------------------------*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "DIERef.h"
 #include "DWARFUnit.h"
+#include "DWARFTypeUnit.h"
 #include "DWARFDebugInfo.h"
 #include "DWARFFormValue.h"
 #include "SymbolFileDWARF.h"
 #include "SymbolFileDWARFDebugMap.h"
 
 DIERef::DIERef()
     : cu_offset(DW_INVALID_OFFSET), die_offset(DW_INVALID_OFFSET) {}
 
DIERef::DIERef(lldb::user_id_t uid, SymbolFileDWARF *dwarf)
   }
 }
 
 DIERef::DIERef(const DWARFFormValue &form_value)
     : cu_offset(DW_INVALID_OFFSET), die_offset(DW_INVALID_OFFSET) {
   if (form_value.IsValid()) {
     const DWARFUnit *dwarf_cu = form_value.GetCompileUnit();
     if (dwarf_cu) {
+      // Replace the compile unit with the type signature compile unit
+      // and DIE for the type signature. When a type is referred to by a
+      // DW_FORM_ref_sig8 form, the real information for the type in
+      // contained in a DW_TAG_type_unit.
+      if (form_value.Form() == DW_FORM_ref_sig8) {
+        uint64_t type_sig = form_value.Unsigned();
+        auto debug_info = dwarf_cu->GetSymbolFileDWARF()->DebugInfo();
+        auto tu = debug_info->GetTypeUnitForSignature(type_sig);
+        if (tu) {
+          cu_offset = tu->GetOffset();
+          die_offset = tu->GetTypeUnitDIEOffset();
+        }
+      } else {
         if (dwarf_cu->GetBaseObjOffset() != DW_INVALID_OFFSET)
           cu_offset = dwarf_cu->GetBaseObjOffset();
         else
           cu_offset = dwarf_cu->GetOffset();
       }
+    }
     die_offset = form_value.Reference();
   }
 }
 
 lldb::user_id_t DIERef::GetUID(SymbolFileDWARF *dwarf) const {
   //----------------------------------------------------------------------
   // Each SymbolFileDWARF will set its ID to what is expected.
   //

source/Plugins/SymbolFile/DWARF/DWARFASTParserClang.cpp

if (log) {
 
       dwarf->GetObjectFile()->GetModule()->LogMessage(
           log, "SymbolFileDWARF::ParseType (die = 0x%8.8x, decl_ctx = %p (die "
                "0x%8.8x)) %s name = '%s')",
           die.GetOffset(), static_cast<void *>(context),
           context_die.GetOffset(), die.GetTagAsCString(), die.GetName());
     }
     Type *type_ptr = dwarf->GetDIEToType().lookup(die.GetDIE());
-    TypeList *type_list = dwarf->GetTypeList();
     if (type_ptr == NULL) {
+
+      // If we have .debug_types defer to the complete version in the
+      // .debug_types section, not the broken incomplete definition in real
+      // compile units that are only there so addresses can be assigned to
+      // static values.
+      auto signature_die = die.GetAttributeValueAsReferenceDIE(DW_AT_signature);
+      if (signature_die) {
+        type_sp = ParseTypeFromDWARF(sc, signature_die, log, type_is_new_ptr);
+        if (type_sp) {
+          dwarf->GetDIEToType()[die.GetDIE()] = type_sp.get();
+          return type_sp;
+        }
+      }
+
       if (type_is_new_ptr)
         *type_is_new_ptr = true;
 
       const dw_tag_t tag = die.Tag();
 
       bool is_forward_declaration = false;
       DWARFAttributes attributes;
       const char *type_name_cstr = NULL;

         }
 
         if (symbol_context_scope != NULL) {
           type_sp->SetSymbolContextScope(symbol_context_scope);
         }
 
         // We are ready to put this type into the uniqued list up at the module
         // level
-        type_list->Insert(type_sp);
+        dwarf->GetTypeList()->Insert(type_sp);
 
         dwarf->GetDIEToType()[die.GetDIE()] = type_sp.get();
       }
     } else if (type_ptr != DIE_IS_BEING_PARSED) {
       type_sp = type_ptr->shared_from_this();
     }
   }
   return type_sp;

source/Plugins/SymbolFile/DWARF/DWARFCompileUnit.h

 #define SymbolFileDWARF_DWARFCompileUnit_h_
 
 #include "DWARFUnit.h"
 
 class DWARFCompileUnit : public DWARFUnit {
   friend class DWARFUnit;
 
 public:
-  static DWARFUnitSP Extract(SymbolFileDWARF *dwarf2Data,
-                             const lldb_private::DWARFDataExtractor &debug_info,
-                             lldb::offset_t *offset_ptr);
+  static DWARFCompileUnitSP Extract(SymbolFileDWARF *dwarf2Data,
   void Dump(lldb_private::Stream *s) const override;
 
   //------------------------------------------------------------------
   /// Get the data that contains the DIE information for this unit.
   ///
   /// @return
   ///   The correct data (.debug_types for DWARF 4 and earlier, and
   ///   .debug_info for DWARF 5 and later) for the DIE information in
uint32_t GetHeaderByteSize() const override {
     return m_is_dwarf64 ? 23 : 11;
   }
 
 private:
   DWARFCompileUnit(SymbolFileDWARF *dwarf2Data);
   DISALLOW_COPY_AND_ASSIGN(DWARFCompileUnit);
 };
 
 #endif // SymbolFileDWARF_DWARFCompileUnit_h_

source/Plugins/SymbolFile/DWARF/DWARFCompileUnit.cpp

 //===-- DWARFCompileUnit.cpp ------------------------------------*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "DWARFCompileUnit.h"
 
 #include "SymbolFileDWARF.h"
 
 using namespace lldb;
 using namespace lldb_private;
 
-extern int g_verbose;
-
 DWARFCompileUnit::DWARFCompileUnit(SymbolFileDWARF *dwarf2Data)
     : DWARFUnit(dwarf2Data) {}
 
-DWARFUnitSP DWARFCompileUnit::Extract(SymbolFileDWARF *dwarf2Data,
+DWARFCompileUnitSP DWARFCompileUnit::Extract(SymbolFileDWARF *dwarf2Data,
                                       const DWARFDataExtractor &debug_info,
                                       lldb::offset_t *offset_ptr) {
   // std::make_shared would require the ctor to be public.
-  std::shared_ptr<DWARFCompileUnit> cu_sp(new DWARFCompileUnit(dwarf2Data));
-
-  cu_sp->m_offset = *offset_ptr;
-
-  if (debug_info.ValidOffset(*offset_ptr)) {
-    dw_offset_t abbr_offset;
-    const DWARFDebugAbbrev *abbr = dwarf2Data->DebugAbbrev();
-    cu_sp->m_length = debug_info.GetDWARFInitialLength(offset_ptr);
-    cu_sp->m_is_dwarf64 = debug_info.IsDWARF64();
-    cu_sp->m_version = debug_info.GetU16(offset_ptr);
-    abbr_offset = debug_info.GetDWARFOffset(offset_ptr);
-    cu_sp->m_addr_size = debug_info.GetU8(offset_ptr);
-
-    bool length_OK =
-        debug_info.ValidOffset(cu_sp->GetNextCompileUnitOffset() - 1);
-    bool version_OK = SymbolFileDWARF::SupportedVersion(cu_sp->m_version);
-    bool abbr_offset_OK =
-        dwarf2Data->get_debug_abbrev_data().ValidOffset(abbr_offset);
-    bool addr_size_OK = (cu_sp->m_addr_size == 4) || (cu_sp->m_addr_size == 8);
-
-    if (length_OK && version_OK && addr_size_OK && abbr_offset_OK &&
-        abbr != NULL) {
-      cu_sp->m_abbrevs = abbr->GetAbbreviationDeclarationSet(abbr_offset);
-      return cu_sp;
-    }
-
-    // reset the offset to where we tried to parse from if anything went wrong
-    *offset_ptr = cu_sp->m_offset;
-  }
-
-  return nullptr;
+  DWARFCompileUnitSP cu_sp(new DWARFCompileUnit(dwarf2Data));
+  if (cu_sp->ExtractHeader(dwarf2Data, debug_info, offset_ptr))
+  return DWARFCompileUnitSP();
 }
 
 void DWARFCompileUnit::Dump(Stream *s) const {
   s->Printf("0x%8.8x: Compile Unit: length = 0x%8.8x, version = 0x%4.4x, "
             "abbr_offset = 0x%8.8x, addr_size = 0x%2.2x (next CU at "
             "{0x%8.8x})\n",
             m_offset, m_length, m_version, GetAbbrevOffset(), m_addr_size,
             GetNextCompileUnitOffset());
 }
 
 
 const lldb_private::DWARFDataExtractor &DWARFCompileUnit::GetData() const {
   return m_dwarf->get_debug_info_data();
 }

source/Plugins/SymbolFile/DWARF/DWARFDataExtractor.h

public:
   uint64_t GetDWARFInitialLength(lldb::offset_t *offset_ptr) const;
 
   dw_offset_t GetDWARFOffset(lldb::offset_t *offset_ptr) const;
 
   size_t GetDWARFSizeofInitialLength() const { return m_is_dwarf64 ? 12 : 4; }
   size_t GetDWARFSizeOfOffset() const { return m_is_dwarf64 ? 8 : 4; }
   bool IsDWARF64() const { return m_is_dwarf64; }
 
+  //------------------------------------------------------------------
+  /// Slide the data in the buffer so that access to the data will
+  /// start at offset \a offset.
+  ///
+  /// This is currently used to provide access to the .debug_types
+  /// section and pretend it starts at at offset of the size of the
+  /// .debug_info section. This allows a minimally invasive change to
+  /// add support for .debug_types by allowing all DIEs to have unique
+  /// offsets and thus allowing no code to change in the DWARF parser.
+  /// Modifying the offsets in the .debug_types doesn't affect
+  /// anything because since all info in the .debug_types is type unit
+  /// relative and no types within a type unit can refer to any DIEs
+  /// outside of the type unit without using DW_AT_signature. It also
+  /// sets us up to move to DWARF5 where there is no .debug_types
+  /// section as compile units and type units are in the .debug_info.
+  ///
+  /// @param[in] offset
+  ///   The amount to slide the data contents by.
+  //------------------------------------------------------------------
+  void OffsetData(lldb::offset_t offset) {
+    if (GetByteSize())
+      m_start -= offset;
+  }
+
 protected:
   mutable bool m_is_dwarf64;
 };
 }
 
 #endif // liblldb_DWARFDataExtractor_h_

source/Plugins/SymbolFile/DWARF/DWARFDebugInfo.h

 //===-- DWARFDebugInfo.h ----------------------------------------*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef SymbolFileDWARF_DWARFDebugInfo_h_
 #define SymbolFileDWARF_DWARFDebugInfo_h_
 
 #include <map>
+#include <unordered_map>
 #include <vector>
 
 #include "DWARFUnit.h"
 #include "DWARFDIE.h"
 #include "SymbolFileDWARF.h"
 #include "lldb/Core/STLUtils.h"
 #include "lldb/lldb-private.h"
 
+class DWARFTypeUnit;
+
 typedef std::multimap<const char *, dw_offset_t, CStringCompareFunctionObject>
     CStringToDIEMap;
 typedef CStringToDIEMap::iterator CStringToDIEMapIter;
 typedef CStringToDIEMap::const_iterator CStringToDIEMapConstIter;
 
 class DWARFDebugInfo {
 public:
   typedef dw_offset_t (*Callback)(SymbolFileDWARF *dwarf2Data,
                                   DWARFUnit *cu,
                                   DWARFDebugInfoEntry *die,
                                   const dw_offset_t next_offset,
                                   const uint32_t depth, void *userData);
 
   DWARFDebugInfo();
   void SetDwarfData(SymbolFileDWARF *dwarf2Data);
 
   size_t GetNumCompileUnits();
   bool ContainsCompileUnit(const DWARFUnit *cu) const;
   DWARFUnit *GetCompileUnitAtIndex(uint32_t idx);
+  DWARFTypeUnit *GetTypeUnitForSignature(uint64_t type_sig);
   DWARFUnit *GetCompileUnit(dw_offset_t cu_offset, uint32_t *idx_ptr = NULL);
   DWARFUnit *GetCompileUnitContainingDIEOffset(dw_offset_t die_offset);
   DWARFUnit *GetCompileUnit(const DIERef &die_ref);
   DWARFDIE GetDIEForDIEOffset(dw_offset_t die_offset);
   DWARFDIE GetDIE(const DIERef &die_ref);
 
   enum {
     eDumpFlag_Verbose = (1 << 0),  // Verbose dumping
     eDumpFlag_ShowForm = (1 << 1), // Show the DW_form type
     eDumpFlag_ShowAncestors =
         (1 << 2) // Show all parent DIEs when dumping single DIEs
   };
 
   DWARFDebugAranges &GetCompileUnitAranges();
 
 protected:
   static bool OffsetLessThanCompileUnitOffset(dw_offset_t offset,
                                               const DWARFUnitSP &cu_sp);
 
   typedef std::vector<DWARFUnitSP> CompileUnitColl;
+  typedef std::unordered_map<uint64_t, uint32_t> TypeSignatureMap;
 
   //----------------------------------------------------------------------
   // Member variables
   //----------------------------------------------------------------------
   SymbolFileDWARF *m_dwarf2Data;
   CompileUnitColl m_compile_units;
+  TypeSignatureMap m_type_sig_to_cu_index;
   std::unique_ptr<DWARFDebugAranges>
       m_cu_aranges_ap; // A quick address to compile unit table
 
 private:
   // All parsing needs to be done partially any managed by this class as
   // accessors are called.
   void ParseCompileUnitHeadersIfNeeded();
 
   DISALLOW_COPY_AND_ASSIGN(DWARFDebugInfo);
 };
 
 #endif // SymbolFileDWARF_DWARFDebugInfo_h_

source/Plugins/SymbolFile/DWARF/DWARFDebugInfo.cpp

 #include "lldb/Utility/Stream.h"
 
 #include "DWARFCompileUnit.h"
 #include "DWARFDebugAranges.h"
 #include "DWARFDebugAranges.h"
 #include "DWARFDebugInfo.h"
 #include "DWARFDebugInfoEntry.h"
 #include "DWARFFormValue.h"
+#include "DWARFTypeUnit.h"
 #include "LogChannelDWARF.h"
 
 using namespace lldb;
 using namespace lldb_private;
 using namespace std;
 
 //----------------------------------------------------------------------
 // Constructor
if (m_cu_aranges_ap.get() == NULL && m_dwarf2Data) {
     }
 
     const bool minimize = true;
     m_cu_aranges_ap->Sort(minimize);
   }
   return *m_cu_aranges_ap.get();
 }
 
 void DWARFDebugInfo::ParseCompileUnitHeadersIfNeeded() {
-  if (m_compile_units.empty()) {
-    if (m_dwarf2Data != NULL) {
-      lldb::offset_t offset = 0;
-      DWARFUnitSP cu_sp;
-      const auto &debug_info_data = m_dwarf2Data->get_debug_info_data();
-      while ((cu_sp = DWARFCompileUnit::Extract(m_dwarf2Data, debug_info_data,
-                                                &offset))) {
-        m_compile_units.push_back(cu_sp);
-
-        offset = cu_sp->GetNextCompileUnitOffset();
-      }
-    }
+  if (!m_compile_units.empty())
+    return;
+  
+  if (m_dwarf2Data == nullptr)
+    return;
+
+  DWARFCompileUnitSP cu_sp;
+  
+  // If we have a separate .debug_types section, it means we are using DWARF4
+  // or earlier where type units are a separate section. All data in the
+  // .debug_types section doesn't refer directly to any other DIEs, all
+  // references are done via type signatures. With this in mind, we can support
+  // parsing .debug_types by pretending that the data in .debug_types
+  // immediately follows all data in the .debug_info section. This allows all
+  // types in the .debug_types to have a unique DIE offset that is:
+  //    offset = sizeof(.debug_info) + debug_type_offset
+  //
+  // So below we set "offset" to be the size of the .debug_info section. We have
+  // modified the debug_types_data to know that its first byte starts at this
+  // offset.
+  auto debug_types_data = m_dwarf2Data->get_debug_types_data();
+  if (debug_types_data.GetByteSize() == 0)
+    return;
+  offset = debug_info_data.GetByteSize();
+  DWARFTypeUnitSP tu_sp;
+  while ((tu_sp = DWARFTypeUnit::Extract(m_dwarf2Data, debug_types_data,
+                                         &offset))) {
+    m_type_sig_to_cu_index[tu_sp->GetTypeSignature()] = m_compile_units.size();
+    m_compile_units.push_back(tu_sp);
+    offset = tu_sp->GetNextCompileUnitOffset();
   }
 }
 
 size_t DWARFDebugInfo::GetNumCompileUnits() {
   ParseCompileUnitHeadersIfNeeded();
   return m_compile_units.size();
 }
 

 DWARFDIE
 DWARFDebugInfo::GetDIE(const DIERef &die_ref) {
   DWARFUnit *cu = GetCompileUnit(die_ref);
   if (cu)
     return cu->GetDIE(die_ref.die_offset);
   return DWARFDIE(); // Not found
 }
 
+DWARFTypeUnit *DWARFDebugInfo::GetTypeUnitForSignature(uint64_t type_sig) {
+  auto pos = m_type_sig_to_cu_index.find(type_sig);
+  if (pos != m_type_sig_to_cu_index.end())
+    return GetCompileUnitAtIndex(pos->second)->GetAsTypeUnit();
+  return nullptr;
+}

source/Plugins/SymbolFile/DWARF/DWARFFormValue.h

 //===-- DWARFFormValue.h ----------------------------------------*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef SymbolFileDWARF_DWARFFormValue_h_
 #define SymbolFileDWARF_DWARFFormValue_h_
 
+#include "DWARFDIE.h"
 #include "DWARFDataExtractor.h"
 #include <stddef.h> // for NULL
 
 class DWARFUnit;
 
 class DWARFFormValue {
 public:
   typedef struct ValueTypeTag {
public:
   void SetForm(dw_form_t form) { m_form = form; }
   const ValueType &Value() const { return m_value; }
   void Dump(lldb_private::Stream &s) const;
   bool ExtractValue(const lldb_private::DWARFDataExtractor &data,
                     lldb::offset_t *offset_ptr);
   const uint8_t *BlockData() const;
   uint64_t Reference() const;
   uint64_t Reference(dw_offset_t offset) const;
+  DWARFDIE GetTypeSignatureDIE() const;
   bool Boolean() const { return m_value.value.uval != 0; }
   uint64_t Unsigned() const { return m_value.value.uval; }
   void SetUnsigned(uint64_t uval) { m_value.value.uval = uval; }
   int64_t Signed() const { return m_value.value.sval; }
   void SetSigned(int64_t sval) { m_value.value.sval = sval; }
   const char *AsCString() const;
   dw_addr_t Address() const;
   bool IsValid() const { return m_form != 0; }

source/Plugins/SymbolFile/DWARF/DWARFFormValue.cpp

uint64_t DWARFFormValue::Reference() const {
   case DW_FORM_ref4:
   case DW_FORM_ref8:
   case DW_FORM_ref_udata:
     assert(m_cu); // CU must be valid for DW_FORM_ref forms that are compile
                   // unit relative or we will get this wrong
     die_offset += m_cu->GetOffset();
     break;
 
+  case DW_FORM_ref_sig8:
+    // CU must be valid since we will need to back up into the debug info and
+    // find the DIE offset of the type in the type unit.
+    assert(m_cu);
+    die_offset = m_cu->FindTypeSignatureDIEOffset(m_value.value.uval);
+    break;
+
   default:
     break;
   }
 
   return die_offset;
 }
 
 uint64_t DWARFFormValue::Reference(dw_offset_t base_offset) const {
   uint64_t die_offset = m_value.value.uval;
   switch (m_form) {
   case DW_FORM_ref1:
   case DW_FORM_ref2:
   case DW_FORM_ref4:
   case DW_FORM_ref8:
   case DW_FORM_ref_udata:
     die_offset += base_offset;
     break;
 
+  case DW_FORM_ref_sig8:
+    // CU must be valid since we will need to back up into the debug info and
+    // find the DIE offset of the type in the type unit.
+    assert(m_cu);
+    die_offset =
+        m_cu->FindTypeSignatureDIEOffset(m_value.value.uval) + base_offset;
+    break;
+
   default:
     break;
   }
 
   return die_offset;
 }
 
+DWARFDIE DWARFFormValue::GetTypeSignatureDIE() const {
+  if (m_form == DW_FORM_ref_sig8) {
+    // CU must be valid since we will need to back up into the debug info and
+    // find the DIE offset of the type in the type unit.
+    assert(m_cu);
+    return m_cu->FindTypeSignatureDIE(m_value.value.uval);
+  }
+
+  return DWARFDIE();
+}
+
 const uint8_t *DWARFFormValue::BlockData() const { return m_value.data; }
 
 bool DWARFFormValue::IsBlockForm(const dw_form_t form) {
   switch (form) {
   case DW_FORM_exprloc:
   case DW_FORM_block:
   case DW_FORM_block1:
   case DW_FORM_block2:

source/Plugins/SymbolFile/DWARF/DWARFTypeUnit.h

+//===-- DWARFTypeUnit.h --------------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef SymbolFileDWARF_DWARFTypeUnit_h_
+#define SymbolFileDWARF_DWARFTypeUnit_h_
+
+#include "DWARFUnit.h"
+
+class DWARFTypeUnit : public DWARFUnit {
+  friend class DWARFUnit;
+
+public:
+  virtual ~DWARFTypeUnit();
+
+  static DWARFTypeUnitSP Extract(SymbolFileDWARF *dwarf2Data,
+                                 const lldb_private::DWARFDataExtractor &data,
+                                 lldb::offset_t *offset_ptr);
+
+  void Dump(lldb_private::Stream *s) const override;
+
+  //------------------------------------------------------------------
+  /// Get the data that contains the DIE information for this unit.
+  ///
+  /// @return
+  ///   The correct data (.debug_types for DWARF 4 and earlier, and
+  ///   .debug_info for DWARF 5 and later) for the DIE information in
+  ///   this unit.
+  //------------------------------------------------------------------
+  const lldb_private::DWARFDataExtractor &GetData() const override;
+  
+
+  //------------------------------------------------------------------
+  /// Get the size in bytes of the header.
+  ///
+  /// @return
+  ///     Byte size of the type unit header
+  //------------------------------------------------------------------
+  uint32_t GetHeaderByteSize() const override {
+    return m_is_dwarf64 ? 39 : 23;
+  }
+
+  //------------------------------------------------------------------
+  /// Get the type DIE offset if this is a type unit.
+  ///
+  /// If this unit is a type unit, return type DIE offset for the type
+  /// in this unit.
+  ///
+  /// @return
+  ///   The DIE offset that points to the type unit type if this is
+  ///   a type unit, otherwise return DW_INVALID_OFFSET.
+  //------------------------------------------------------------------
+  dw_offset_t GetTypeUnitDIEOffset() {
+    return m_offset + m_type_offset;
+  }
+  
+  //------------------------------------------------------------------
+  /// Get the type signature for the type contained in this unit.
+  ///
+  /// @return
+  ///   The 64 bit type signature for the type contained in this type
+  ///   unit. This value will not be valid if this compile unit is not
+  ///   a type unit.
+  //------------------------------------------------------------------
+  uint64_t GetTypeSignature() const { return m_type_signature; }
+
+  //------------------------------------------------------------------
+  /// Get the type DIE if this is a type unit.
+  ///
+  /// If this unit is a type unit, return type contained within it as
+  /// a DWARFDIE object.
+  ///
+  /// @return
+  ///   The DIE representing the type if this is a type unit, or an
+  ///   invalid DWARFDIE if this is not a type unit.
+  //------------------------------------------------------------------
+  DWARFDIE GetTypeUnitDIE();
+
+protected:
+  // Type signature contained in a type unit which will be valid (non-zero)
+  // for type units only.
+  uint64_t m_type_signature = 0;
+  // Compile unit relative type offset for type units only.
+  dw_offset_t m_type_offset = DW_INVALID_OFFSET;
+
+private:
+  DWARFTypeUnit(SymbolFileDWARF *dwarf2Data);
+  DISALLOW_COPY_AND_ASSIGN(DWARFTypeUnit);
+};
+
+#endif // SymbolFileDWARF_DWARFTypeUnit_h_

source/Plugins/SymbolFile/DWARF/DWARFTypeUnit.cpp

+//===-- DWARFTypeUnit.cpp ---------------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "DWARFTypeUnit.h"
+#include "DWARFDataExtractor.h"
+#include "SymbolFileDWARF.h"
+
+using namespace lldb;
+using namespace lldb_private;
+
+DWARFTypeUnit::DWARFTypeUnit(SymbolFileDWARF *dwarf2Data)
+  : DWARFUnit(dwarf2Data) {}
+
+DWARFTypeUnit::~DWARFTypeUnit() {}
+
+void DWARFTypeUnit::Dump(Stream *s) const {
+  s->Printf("0x%8.8x: Type Unit: length = 0x%8.8x, version = 0x%4.4x, "
+            "abbr_offset = 0x%8.8x, addr_size = 0x%2.2x, "
+            "type_signature = 0x%16.16" PRIx64 ", type_offset = 0x%8.8x "
+            "(next CU at {0x%8.8x})\n",
+            m_offset, m_length, m_version, GetAbbrevOffset(), m_addr_size,
+            m_type_signature, m_type_offset, GetNextCompileUnitOffset());
+}
+
+DWARFTypeUnitSP DWARFTypeUnit::Extract(SymbolFileDWARF *dwarf,
+    const lldb_private::DWARFDataExtractor &data, lldb::offset_t *offset_ptr) {
+  // std::make_shared would require the ctor to be public.
+  std::shared_ptr<DWARFTypeUnit> cu_sp(new DWARFTypeUnit(dwarf));
+  // Out of memory?
+  if (!cu_sp)
+    return nullptr;
+  if (cu_sp->ExtractHeader(dwarf, data, offset_ptr)) {
+    cu_sp->m_type_signature = data.GetU64(offset_ptr);
+    cu_sp->m_type_offset = data.GetDWARFOffset(offset_ptr);
+    return cu_sp;
+  }
+  // reset the offset to where we tried to parse from if anything went wrong
+  *offset_ptr = cu_sp->m_offset;
+  return nullptr;
+}
+
+DWARFDIE DWARFTypeUnit::GetTypeUnitDIE() {
+  // The type offset is compile unit relative, so we need to add the compile
+  // unit offset to ensure we get the correct DIE.
+  return GetDIE(GetTypeUnitDIEOffset());
+}
+
+const lldb_private::DWARFDataExtractor &DWARFTypeUnit::GetData() const {
+  // In DWARF 5, type units are in the .debug_info section. Prior to DWARF 5
+  // type units are in the .debug_types section.
+  if (GetVersion() < 5)
+    return m_dwarf->get_debug_types_data();
+  else
+    return m_dwarf->get_debug_info_data();
+}
+

source/Plugins/SymbolFile/DWARF/DWARFUnit.h

 #define SymbolFileDWARF_DWARFUnit_h_
 
 #include "DWARFDIE.h"
 #include "DWARFDebugInfoEntry.h"
 #include "lldb/lldb-enumerations.h"
 #include "llvm/Support/RWMutex.h"
 
 class DWARFUnit;
+class DWARFTypeUnit;
 class DWARFCompileUnit;
 class NameToDIE;
 class SymbolFileDWARF;
 class SymbolFileDWARFDwo;
 
 typedef std::shared_ptr<DWARFUnit> DWARFUnitSP;
+typedef std::shared_ptr<DWARFTypeUnit> DWARFTypeUnitSP;
+typedef std::shared_ptr<DWARFCompileUnit> DWARFCompileUnitSP;
 
 enum DWARFProducer {
   eProducerInvalid = 0,
   eProducerClang,
   eProducerGCC,
   eProducerLLVMGCC,
   eProcucerOther
 };
 
 class DWARFUnit {
   friend class DWARFCompileUnit;
 
 public:
   virtual ~DWARFUnit();
 
+  bool ExtractHeader(SymbolFileDWARF *dwarf,
+                     const lldb_private::DWARFDataExtractor &data,
+                     lldb::offset_t *offset_ptr);
+
+  DWARFTypeUnit *GetAsTypeUnit();
+  DWARFCompileUnit *GetAsCompileUnit();
   size_t ExtractDIEsIfNeeded(bool cu_die_only);
   DWARFDIE LookupAddress(const dw_addr_t address);
   size_t AppendDIEsWithTag(const dw_tag_t tag,
                            DWARFDIECollection &matching_dies,
                            uint32_t depth = UINT32_MAX) const;
   bool Verify(lldb_private::Stream *s) const;
   virtual void Dump(lldb_private::Stream *s) const = 0;
   //------------------------------------------------------------------
public:
   bool IsDWARF64() const { return m_is_dwarf64; }
 
   bool GetIsOptimized();
 
   SymbolFileDWARFDwo *GetDwoSymbolFile() const;
 
   dw_offset_t GetBaseObjOffset() const;
 
+  DWARFDIE FindTypeSignatureDIE(uint64_t type_sig) const;
+
+  dw_offset_t FindTypeSignatureDIEOffset(uint64_t type_sig) const;  
+
 protected:
   DWARFUnit(SymbolFileDWARF *dwarf);
 
   SymbolFileDWARF *m_dwarf = nullptr;
   std::unique_ptr<SymbolFileDWARFDwo> m_dwo_symbol_file;
   const DWARFAbbreviationDeclarationSet *m_abbrevs = nullptr;
   void *m_user_data = nullptr;
   // The compile unit debug information entry item

source/Plugins/SymbolFile/DWARF/DWARFUnit.cpp

 #include "Plugins/Language/ObjC/ObjCLanguage.h"
 #include "lldb/Core/Module.h"
 #include "lldb/Host/StringConvert.h"
 #include "lldb/Symbol/CompileUnit.h"
 #include "lldb/Symbol/LineTable.h"
 #include "lldb/Symbol/ObjectFile.h"
 #include "lldb/Utility/Timer.h"
 
+#include "DWARFCompileUnit.h"
 #include "DWARFDIECollection.h"
 #include "DWARFDebugAbbrev.h"
 #include "DWARFDebugAranges.h"
 #include "DWARFDebugInfo.h"
+#include "DWARFTypeUnit.h"
 #include "LogChannelDWARF.h"
 #include "SymbolFileDWARFDebugMap.h"
 #include "SymbolFileDWARFDwo.h"
 
 using namespace lldb;
 using namespace lldb_private;
 using namespace std;
 
void DWARFUnit::ClearDIEs(bool keep_compile_unit_die) {
   }
 
   if (m_dwo_symbol_file)
     m_dwo_symbol_file->GetCompileUnit()->ClearDIEs(keep_compile_unit_die);
 }
 
 void DWARFUnit::BuildAddressRangeTable(SymbolFileDWARF *dwarf,
                                        DWARFDebugAranges *debug_aranges) {
+  // No addresses in a type unit
+  if (GetAsTypeUnit())
+    return;
+
   // This function is usually called if there in no .debug_aranges section in
   // order to produce a compile unit level set of address ranges that is
   // accurate.
 
   size_t num_debug_aranges = debug_aranges->GetNumRanges();
 
   // First get the compile unit DIE only and check if it has a DW_AT_ranges
   const DWARFDebugInfoEntry *die = GetUnitDIEPtrOnly();
if (m_func_aranges_ap.get() == NULL) {
     }
 
     const bool minimize = false;
     m_func_aranges_ap->Sort(minimize);
   }
   return *m_func_aranges_ap.get();
 }
 
+DWARFDIE DWARFUnit::FindTypeSignatureDIE(uint64_t type_sig) const {
+  if (auto cu = m_dwarf->DebugInfo()->GetTypeUnitForSignature(type_sig))
+    return cu->GetTypeUnitDIE();
+  return DWARFDIE();
+}
+
+dw_offset_t
+DWARFUnit::FindTypeSignatureDIEOffset(uint64_t type_sig) const {
+  if (auto cu = m_dwarf->DebugInfo()->GetTypeUnitForSignature(type_sig))
+    return cu->GetTypeUnitDIEOffset();
+  return DW_INVALID_OFFSET;
+}
+
+DWARFTypeUnit *DWARFUnit::GetAsTypeUnit() {
+  if (GetUnitDIEOnly().Tag() == DW_TAG_type_unit)
+    return static_cast<DWARFTypeUnit *>(this);
+  return nullptr;
+}
+
+DWARFCompileUnit *DWARFUnit::GetAsCompileUnit() {
+  if (GetUnitDIEOnly().Tag() == DW_TAG_compile_unit)
+    return static_cast<DWARFCompileUnit *>(this);
+  return nullptr;
+}
+
+bool
+DWARFUnit::ExtractHeader(SymbolFileDWARF *dwarf,
+                         const lldb_private::DWARFDataExtractor &data,
+                         lldb::offset_t *offset_ptr) {
+  m_offset = *offset_ptr;
+
+  if (data.ValidOffset(*offset_ptr)) {
+    dw_offset_t abbr_offset;
+    const DWARFDebugAbbrev *abbr = dwarf->DebugAbbrev();
+    m_length = data.GetDWARFInitialLength(offset_ptr);
+    m_is_dwarf64 = data.IsDWARF64();
+    m_version = data.GetU16(offset_ptr);
+    abbr_offset = data.GetDWARFOffset(offset_ptr);
+    m_addr_size = data.GetU8(offset_ptr);
+
+    bool length_OK = data.ValidOffset(GetNextCompileUnitOffset() - 1);
+    bool version_OK = SymbolFileDWARF::SupportedVersion(m_version);
+    bool abbr_offset_OK =
+        dwarf->get_debug_abbrev_data().ValidOffset(abbr_offset);
+    bool addr_size_OK = (m_addr_size == 4) || (m_addr_size == 8);
+
+    if (length_OK && version_OK && addr_size_OK && abbr_offset_OK && abbr) {
+      m_abbrevs = abbr->GetAbbreviationDeclarationSet(abbr_offset);
+      return m_abbrevs != nullptr;
+    }
+
+    // reset the offset to where we tried to parse from if anything went wrong
+    *offset_ptr = m_offset;
+  }
+  return false;
+}
+

source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.cpp

 uint32_t SymbolFileDWARF::CalculateAbilities() {
   uint32_t abilities = 0;
   if (m_obj_file != NULL) {
     const Section *section = NULL;
     const SectionList *section_list = m_obj_file->GetSectionList();
     if (section_list == NULL)
       return 0;
 
-    // On non Apple platforms we might have .debug_types debug info that is
-    // created by using "-fdebug-types-section". LLDB currently will try to
-    // load this debug info, but it causes crashes during debugging when types
-    // are missing since it doesn't know how to parse the info in the
-    // .debug_types type units. This causes all complex debug info types to be
-    // unresolved. Because this causes LLDB to crash and since it really
-    // doesn't provide a solid debuggiung experience, we should disable trying
-    // to debug this kind of DWARF until support gets added or deprecated.
-    if (section_list->FindSectionByName(ConstString(".debug_types"))) {
-      m_obj_file->GetModule()->ReportWarning(
-        "lldb doesn’t support .debug_types debug info");
-      return 0;
-    }
-
     uint64_t debug_abbrev_file_size = 0;
     uint64_t debug_info_file_size = 0;
     uint64_t debug_line_file_size = 0;
 
     section = section_list->FindSectionByName(GetDWARFMachOSegmentName()).get();
 
     if (section)
       section_list = &section->GetChildren();
if (debug_line_file_size > 0)
       abilities |= LineTables;
   }
   return abilities;
 }
 
 const DWARFDataExtractor &
 SymbolFileDWARF::GetCachedSectionData(lldb::SectionType sect_type,
                                       DWARFDataSegment &data_segment) {
-  llvm::call_once(data_segment.m_flag, [this, sect_type, &data_segment] {
+  llvm::call_once(data_segment.m_flag, [&] {
     this->LoadSectionData(sect_type, std::ref(data_segment.m_data));
+    if (sect_type == eSectionTypeDWARFDebugTypes) {
+      // To add .debug_types support in DWARF 4 and earlier with minimally
+      // invasive changes to the current DWARF parsing code, we pretend that
+      // any DIEs in .debug_types start at the end of the .debug_info section.
+      // All info in .debug_types is relative and has no external DIE
+      // references unless thay are DW_AT_signature references, so the DIE
+      // offset for things in the .debug_types. If we do this, then we can
+      // just add the type units to the compile units collection and treat all
+      // information just as we do for all other information in the DWARF and
+      // everything just works. If we were to try to split this out, we would
+      // end up having to change a TON of code. Also DWARF 5 will have compile
+      // and type units in the .debug_info, so coding it this way will prepare
+      // use for an easy transition to DWARF 5.
+      uint64_t debug_info_size = get_debug_info_data().GetByteSize();
+      data_segment.m_data.OffsetData(debug_info_size);
+    }
   });
   return data_segment.m_data;
 }
 
 void SymbolFileDWARF::LoadSectionData(lldb::SectionType sect_type,
                                       DWARFDataExtractor &data) {
   ModuleSP module_sp(m_obj_file->GetModule());
   const SectionList *section_list = module_sp->GetSectionList();
if (comp_unit) {
 
             LanguageType cu_language = DWARFUnit::LanguageTypeFromDWARF(
                 cu_die.GetAttributeValueAsUnsigned(DW_AT_language, 0));
 
             bool is_optimized = dwarf_cu->GetIsOptimized();
             cu_sp.reset(new CompileUnit(
                 module_sp, dwarf_cu, cu_file_spec, dwarf_cu->GetID(),
                 cu_language, is_optimized ? eLazyBoolYes : eLazyBoolNo));
-            if (cu_sp) {
+            if (dwarf_cu->GetAsCompileUnit() && cu_sp) {
               // If we just created a compile unit with an invalid file spec,
               // try and get the first entry in the supports files from the
               // line table as that should be the compile unit.
               if (!cu_file_spec) {
                 cu_file_spec = cu_sp->GetSupportFiles().GetFileSpecAtIndex(1);
                 if (cu_file_spec) {
                   (FileSpec &)(*cu_sp) = cu_file_spec;
                   // Also fix the invalid file spec which was copied from the
                   // compile unit.
                   cu_sp->GetSupportFiles().Replace(0, cu_file_spec);
                 }
               }
+            }
 
-              dwarf_cu->SetUserData(cu_sp.get());
 
-              // Figure out the compile unit index if we weren't given one
-              if (cu_idx == UINT32_MAX)
-                DebugInfo()->GetCompileUnit(dwarf_cu->GetOffset(), &cu_idx);
 
-              m_obj_file->GetModule()->GetSymbolVendor()->SetCompileUnitAtIndex(
-                  cu_idx, cu_sp);
-            }
-          }
-        }
-      }
-    }
-  }
   return cu_sp;
 }
 
 uint32_t SymbolFileDWARF::GetNumCompileUnits() {
   DWARFDebugInfo *info = DebugInfo();
   if (info)
     return info->GetNumCompileUnits();
   return 0;