Add support for type units (.debug_types) to LLDB in a way that is compatible with DWARF 5

[clayborg]

Summary

This patch adds support for type units and parsing the .debug_types section which allows LLDB to debug binaries that were built with "-gdwarf-4 -fdebug-types-section". This patch takes into account how DWARF 5 will represent type units: all compile units will contain extra fields and all type units are in the .debug_info section.

Normally this change would be very invasive because we would need to make a way for DIEs in the .debug_types section to have unique IDs (lldb::user_id_t). This patch takes advantage of the fact that debug info in the .debug_types section can't point to any other DWARF directly. There are only CU relative references and type signatures that might point to other debug info, but indirected through the type signature. This means the offset of the DIEs in the ,.debug_types section are not important. Given all of this, there are the important takes from this patch:

• Modify DWARFCompileUnit to contain extra fields: type signature and type offset. This is the way DWARF5 is doing it, so we will need to do that anyway in the near future.
• Slide the offset of all DIEs in .debug_types by the size of the .debug_info section. This allows every DIE to have unique offset and allows this patch to "just work" with the existing DWARF parser
• Anyone extracting attribute values must use the DWARFDIE::GetData() or DWARFCompileUnit::GetData() to make sure they are getting the right info (.debug_info or .debug_types) to extract from
• LLDB testing adds a new "_dwarf_debug_types" variant (like we add the _dwarf, _dwarf_dsym, _dwo) on platforms that support the .debug_types

I would love to get this in. Anything is better that what LLDB does now when it debugs binaries with .debug_types: crash a lot. So we should get this in and iterate on it.

[labath]

The new tests which are not passing need to be disabled -- we cannot have a red tree while this is being developed.

[clayborg]

Yeah, I was having trouble getting a real read on what was failing due to a non Ubuntu setup at work for me. Is there any way to run a patch on one of the buildbots? If not, can you email me the list of what is failing?

[labath]

In D32167#729508, @clayborg wrote:

Yeah, I was having trouble getting a real read on what was failing due to a non Ubuntu setup at work for me. Is there any way to run a patch on one of the buildbots? If not, can you email me the list of what is failing?

I can send you the list tomorrow, but I am not sure how helpful will it be... When I tried to run it today, it looked like the list was quite large.

What is your development setup nowadays? I am sure you will get some failures on non-ubuntu machines, but I wouldn't expect it to be that many of them. In any case, I think we should work towards eliminating these environment dependencies, particularly if you're going to be doing linux-based development now...

[aprantl]

packages/Python/lldbsuite/test/make/Makefile.rules

197

This shouldn't be necessary on any platform LLDB cares about. DWARF 4 should be the default everywhere.

packages/Python/lldbsuite/test/plugins/builder_base.py

204

Why does the type unit configuration care about whether there are dsyms or not? Shouldn't this be orthogonal?

packages/Python/lldbsuite/test/test_categories.py

27

This is conflating two things (-fdebug-types-section and type units) DWARF5 doesn't have a debug_types section but it still offers type units. Clang doesn't implement this yet, but GCC might, I haven't checked.

source/Plugins/SymbolFile/DWARF/DWARFCompileUnit.cpp

1109

if (auto cu = ...)

1116

if (auto cu = ...)

source/Plugins/SymbolFile/DWARF/DWARFCompileUnit.h

169

/// everywhere?

source/Plugins/SymbolFile/DWARF/DWARFDataExtractor.h

51

I think the //--- line might confuse Doxygen, but I'm not sure.

source/Plugins/SymbolFile/DWARF/DWARFDebugInfo.cpp

121

Is that check common LLDB style?

[jasonmolenda]

Looks fine to me, I don't know the debug_types sections that you're supporting but I read over the code and don't have any problems.

source/Plugins/SymbolFile/DWARF/DWARFDIE.h

92

DWARFDIE::GetData() handles the difference between debug_info / debug_types so the callers of this method don't need to worry about that, correct? Maybe the comment could say

// Get the data that contains the attribute values for this DIE. Support
// for .debug_types means that any DIE can have its data either in the
// .debug_info or the .debug_types section; this method will return the
// correct section data.

or something.

[labath]

This is the list I am getting when using ToT clang targetting x86_64 linux.

Note that there are a couple of non-type-units test broken by this. This needs to be addressed first.

=============
Issue Details
=============
FAIL: test_SBType_template_aspects_dwarf_type_units (lang/cpp/stl/TestSTL.py)
FAIL: test_char_type_dwarf_type_units (types/TestIntegerTypes.py)
FAIL: test_char_type_dwarf_type_units (types/TestIntegerTypesExpr.py)
FAIL: test_double_type_dwarf_type_units (types/TestFloatTypes.py)
FAIL: test_double_type_dwarf_type_units (types/TestFloatTypesExpr.py)
FAIL: test_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libcxx/atomic/TestLibCxxAtomic.py)
FAIL: test_dwarf_type_units (python_api/type/TestTypeList.py)
FAIL: test_dwarf_type_units (python_api/value/empty_class/TestValueAPIEmptyClass.py)
FAIL: test_dwarf_type_units (lang/cpp/wchar_t/TestCxxWCharT.py)
FAIL: test_dwarf_type_units (expression_command/call-function/TestCallStopAndContinue.py)
FAIL: test_dwarf_type_units (expression_command/formatters/TestFormatters.py)
FAIL: test_dwarf_type_units (python_api/sbvalue_persist/TestSBValuePersist.py)
FAIL: test_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libcxx/function/TestLibCxxFunction.py)
FAIL: test_enum_args_dwarf_type_units (lang/cpp/template/TestTemplateArgs.py)
FAIL: test_eval_index_variable_dwarf_type_units (functionalities/non-overlapping-index-variable-i/TestIndexVariable.py)
FAIL: test_float_type_dwarf_type_units (types/TestFloatTypesExpr.py)
FAIL: test_float_type_dwarf_type_units (types/TestFloatTypes.py)
FAIL: test_force_synth_off_dwarf_type_units (python_api/formatters/TestFormattersSBAPI.py)
FAIL: test_formatters_api_dwarf_type_units (python_api/formatters/TestFormattersSBAPI.py)
FAIL: test_get_dynamic_vals_dwarf_type_units (lang/cpp/dynamic-value/TestDynamicValue.py)
FAIL: test_int_type_dwarf_type_units (types/TestIntegerTypesExpr.py)
FAIL: test_int_type_dwarf_type_units (types/TestIntegerTypes.py)
FAIL: test_integer_args_dwarf_type_units (lang/cpp/template/TestTemplateArgs.py)
FAIL: test_long_long_type_dwarf_type_units (types/TestIntegerTypesExpr.py)
FAIL: test_long_long_type_dwarf_type_units (types/TestIntegerTypes.py)
FAIL: test_long_type_dwarf_type_units (types/TestIntegerTypes.py)
FAIL: test_long_type_dwarf_type_units (types/TestIntegerTypesExpr.py)
FAIL: test_operator_new_dwarf_type_units (lang/cpp/global_operators/TestCppGlobalOperators.py)
FAIL: test_rdar10960550_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-python-synth/TestDataFormatterPythonSynth.py)
FAIL: test_recursive_unique_ptr_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libstdcpp/unique_ptr/TestDataFormatterStdUniquePtr.py)
FAIL: test_short_type_dwarf_type_units (types/TestIntegerTypesExpr.py)
FAIL: test_short_type_dwarf_type_units (types/TestIntegerTypes.py)
FAIL: test_unsigned_char_type_dwarf_type_units (types/TestIntegerTypesExpr.py)
FAIL: test_unsigned_char_type_dwarf_type_units (types/TestIntegerTypes.py)
FAIL: test_unsigned_int_type_dwarf_type_units (types/TestIntegerTypesExpr.py)
FAIL: test_unsigned_int_type_dwarf_type_units (types/TestIntegerTypes.py)
FAIL: test_unsigned_long_long_type_dwarf_type_units (types/TestIntegerTypes.py)
FAIL: test_unsigned_long_long_type_dwarf_type_units (types/TestIntegerTypesExpr.py)
FAIL: test_unsigned_long_type_dwarf_type_units (types/TestIntegerTypesExpr.py)
FAIL: test_unsigned_long_type_dwarf_type_units (types/TestIntegerTypes.py)
FAIL: test_unsigned_short_type_dwarf_type_units (types/TestIntegerTypes.py)
FAIL: test_unsigned_short_type_dwarf_type_units (types/TestIntegerTypesExpr.py)
FAIL: test_value_cast_with_regular_inheritance_dwarf_type_units (lang/cpp/dynamic-value/TestCppValueCast.py)
FAIL: test_virtual_madness_dwarf_type_units (lang/cpp/virtual/TestVirtual.py)
FAIL: test_with_dwarf_type_units (expression_command/call-function/TestCallStdStringFunction.py)
FAIL: test_with_expr_parser_dwarf_type_units (lang/cpp/class_types/TestClassTypes.py)
FAIL: test_with_gmodules (functionalities/data-formatter/stringprinter/TestStringPrinter.py)
FAIL: test_with_python_api_dwarf_type_units (lang/cpp/class_static/TestStaticVariables.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libstdcpp/vbool/TestDataFormatterStdVBool.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libcxx/vbool/TestDataFormatterLibcxxVBool.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/format-propagation/TestFormatPropagation.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-synthtype/TestDataFormatterSynthType.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/breakpoint/inlined_breakpoints/TestInlinedBreakpoints.py)
FAIL: test_with_run_command_dwarf_type_units (lang/cpp/class_types/TestClassTypes.py)
FAIL: test_with_run_command_dwarf_type_units (lang/cpp/static_methods/TestCPPStaticMethods.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libcxx/vector/TestDataFormatterLibcxxVector.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-python-synth/TestDataFormatterPythonSynth.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/language_category_updates/TestDataFormatterLanguageCategoryUpdates.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libstdcpp/map/TestDataFormatterStdMap.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-smart-array/TestDataFormatterSmartArray.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libstdcpp/list/TestDataFormatterStdList.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/user-format-vs-summary/TestUserFormatVsSummary.py)
FAIL: test_with_run_command_dwarf_type_units (lang/cpp/diamond/TestDiamond.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libstdcpp/iterator/TestDataFormatterStdIterator.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-cpp/TestDataFormatterCpp.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libcxx/string/TestDataFormatterLibcxxString.py)
FAIL: test_with_run_command_dwarf_type_units (python_api/sbdata/TestSBData.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libstdcpp/vector/TestDataFormatterStdVector.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/refpointer-recursion/TestDataFormatterRefPtrRecursion.py)
FAIL: test_with_run_command_dwarf_type_units (lang/cpp/overloaded-functions/TestOverloadedFunctions.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/synthcapping/TestSyntheticCapping.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-enum-format/TestDataFormatterEnumFormat.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-synthval/TestDataFormatterSynthVal.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-categories/TestDataFormatterCategories.py)
FAIL: test_with_run_command_dwarf_type_units (lang/cpp/auto/TestCPPAuto.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libcxx/multimap/TestDataFormatterLibccMultiMap.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/var-in-aggregate-misuse/TestVarInAggregateMisuse.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-disabling/TestDataFormatterDisabling.py)
FAIL: test_with_run_command_dwarf_type_units (lang/cpp/this/TestCPPThis.py)
FAIL: test_with_run_command_dwarf_type_units (lang/cpp/chained-calls/TestCppChainedCalls.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libcxx/unordered/TestDataFormatterUnordered.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libstdcpp/string/TestDataFormatterStdString.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-globals/TestDataFormatterGlobals.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/summary-string-onfail/Test-rdar-9974002.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libcxx/iterator/TestDataFormatterLibccIterator.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/varscript_formatting/TestDataFormatterVarScriptFormatting.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-synth/TestDataFormatterSynth.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/hexcaps/TestDataFormatterHexCaps.py)
FAIL: test_with_run_command_dwarf_type_units (lang/cpp/class_static/TestStaticVariables.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libstdcpp/smart_ptr/TestDataFormatterStdSmartPtr.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libstdcpp/tuple/TestDataFormatterStdTuple.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/type_completion/TestTypeCompletion.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/recursion/TestValueObjectRecursion.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libcxx/list/TestDataFormatterLibcxxList.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libstdcpp/unique_ptr/TestDataFormatterStdUniquePtr.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-script/TestDataFormatterScript.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-named-summaries/TestDataFormatterNamedSummaries.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libcxx/map/TestDataFormatterLibccMap.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-skip-summary/TestDataFormatterSkipSummary.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libcxx/multiset/TestDataFormatterLibcxxMultiSet.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-advanced/TestDataFormatterAdv.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/frameformat_smallstruct/TestFrameFormatSmallStruct.py)
FAIL: test_with_run_command_dwarf_type_units (functionalities/data-formatter/data-formatter-stl/libcxx/list/loop/TestDataFormatterLibcxxListLoop.py)
ERROR: test_dwarf (functionalities/archives/TestBSDArchives.py)
ERROR: test_dwarf_type_units (functionalities/archives/TestBSDArchives.py)
ERROR: test_dwarf_type_units (python_api/value/linked_list/TestValueAPILinkedList.py)
ERROR: test_dwo (functionalities/archives/TestBSDArchives.py)
ERROR: test_expr_dwarf (lang/c/shared_lib_stripped_symbols/TestSharedLibStrippedSymbols.py)
ERROR: test_expr_dwarf_type_units (lang/c/shared_lib_stripped_symbols/TestSharedLibStrippedSymbols.py)
ERROR: test_expr_dwo (lang/c/shared_lib_stripped_symbols/TestSharedLibStrippedSymbols.py)
ERROR: test_expr_gmodules (lang/c/shared_lib_stripped_symbols/TestSharedLibStrippedSymbols.py)
ERROR: test_gmodules (functionalities/archives/TestBSDArchives.py)
UNEXPECTED SUCCESS: test_dwarf (functionalities/thread/exit_during_break/TestExitDuringBreak.py)
UNEXPECTED SUCCESS: test_dwarf_type_units (functionalities/thread/exit_during_break/TestExitDuringBreak.py)
UNEXPECTED SUCCESS: test_dwo (functionalities/thread/exit_during_break/TestExitDuringBreak.py)
UNEXPECTED SUCCESS: test_gmodules (functionalities/thread/exit_during_break/TestExitDuringBreak.py)
UNEXPECTED SUCCESS: test_lldbmi_gdb_set_target_async_off (tools/lldb-mi/TestMiGdbSetShow.py)
UNEXPECTED SUCCESS: test_lldbmi_process_output (tools/lldb-mi/syntax/TestMiSyntax.py)
UNEXPECTED SUCCESS: test_lldbmi_settings_set_target_run_args_after (tools/lldb-mi/interpreter/TestMiInterpreterExec.py)
UNEXPECTED SUCCESS: test_process_interrupt_dwarf_type_units (functionalities/thread/state/TestThreadStates.py)
UNEXPECTED SUCCESS: test_process_interrupt_dwo (functionalities/thread/state/TestThreadStates.py)
UNEXPECTED SUCCESS: test_process_interrupt_gmodules (functionalities/thread/state/TestThreadStates.py)
UNEXPECTED SUCCESS: test_restart_bug_dwarf (functionalities/signal/raise/TestRaise.py)
UNEXPECTED SUCCESS: test_restart_bug_dwarf_type_units (functionalities/signal/raise/TestRaise.py)
UNEXPECTED SUCCESS: test_sb_api_listener_resume_dwarf (api/multithreaded/TestMultithreaded.py)
UNEXPECTED SUCCESS: test_sb_api_listener_resume_dwarf_type_units (api/multithreaded/TestMultithreaded.py)
UNEXPECTED SUCCESS: test_sb_api_listener_resume_dwo (api/multithreaded/TestMultithreaded.py)
UNEXPECTED SUCCESS: test_sb_api_listener_resume_gmodules (api/multithreaded/TestMultithreaded.py)
UNEXPECTED SUCCESS: test_with_dwarf (lang/cpp/printf/TestPrintf.py)
UNEXPECTED SUCCESS: test_with_dwo (lang/cpp/printf/TestPrintf.py)
UNEXPECTED SUCCESS: test_with_gmodules (lang/cpp/printf/TestPrintf.py)

===================
Test Result Summary
===================
Test Methods:       3220
Reruns:                0
Success:            2031
Expected Failure:    187
Failure:             103
Error:                 9
Exceptional Exit:      0
Unexpected Success:   19
Skip:                871
Timeout:               0
Expected Timeout:      0

[clayborg]

Updated to top of tree sources and verified it passes all tests on linux.

[davide]

This commit has no tests. It should have many. It's very big, so it could be split in several pieces.
I'd really appreciate if you can take the time to do so. For now, some comments.

include/lldb/lldb-enumerations.h

659

this needs a comment.

packages/Python/lldbsuite/test/lldbtest.py

718

spurious whitespaces, please revert.

1782

ditto.

source/Plugins/SymbolFile/DWARF/DIERef.cpp

50

Why? This needs to be explained in a comment.

source/Plugins/SymbolFile/DWARF/DWARFDebugInfo.h

64

any reason why you can't use LLVM adt?

[clayborg]

In D32167#1019399, @davide wrote:

This commit has no tests. It should have many.

I has full testing. Please read the commit and notice there is a new debug info flavor that is added. So of course it is tested.

It's very big, so it could be split in several pieces.

This is one thing that goes together. It isn't that big. I adds support for .debug_types into the ObjectFile and parses the information. How would you propose breaking this up?

I'd really appreciate if you can take the time to do so. For now, some comments.

I believe I did take the time to test this properly. Let me know if you still think otherwise.

packages/Python/lldbsuite/test/lldbtest.py

718

It is removing spurious whitespaces and there are other changes in this file. Most editors these days will remove spurious spaces and we should strive to remove them where possibly no?

1782

Again, removing spurious whitespaces?

packages/Python/lldbsuite/test/make/Makefile.rules

197

Ok, I can remove this.

packages/Python/lldbsuite/test/plugins/builder_base.py

204

All other "def buildXXX" functions do this, so I am following the convention.

packages/Python/lldbsuite/test/test_categories.py

27

So what is the suggestion here?

source/Plugins/SymbolFile/DWARF/DIERef.cpp

50

Will do

source/Plugins/SymbolFile/DWARF/DWARFDataExtractor.h

51

It doesn't. I checked the docs that show up in the doxygen in http://lldb.llvm.org/cpp_reference/html/index.html and these trailing comments don't hose it up.

source/Plugins/SymbolFile/DWARF/DWARFDebugInfo.cpp

121

I am copying the first loop.

source/Plugins/SymbolFile/DWARF/DWARFDebugInfo.h

64

No reason. IMHO STL is fine unless there is a real reason llvm adt is better?

[labath]

So, I tried this out on my machine, and got just one failure:

======================================================================
FAIL: test_with_run_command_dwarf_type_units (TestInlinedBreakpoints.InlinedBreakpointsTestCase)
   Test 'b basic_types.cpp:176' does break (where int.cpp includes basic_type.cpp).
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1827, in dwarf_type_units_test_method
    return attrvalue(self)
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/functionalities/breakpoint/inlined_breakpoints/TestInlinedBreakpoints.py", line 24, in test_with_run_command
    self.inlined_breakpoints()
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/functionalities/breakpoint/inlined_breakpoints/TestInlinedBreakpoints.py", line 46, in inlined_breakpoints
    self, "basic_type.cpp", self.line, num_expected_locations=0)
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 376, in run_break_set_by_file_and_line
    num_locations=num_expected_locations)
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 572, in check_breakpoint_result
    out_num_locations))
AssertionError: False is not True : Expecting 0 locations, got 1.

However, I am not so sure about the proliferation of debug info variants that we seem to be having. Right now we have two outstanding patches wanting to add a debug variant, which would bring the total amount of variants per test to 6. I'm not sure this is a tractable way forward.
IIUC, clang only puts "non-trivial" types in type units. I'm not sure how many of our tests even define classes/structs (i.e., for how many tests this debug info variant would actually be testing something new).

source/Plugins/SymbolFile/DWARF/DWARFCompileUnit.h

194

It seems weird to have an IsTypeUnit method on a compile unit, when a dwarf compile unit is clearly not a type unit. Would some of the refactors that Jan is doing (I haven't looked at those too closely as it's not really my area, but I know he is adding a DwarfUnit class) help with this?

[emaste]

packages/Python/lldbsuite/test/make/Makefile.rules

197

FreeBSD still defaults to DWARF2 at present.

[clayborg]

In D32167#1019467, @labath wrote:

So, I tried this out on my machine, and got just one failure:

======================================================================
FAIL: test_with_run_command_dwarf_type_units (TestInlinedBreakpoints.InlinedBreakpointsTestCase)
   Test 'b basic_types.cpp:176' does break (where int.cpp includes basic_type.cpp).
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1827, in dwarf_type_units_test_method
    return attrvalue(self)
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/functionalities/breakpoint/inlined_breakpoints/TestInlinedBreakpoints.py", line 24, in test_with_run_command
    self.inlined_breakpoints()
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/functionalities/breakpoint/inlined_breakpoints/TestInlinedBreakpoints.py", line 46, in inlined_breakpoints
    self, "basic_type.cpp", self.line, num_expected_locations=0)
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 376, in run_break_set_by_file_and_line
    num_locations=num_expected_locations)
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 572, in check_breakpoint_result
    out_num_locations))
AssertionError: False is not True : Expecting 0 locations, got 1.

I saw this too. When I ran _any_ of the variants manually, they all failed to set any breakpoints. Not sure this test is testing what it claims to be testing.

However, I am not so sure about the proliferation of debug info variants that we seem to be having. Right now we have two outstanding patches wanting to add a debug variant, which would bring the total amount of variants per test to 6. I'm not sure this is a tractable way forward.
IIUC, clang only puts "non-trivial" types in type units. I'm not sure how many of our tests even define classes/structs (i.e., for how many tests this debug info variant would actually be testing something new).

We could check after the build if the binary for the test even has a .debug_types sections and if not abort? That was we don't have to mark up the test in any way (like "this tests has structures or enums"), but we could ensure we test with .debug_types when needed.

[clayborg]

In D32167#1019504, @clayborg wrote:

In D32167#1019467, @labath wrote:

So, I tried this out on my machine, and got just one failure:

======================================================================
FAIL: test_with_run_command_dwarf_type_units (TestInlinedBreakpoints.InlinedBreakpointsTestCase)
   Test 'b basic_types.cpp:176' does break (where int.cpp includes basic_type.cpp).
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1827, in dwarf_type_units_test_method
    return attrvalue(self)
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/functionalities/breakpoint/inlined_breakpoints/TestInlinedBreakpoints.py", line 24, in test_with_run_command
    self.inlined_breakpoints()
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/functionalities/breakpoint/inlined_breakpoints/TestInlinedBreakpoints.py", line 46, in inlined_breakpoints
    self, "basic_type.cpp", self.line, num_expected_locations=0)
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 376, in run_break_set_by_file_and_line
    num_locations=num_expected_locations)
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 572, in check_breakpoint_result
    out_num_locations))
AssertionError: False is not True : Expecting 0 locations, got 1.

I saw this too. When I ran _any_ of the variants manually, they all failed to set any breakpoints. Not sure this test is testing what it claims to be testing.

Actually I can repro this and will look into it

[labath]

In D32167#1019504, @clayborg wrote:

In D32167#1019467, @labath wrote:

However, I am not so sure about the proliferation of debug info variants that we seem to be having. Right now we have two outstanding patches wanting to add a debug variant, which would bring the total amount of variants per test to 6. I'm not sure this is a tractable way forward.
IIUC, clang only puts "non-trivial" types in type units. I'm not sure how many of our tests even define classes/structs (i.e., for how many tests this debug info variant would actually be testing something new).

We could check after the build if the binary for the test even has a .debug_types sections and if not abort? That was we don't have to mark up the test in any way (like "this tests has structures or enums"), but we could ensure we test with .debug_types when needed.

That would help somewhat, but that's still a very "shotgun" approach to testing this. What I would try to do (and I think this is what Davide had in mind as well) is to identify the cases that are interesting from the type unit perspective (a single class in type unit, two classes in type units referencing each other, class referencing an enum, etc. (I'm not really sure about what all that goes into a type unit)) and then write specific tests for that (or reuse existing ones). For example, plenty of tests do breakpoint setting or unwinding and those probably don't have to be tested separately with type units (or maybe one test that checks that setting breakpoint on a method in type unit is sufficient).

[jingham]

Right now, which tests are "run all variants" is represents more history than policy. Opt out of all variants is opt in, and while I at least kick out obvious tests that don't really need to run all variants when I touch them, but I don't think we have done a careful audit. I think the general idea of running variants is good, but we really should reduce the number of tests we run this way to the ones that actually test things that might break that way.

Note, I have (though very very occasionally) seen a dsymutil bug that broke line tables. So it would be good to keep a few "set a breakpoint and run to it" tests just to sanity check this. But most tests that just run somewhere and print a backtrace or view an integer variable or such-like do not need to run all variants.

[davide]

In D32167#1019621, @labath wrote:

In D32167#1019504, @clayborg wrote:

In D32167#1019467, @labath wrote:

However, I am not so sure about the proliferation of debug info variants that we seem to be having. Right now we have two outstanding patches wanting to add a debug variant, which would bring the total amount of variants per test to 6. I'm not sure this is a tractable way forward.
IIUC, clang only puts "non-trivial" types in type units. I'm not sure how many of our tests even define classes/structs (i.e., for how many tests this debug info variant would actually be testing something new).

We could check after the build if the binary for the test even has a .debug_types sections and if not abort? That was we don't have to mark up the test in any way (like "this tests has structures or enums"), but we could ensure we test with .debug_types when needed.

That would help somewhat, but that's still a very "shotgun" approach to testing this. What I would try to do (and I think this is what Davide had in mind as well) is to identify the cases that are interesting from the type unit perspective (a single class in type unit, two classes in type units referencing each other, class referencing an enum, etc. (I'm not really sure about what all that goes into a type unit)) and then write specific tests for that (or reuse existing ones). For example, plenty of tests do breakpoint setting or unwinding and those probably don't have to be tested separately with type units (or maybe one test that checks that setting breakpoint on a method in type unit is sufficient).

Precisely. I'm afraid what's proposed here is too coarse to be useful.

[labath]

In D32167#1019635, @jingham wrote:

Note, I have (though very very occasionally) seen a dsymutil bug that broke line tables. So it would be good to keep a few "set a breakpoint and run to it" tests just to sanity check this. But most tests that just run somewhere and print a backtrace or view an integer variable or such-like do not need to run all variants.

The thing here is that I'm not sure everything can be captured by a single "debug info variant" dimension. The current dimensions kinda make sense, as in they alter the way pretty much all of debug info is accessed (we can think of "dwarf" as the basic one, "dsym" and "dwo" are similar in that they place everything in a separate file, I'm not sure about gmodules, but that's because I have no idea on how it works). However, I'm not sure the same goes for type units. For one they are not completely mutually exclusive with the existing variants, so you can have type-units+regular dwarf and type units+dwo (maybe even type units+dsym?). But obviously we can't add another "type unit" dimension and do a cartesian product....

[clayborg]

I am afraid of the opposite: we test what we think we need to test and our simple tests cases don't adequately test the feature we are adding. I can certainly limit the testing to very simple test cases with a one test for a class and one for a enum, but that wouldn't have caught the issues I ran into with static variables in classes that I fixed before submitting this. My point is it is hard to figure out what a compiler, or debugger might hose up without testing all possible cases. Please chime in with opinions as I will go with the flow on this.

[clayborg]

In D32167#1019701, @labath wrote:

In D32167#1019635, @jingham wrote:

Note, I have (though very very occasionally) seen a dsymutil bug that broke line tables. So it would be good to keep a few "set a breakpoint and run to it" tests just to sanity check this. But most tests that just run somewhere and print a backtrace or view an integer variable or such-like do not need to run all variants.

The thing here is that I'm not sure everything can be captured by a single "debug info variant" dimension. The current dimensions kinda make sense, as in they alter the way pretty much all of debug info is accessed (we can think of "dwarf" as the basic one, "dsym" and "dwo" are similar in that they place everything in a separate file, I'm not sure about gmodules, but that's because I have no idea on how it works). However, I'm not sure the same goes for type units. For one they are not completely mutually exclusive with the existing variants, so you can have type-units+regular dwarf and type units+dwo (maybe even type units+dsym?). But obviously we can't add another "type unit" dimension and do a cartesian product....

if you see how horrible type units are in their implementation, you might change your mind. If one has a positive view of type units, you might think "I get one type and one type only and that can only mean good things". I reality this feature's implementation is really messy and there are so many areas that can screw up. For instance if you have a class that has static members, you will get a nice full type in the type unit that defines everything in the class. But with type units you can't refer to anything within the type unit itself other than the top level type itself (the struct/union/class or enum). So you now need to declare a DW_TAG_variable that points to the static member of the class so you can give it an address... What happens? The type is duplicated in each compile unit that uses it and a skeleton incomplete version of the class is represented in the compile unit DWARF (not in a type unit). The top level DW_TAG_structure or DW_TAG_class has a DW_AT_signature that points to the type in the type unit, but the incomplete struct/class has _just_ the static members inside of it (again, in the compile unit, not in the type unit). Then you have a DW_TAG_variable that points to the DW_TAG_member within the incomplete struct or class because there is no way to point to anything within the type if the type is in a type unit. So I believe that we do need to fully test types and run the .debug_types as a separate variant. Each compiler might also do things a bit differently as there is nothing in the .debug_types specification that says that this is how things need to be. And that is just one thing I found while debugging .debug_types... I am sure there are more skeletons in the closet. If we want to support this feature we need to test it. I come to this conclusion after years of dealing with many different "this DWARF does this cool new way to do things" where it has bit us in the past and things break.

[vsk]

I'm concerned about the time it takes to run the test suite. If we can get decent test coverage of this patch without adding a new debug info flavor, that'd be ideal.

packages/Python/lldbsuite/test/lldbtest.py

718

No, we generally avoid making changes that are unrelated to the patch, as they make patch review and subsequent merges harder.

source/Plugins/SymbolFile/DWARF/DWARFDataExtractor.h

58

Please clang-format your diffs.

[clayborg]

clang format
fixed comments
removed white space
fixed functionalities/breakpoint/inlined_breakpoints/TestInlinedBreakpoints.py failures

[aprantl]

packages/Python/lldbsuite/test/test_categories.py

27

I would mention type units in the comment rather than -fdebug-types-section.

source/Plugins/SymbolFile/DWARF/DWARFDebugInfo.cpp

97

We probably should convert this to an early exit.

106

Perhaps add a comment explaining what is being done here?

121

I find hard to follow. What do you think about:

while (debug_types_data.ValidOffset(offset)) {
     cu_sp = DWARFCompileUnit::Extract(m_dwarf2Data, debug_types_data,
                                            &offset, true)));
     if (!cu_sp)
        return;
     m_type_sig_to_cu_index[cu_sp->GetTypeSignature()] =
         m_compile_units.size();
     m_compile_units.push_back(cu_sp);
     offset = cu_sp->GetNextCompileUnitOffset();
   }

?

Also, shouldn't the error be propagated?

[clayborg]

Address Adrian's comments.

[labath]

In D32167#1019702, @clayborg wrote:

I am afraid of the opposite: we test what we think we need to test and our simple tests cases don't adequately test the feature we are adding. I can certainly limit the testing to very simple test cases with a one test for a class and one for a enum, but that wouldn't have caught the issues I ran into with static variables in classes that I fixed before submitting this. My point is it is hard to figure out what a compiler, or debugger might hose up without testing all possible cases. Please chime in with opinions as I will go with the flow on this.

I don't think anyone here is suggesting to test less. The question we are asking is if running all 1000 or so tests doesn't just giv a false sense of security (and makes the testing process longer). A specially crafted test can trigger more corner cases and make it easier to debug future failures that a generic test. If the case of a static variable in a class is interesting, then maybe a test where you have two static variables from the same class defined in two different compilation units (and maybe referenced from a third one) is interesting as well. I'm pretty sure we don't have a test like that right now. Another interesting case which would not be tested in the "separate flavour" mode is the mixed-flags case: have part of your module compiled with type units, part without (and maybe a third part with type units and fission)..

Running the entire dotest test suite in -fdebug-types-section is certainly a good way to catch problems, but it's not the way to write regression tests. FWIW, the way I plan to test the new accelerator tables when I get to the lldb part is to run dotest with the new flags locally during development, use the failures to identify interesting test vectors, and then write regular tests to trigger these.

[probinson]

In D32167#1020032, @labath wrote:

Running the entire dotest test suite in -fdebug-types-section is certainly a good way to catch problems, but it's not the way to write regression tests.

Is there testing in place that is more serious/thorough than the normal regression testing? It might be reasonable to do the full cartesian set at a slower pace. For example LLVM has the concept of "expensive checks" and there are a small number of bots out there that enable them. But it's not something anybody runs normally.

[clayborg]

In D32167#1020032, @labath wrote:

In D32167#1019702, @clayborg wrote:

I am afraid of the opposite: we test what we think we need to test and our simple tests cases don't adequately test the feature we are adding. I can certainly limit the testing to very simple test cases with a one test for a class and one for a enum, but that wouldn't have caught the issues I ran into with static variables in classes that I fixed before submitting this. My point is it is hard to figure out what a compiler, or debugger might hose up without testing all possible cases. Please chime in with opinions as I will go with the flow on this.

I don't think anyone here is suggesting to test less. The question we are asking is if running all 1000 or so tests doesn't just giv a false sense of security (and makes the testing process longer). A specially crafted test can trigger more corner cases and make it easier to debug future failures that a generic test. If the case of a static variable in a class is interesting, then maybe a test where you have two static variables from the same class defined in two different compilation units (and maybe referenced from a third one) is interesting as well. I'm pretty sure we don't have a test like that right now. Another interesting case which would not be tested in the "separate flavour" mode is the mixed-flags case: have part of your module compiled with type units, part without (and maybe a third part with type units and fission)..

Running the entire dotest test suite in -fdebug-types-section is certainly a good way to catch problems, but it's not the way to write regression tests. FWIW, the way I plan to test the new accelerator tables when I get to the lldb part is to run dotest with the new flags locally during development, use the failures to identify interesting test vectors, and then write regular tests to trigger these.

For the accelerator tables you could always just manually index the DWARF and compare the results to the new accelerator tables as a unit test. No further testing needed?

[clayborg]

Would everyone be ok with a solution where we leave the .debug_types tests in as a flavor, but we check the binary after creating it to ensure it has a .debug_types section. If it doesn't, then we skip the test (or pass it?), and if it does, then we go ahead and test it? The only overhead that is added then is some building. Many of the tests don't have types that go into the .debug_types section, so this would be an easy way for us to test .debug_types when it is present and just build and skip if it doesn't. No maintenance needed on any tests, we will just detect and enable if needed?

[clayborg]

Ping for my last comment

[labath]

Sorry, I've been waiting to give others (Davide?) a chance to express their opinion. I personally don't think having a new debug info flavour is a good idea. Tests written specifically to test this functionality will be easier to maintain and debug when they break. And keeping adding new flavours for every compiler option is not a viable long term strategy.

[davide]

As already pointed out, I think this feature should be thought again & have more focused testing. We can have a meeting/discussion about this, and I need to think about it more.
But what you're proposing is basically a sledgehammer and I don't feel confident about having the patch committed in its current state.

[jasonmolenda]

I think we're being a little hasty here. Greg's last suggestion is worth investigation -- how many tests would actually be run for this new variant?

Greg, maybe you could remove the make clean targets from packages/Python/lldbsuite/test/plugins/builder_base.py (I think that'll do it), do a testsuite run to create all the test binaries, and then find out how how many tests are actually going to be run here?

Pavel is right that the explosion of debug variations will quickly become untenable, even with bots, and I'm not sure what a good solution to that is. Running the debug information parsers against the wide variety of programs we include in the testsuite is a good way to test them. This is a problem we should discuss, but if the practical impact of adding the extra testing for .debug_types is that we run 50 tests through an additional variant, I think it is premature to draw the line here and say this must be solved now.

[jankratochvil]

In D32167#1024546, @labath wrote:

I personally don't think having a new debug info flavour is a good idea. Tests written specifically to test this functionality will be easier to maintain and debug when they break. And keeping adding new flavours for every compiler option is not a viable long term strategy.

Here I disagree, from GDB testsuite experience I find more wider testing always better as it will better catch bugs even in unrelated new functionalities being implemented in the future. One can never guess all the combinations in advance - if one could then one needs no regression testing at all. There are more than enough CPU cycles available for testing. I find rather a serious problem racy==unreliable test results which waste human efforts investigating the results and with more parallel testing (make -j) and virtualized infrastructure the racy results happen even more often. As an example one racy testcase I idenitifed is:

packages/Python/lldbsuite/test/functionalities/signal/handle-segv/TestHandleSegv.py
line 33: AssertionError: 4 != 5 (eStateLaunching != eStateStopped)

I think at least these test are racy:

BreakpointAfterJoinTestCase-test
CreateDuringStepTestCase-test_step_inst_with
HandleSegvTestCase-test_inferior_handle_sigsegv
ReturnValueTestCase-test_vector_values
TestTargetSymbolsBuildidSymlink-test_target_symbols_buildid
ThreadSpecificBreakPlusConditionTestCase-test_python
ThreadSpecificBreakTestCase-test_python
ThreadStateTestCase-test_process_interrupt

[davide]

In D32167#1026762, @jankratochvil wrote:

In D32167#1024546, @labath wrote:

I personally don't think having a new debug info flavour is a good idea. Tests written specifically to test this functionality will be easier to maintain and debug when they break. And keeping adding new flavours for every compiler option is not a viable long term strategy.

Here I disagree, from GDB testsuite experience I find more wider testing always better as it will better catch bugs even in unrelated new functionalities being implemented in the future. One can never guess all the combinations in advance - if one could then one needs no regression testing at all. There are more than enough CPU cycles available for testing. I find rather a serious problem racy==unreliable test results which waste human efforts investigating the results and with more parallel testing (make -j) and virtualized infrastructure the racy results happen even more often. As an example one racy testcase I idenitifed is:

packages/Python/lldbsuite/test/functionalities/signal/handle-segv/TestHandleSegv.py
line 33: AssertionError: 4 != 5 (eStateLaunching != eStateStopped)

I think at least these test are racy:

BreakpointAfterJoinTestCase-test
CreateDuringStepTestCase-test_step_inst_with
HandleSegvTestCase-test_inferior_handle_sigsegv
ReturnValueTestCase-test_vector_values
TestTargetSymbolsBuildidSymlink-test_target_symbols_buildid
ThreadSpecificBreakPlusConditionTestCase-test_python
ThreadSpecificBreakTestCase-test_python
ThreadStateTestCase-test_process_interrupt

I'm not necessarily convinced having another column in the matrix is something that can substitute targeted testing.
If you're talking about something complementary, I think we might consider it, as a follow up.
Also, thanks for bringing the problem of the racy tests up. Adding another row in the matrix increases the chances of test failures. I think we might consider addressing the unreliability before making such a big change. Do you have a way of reproducing? [or even better, investigating]

Our bots have been green for a while (we tried to investigate known issues http://lab.llvm.org:8080/green/view/LLDB/) but we can only address what we know :)

[jankratochvil]

In D32167#1026779, @davide wrote:

Do you have a way of reproducing?

It just happens for me each time - on Fedora 27 x86_64 on 16-core (32HT) 2-node NUMA machine having env var MAKEFLAGS=-j32 (and tuned-adm throughput-performance if it matters). Comparing multiple runs makes some tests {Expected,Unexpected,}{Success,Failure,Error} without changing anything.

[or even better, investigating]

When the testcase is then ran separately it sure succeeds as the machine then has no load that time. Maybe one could make some race reproducer helper (such as was GDB read1) but I should analyse more LLDB testsuite races to know the common racy issues in LLDB testsuite.

[davide]

In D32167#1026780, @jankratochvil wrote:

In D32167#1026779, @davide wrote:

Do you have a way of reproducing?

It just happens for me each time - on Fedora 27 x86_64 on 16-core (32HT) 2-node NUMA machine having env var MAKEFLAGS=-j32 (and tuned-adm throughput-performance if it matters). Comparing multiple runs makes some tests {Expected,Unexpected,}{Success,Failure,Error} without changing anything.

[or even better, investigating]

When the testcase is then ran separately it sure succeeds as the machine then has no load that time. Maybe one could make some race reproducer helper (such as was GDB read1) but I should analyse more LLDB testsuite races to know the common racy issues in LLDB testsuite.

That sounds a great helper. Any chance you're interested in working on that? I don't have access to a machine beefy enough to repro the failures, I'm afraid.

[jankratochvil]

In D32167#1026781, @davide wrote:

Any chance you're interested in working on that?

Yes, it is in the shortterm or rather midterm plan.

[clayborg]

In D32167#1026779, @davide wrote:

In D32167#1026762, @jankratochvil wrote:

In D32167#1024546, @labath wrote:

I personally don't think having a new debug info flavour is a good idea. Tests written specifically to test this functionality will be easier to maintain and debug when they break. And keeping adding new flavours for every compiler option is not a viable long term strategy.

Here I disagree, from GDB testsuite experience I find more wider testing always better as it will better catch bugs even in unrelated new functionalities being implemented in the future. One can never guess all the combinations in advance - if one could then one needs no regression testing at all. There are more than enough CPU cycles available for testing. I find rather a serious problem racy==unreliable test results which waste human efforts investigating the results and with more parallel testing (make -j) and virtualized infrastructure the racy results happen even more often. As an example one racy testcase I idenitifed is:

packages/Python/lldbsuite/test/functionalities/signal/handle-segv/TestHandleSegv.py
line 33: AssertionError: 4 != 5 (eStateLaunching != eStateStopped)

I think at least these test are racy:

BreakpointAfterJoinTestCase-test
CreateDuringStepTestCase-test_step_inst_with
HandleSegvTestCase-test_inferior_handle_sigsegv
ReturnValueTestCase-test_vector_values
TestTargetSymbolsBuildidSymlink-test_target_symbols_buildid
ThreadSpecificBreakPlusConditionTestCase-test_python
ThreadSpecificBreakTestCase-test_python
ThreadStateTestCase-test_process_interrupt

I'm not necessarily convinced having another column in the matrix is something that can substitute targeted testing.

If I would have done targeted testing I wouldn't have caught the breakpoint issue that came up with this patch. So I agree with jankratochvil that it is best to test this all the way by adding another row. There is no way we can guess what certain compilers will do and and all the different permutations this can cause. Not sure where ".debug_types" lands in terms of popularity, but I would guess it comes in above other DWARF flavors (dwo, dwz). Can I at least land this and then we can work on future improvements to the rows required. This is only currently enabled on linux where we tend to have fast buildbots. Darwin compilers don't support this with mach-o due to the 255 section limit that mach-o suffers from.

[jankratochvil]

source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.cpp

595

I do not like this DWARFDataExtractor::m_start modification, it sort of corrupts the DataExtractor and various operations stop working then - such as DWARFDataExtractor::GetByteSize(). DWZ patch makes from current dw_offset_t a virtual (remapped) offset and introduces new physical file section offset which is looked up for data extraction. The file offset is represented as DWARFFileOffset in D40474, instead of bool m_is_dwz; there could be some enum { DEBUG_INFO, DEBUG_TYPES, DWZ_DEBUG_INFO } m_where; instead.

[clayborg]

Is there a link to some documentation that explains what DWZ does? I found a bunch of blurbs and man pages, but nothing useful. Nothing is found when I look for "DWZ" in either the DWARF 4 or DWARF 5 spec.

source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.cpp

595

This means that this diff doesn't affect all of the other DWARF code. Nothing in .debug_types will refer to anything else (not DW_FORM_ref_addr, or any external references). So this trick allows us to just treat .debug_info as if .debug_types was appended to the end since nothing in .debug_types refers to any DIE outside of its type unit. This also mirrors what will actually happen with DWARF5 when all of the data is contained inside of the .debug_info section. This allows each DIE to have a unique "ID". Any other change requires a lot of changes to the DWARF parser and logic. So I actually like this feature. We can fix the GetByteSize() if needed. Basically every object in DWARf right now must be able to make a unique 64 bit unsigned integer ID in some way that we can get back to that info and partially parse more. These are handed out as lldb::user_id_t values for types, functions and more. Each flavor of DWARF will encode what they want into here. The normal DWARF it is just the absolute offset within the .debug_info. With .debug_types we just add the size of the .debug_info to the ID. For DWARF in .o files on Darwin, we encode the compile unit index into the top 32 bits and the DIE offset into the lower, DWO does something just as DWZ will need to. DWARFFileOffset doesn't mean much if there are multiple files. We have many competing type uniquing/debug info size reduction strategies being employed here. I can't believe we have DWO, DWZ, and debug types... But we have to make them all work. We can't just use the absolute file offset because DWO used external files where the file offsets could be the same in the external .o files... Not sure how this works with DWZ or what the best option is. I will read up on DWZ so I can propose some viable options. But each new flavor of the day that gets added the DWARF parser is adding a bunch of logic and edge cases. If two technologies (DWZ + DWO, DWZ + debug_types, etc) are used together, we need to ensure they can.

[clayborg]

Found the DWZ stuff:
http://www.dwarfstd.org/ShowIssue.php?issue=120604.1&type=open

[clayborg]

For DWZ files, it seems types can be contained in external DWARF files and are referred to using new forms. The right way to fix this in LLDB it to load each of these external files as a separate DWZ file (no changes needed to anything) and then teach the SymbolFileDWARF that refers to these files to import the types. If we have 24 shared libraries that use a single DWZ, we shouldn't be loading it more than once and inlining the DWARF, we should be using the AST importer to import the type from that file into the AST for the current DWARF file. We do have done this with -gmodules already, so DWZ shouldn't be that different. Let me know if I am missing anything.

[jankratochvil]

In D32167#1062618, @clayborg wrote:

we should be using the AST importer to import the type from that file into the AST for the current DWARF file. We do have done this with -gmodules already, so DWZ shouldn't be that different.

Is AST usable for imported declarations of DW_TAG_variable, DW_TAG_subprogram, DW_TAG_enumeration_type+DW_TAG_enumerators etc.? Still imported DW_TAG_partial_unit cannot contain any code or data addresses which may make it easier for AST.

source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.cpp

595

Any other change requires a lot of changes to the DWARF parser and logic. So I actually like this feature.

I agree it is a fine quick&dirty hack. Just if my DWZ support gets accepted later anyway then this .debug_types feature could be implemented by its framework in a clean way (as a regular DIEs remapping which is required for DWZ anyway).

If two technologies (DWZ + DWO, DWZ + debug_types, etc) are used together, we need to ensure they can.

DWZ + DWO do not make sense to me. I haven't tried to use DWZ for DWO but DWZ finds common DWARF subtrees typically across CUs so it would not find much.

DWZ + debug_types is explicitly supported by the DWZ tool although that is IMO for compatibility only, DWZ can make the common type references slightly smaller than debug_types. I will sure need to implement debug_types support into my DWZ-for-LLDB patchset later.

[jankratochvil]

In D32167#1062614, @clayborg wrote:

Found the DWZ stuff:
http://www.dwarfstd.org/ShowIssue.php?issue=120604.1&type=open

BTW the current DWZ tool supports only DWARF-4 and the DWZ tags are non-standard GNU extensions. Therefore DW_FORM_GNU_ref_alt in existing DWZ files is DW_FORM_ref_alt in the proposal you found. I am not aware of any real documentation for this DWARF-4 GNU extension. But DWARF-5 further modified the proposal so it became DW_FORM_ref_sup4+DW_FORM_ref_sup8 in the DWARF-5 standard.

My DWZ-for-LLDB patchset implements the DWARF-4 non-standard GNU extensions as it is used in RHEL-7 (and possibly RHEL-8, not sure yet) which will be supported at least till 2024. It is also present in all recent Fedoras. Currently the DWZ tool has not yet been ported to DWARF-5, Mark Wielaard from Red Hat plans to do so.

[jankratochvil]

source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.cpp

595

For DWARF in .o files on Darwin, we encode the compile unit index into the top 32 bits and the DIE offset into the lower

BTW that prevents implementing DWARF64 so I wanted to prevent using the 32/32 split. Currently Fedora/RHEL already contains files with .debug_info size near the 32-bit limit:

http://ftp.muni.cz/pub/linux/fedora/linux/development/rawhide/Everything/x86_64/debug/tree/Packages/q/qt5-qtwebkit-debuginfo-5.212.0-0.20.alpha2.fc29.x86_64.rpm
/usr/lib/debug/usr/lib64/libQt5WebKit.so.5.212.0-5.212.0-0.20.alpha2.fc29.x86_64.debug
.debug_info size = 0x9bff1b08 = 2.4GiB

Although that is because neither DWZ (as it would run out of memory) nor .debug_types (as it is expected DWZ will handle the duplicities) are applied and so all CUs are separate and even if it was >4GB there would still be no need for 64-bit DW_FORM_ref_addr. Still we are already close to the 32-bit limit, also shifting the split for example to 40/24 would make some limit on number of CUs etc.

[clayborg]

Update patch to work with changes from:

https://reviews.llvm.org/D45170

[probinson]

I am looking at making the LLVM parser handle type units across v4/v5, and the trick of pretending .debug_types is pasted onto the end of .debug_info seems like a pretty convenient fiction. It unifies DIE handling, and still allows the dumper to distinguish which bits are in which section, which is traditionally how the dumper UI thinks about DWARF. So as long as you and Jan are on the same page there, I'm happy to follow suit.

source/Plugins/SymbolFile/DWARF/DIERef.cpp

53

... the type is

[clayborg]

Ping. I would like to get this in. As for testing, the full matrix should be run on .debug_types as it is the main form of type uniquing that is defined by the DWARF spec. Any formats that are fully accepted and written into the DWARF spec should be fully tested to ensure everything works. .debug_types is only enabled on linux right now and our build bots are fast on these systems.

[labath]

Maybe we could start by splitting of the ObjectFile recognition code for the debug_types section into a separate patch. That should be sufficiently non-controversial and it would reduce the number of files touched by this patch a bit.

[clayborg]

In D32167#1087627, @labath wrote:

Maybe we could start by splitting of the ObjectFile recognition code for the debug_types section into a separate patch. That should be sufficiently non-controversial and it would reduce the number of files touched by this patch a bit.

Sure thing. Anything to get this moving.

[clayborg]

In D32167#1087627, @labath wrote:

Maybe we could start by splitting of the ObjectFile recognition code for the debug_types section into a separate patch. That should be sufficiently non-controversial and it would reduce the number of files touched by this patch a bit.

I broke out the object file parsing code:

https://reviews.llvm.org/D46529

[labath]

source/Plugins/SymbolFile/DWARF/DWARFDataExtractor.h

35–58

This looks like a thing we could focus on next. Sliding the contents of the data buffer seems like a useful fiction, but the way it is implemented now, it relies on undefined behavior (out-of-range pointers, possibly even wrapping the address space) and it violates the invariants that we have some far been assuming about data extractors (namely, that for a DataExtractor, all offsets from 0 to GetByteSize()-1 are dereferencable). For example, for a "slid" data extractor:

• ValidOffset(0) will return true, even though that is clearly not the case
• Append(data2) will try to access data at *m_start and probably crash
• so will Checksum()

Since there seems to be a consensus (at least, nobody has proposed an alternative solution) that this is the way we should treat type units, I think it makes sense to spend some time to make sure we built it on solid foundations. To me, this sliding seems like a fundamental change in the nature of the data extractors, and so I think it should be done in the base DataExtractor class. So I'd propose to move this into a separate patch, where we can make sure that all other DataExtractor APIs do something sensible even if the extractor has been "slid".

[clayborg]

Broke up with patch further by submitting:

[PATH] D46529: Add support to object files for accessing the .debug_types section
https://reviews.llvm.org/D46529

[PATCH] D46606: General cleanup to minimize the .debug_types patch
https://reviews.llvm.org/D46606

This makes this patch easier to read and focus only on the .debug_types patch

[labath]

The patch looks much better now, but I think we still need to discuss the data extractor sliding issue, as right now that's very hacky.

[probinson]

So, I've been attacking the type-units-in-.debug_info problem in LLVM, and what I've done (so far) in the LLVM parser is to factor out a DWARFUnitHeader class, and made DWARFUnit derive from that. This makes it feasible to parse a unit header before deciding what kind of unit we are looking at, which is necessary when .debug_info contains both type and compile units. I haven't posted that patch yet as it's an NFC prep for more work getting to the point of actually allowing mix-n-match units in .debug_info sections. But I'll post that patch today so people can at least look at it.

Mainly Greg and I should not take diverging approaches when we can converge instead.

[clayborg]

In D32167#1094173, @labath wrote:

The patch looks much better now, but I think we still need to discuss the data extractor sliding issue, as right now that's very hacky.

It makes things just work with .debug_info as is pretends that the .debug_types is just appended to the end of the .debug_info (offset wise). All info in .debug_types has no DW_FORM_ref_addr or any references that are outside of the current type unit so the info in .debug_types is position independent. I would like to keep this in DWARFDataExtractor only if possible as it is hacky and won't work in a general case. Feel free to suggest any other solution that doesn't require checked if the data is from .debug_types and adding an offset all over the code, but I couldn't think of one.

[clayborg]

I really need to get this in as well. It is a very important feature and I am trying to do the right thing and get it into open source so we don't end up keeping the patch locally. So any help on expediting any comments and iterations would be great.

[probinson]

I can't coment on the xcode and python parts. The C++ all looks plausible.
I think the "sliding" trick will be trickier in the LLVM parser, as it has to handle .o files (which might have multiple instances of .debug_types in v4, and multiple instances of .debug_info in v5) but is conceptually nice. The sliding should be kept to DWARFDataExtractor and not pushed down into DataExtractor, IMO, because the problem is (I think) pretty specific to DWARF.

[labath]

The thing here is that I don't see how can be done in a non-hacky way in DWARFDataExtractor while DWARFDataExtractor remains a (subclass of) DataExtractor. And it is a hack because as soon as you slide the extractor, a number of its functions become booby-trapped (explode as soon as you call them) or just return nonsensical results. And I'd like us to implement it in a way that it is not a hack, on any level.

I see several ways of achieving this (in decreasing order of intrusiveness):

1. do this in the DataExtractor class. This would require us to change (extend) the definition of the class. Right now, I would define this class as "a view of a range of bytes with a bunch of utility functions for accessing them". After this change it would become something like "a view of a range of bytes with a bunch of utility functions for accessing them, where the accessor functions automatically apply a bias to all offsets". That's precisely the kind of change you are making now, only the difference would be that we would state this up-front rather than doing it behind the DataExtractor's back and hoping it works. In practice, this would be as simple as adding another offset_t m_bias; field to the class and going through it's methods to make sure the do something reasonable with it. "Something reasonable" would generally mean "subtract it from the input offset", but there are a couple of functions that would need to do something different, (e.g., ValidOffset() would return true for values from m_bias to m_bias + (m_end-m_start) -1 because those are the only valid offsets, etc.). This is my preferred solution because it provides with a consistent DataExtractor interface (and in terms of code churn, I don't think it's harder to implement than any of the other solutions). It's true that dwarf type units will probably be the only user of these "slid" extractors, but I don't think that matters as their functionality can be defined in a clear and self-consistent manner without any references to dwarf.

2. Add the m_bias field to the DWARFDataExtractor class, but stop the class from publicly inheriting from DataExtractor (it can inherit privately, or have it as a member). The result of this is the same as the first option - we get to go through the DataExtractor interface and make sure all its functions do something sensible. The difference is that we may not need to go through that all methods, only those that are used for dwarf parsing; and the sliding trick can then be kept dwarf-specific.

3. Keep DWARFDataExtractor as-is, and add a new SlidingDWARFExtractor class instead (which will then to the m_bias trick and so on); then convert all places that need this to use the new class instead. This is a variation on the second option. I don't see it as particularly interesting, as it the DWARFDataExtractor is already a very small class, but it would still provide us self-consistent APIs everywhere.

I can also see some kind of a solution involving a ConcatenatingDataExtractor, which would contain two or more DataExtractors as it's backing store and then pretend they represent one continuous address space from 0 to sum(sizes) (no sliding necessary). This would achieve what you wanted at a higher level. In a way it would be nicer because there would be a single canonical bytestream for each symbol file and you wouldn't have to ask each DIE for it's own extractor, but I expect it to be a bit harder to implement, and may have a runtime cost.

The next solution which does not require sliding is to modify your previous patch D46606 and insert some kind of offset fixup code there. It seems that in every place you call the new GetData() function, you have an offset around (makes sense, as the reason you are calling GetData is because you want to parse something). We could modify that function to take a virtual (slid) offset as a parameter, and return an offset which is suitable for usage in the returned extractor. So, the usage would become something like std::tie(extractor, real_offset) = die.GetData(virtual_offset);. I'm not sure how nice this will end up in practice, but it's a possibility.

Let me know what you think of these. I'm open to other options as well. The thing I'd like avoid is an implementation that relies on putting objects in an invalid state or invoking undefined behavior.

[probinson]

I haven't given Pavel's comments as much thought as they deserve, but here are some of mine.

It's a worthwhile point that a class ought to represent some useful and reasonably well defined abstraction. DataExtractor has one, and DWARFDataExtractor builds on that (at least in the LLVM parser) by handling relocations; the most reasonable way to handle multiple buffers would be with multiple extractors, building on top of what's there rather than hacking the insides.

One aspect I am looking at is that the LLVM parser needs to handle .o files, which can have multiple .debug_types (in v4) or .debug_info (in v5) sections, because the units are broken up into COMDATs. So, the dumper needs to be able to handle an arbitrary number of either kind of section, where LLDB needs at most one of each section. One way of dealing with all this is to have a vector or deque of sections, each section has its DWARFDataExtractor and knows its own "base offset" which would be the sum of the sizes of the preceding sections. I don't know how awkward it might be to turn this into a meta-extractor that hid details from consumers, but then I think it probably would not be all that hard to have consumers be aware that there can be multiple lumps of DWARF and get the offsets right on their own.

[clayborg]

After reading Pavel's and Paul's comments, I will come up with a solution that allows multiple sections to be combined into one large section. That way it will be all legal and easy to extend. If we have just .debug_info, then we can use a base class that just calls directly through to DWARFDataExtractor. If we have more that one section that need to act as one section, I will make a new class that can put N sections together. I'll post a patch on its own that implements this with tests and we can iterate on that and make it work for all of our needs, then switch this patch over to use it. Sound good?

[probinson]

Go for it. It sounds like an abstraction that hides the multiple-section nature of the beast will particularly help LLDB; on the LLVM side the dumper will want to remain a bit more cognizant of the underlying sections. But we can surely leverage ideas from each other, which will ultimately help with converging on the Grand Unified Parser.

[labath]

Sounds like a plan.

[jankratochvil]

Just a ping, that ConcatenatingDataExtractor is still not coded, even as some draft patch? Just checking to prevent duplicating some existing work.
FYI providing a rebase with few simple conflicts resolved. I had to drop conflicting lldb.xcodeproj/project.pbxproj changes there.

[clayborg]

In D32167#1212783, @jankratochvil wrote:

Just a ping, that ConcatenatingDataExtractor is still not coded, even as some draft patch? Just checking to prevent duplicating some existing work.
FYI providing a rebase with few simple conflicts resolved. I had to drop conflicting lldb.xcodeproj/project.pbxproj changes there.

I have stalled on this due to performance. Anything I do will ruin the performance of 99% of the DWARF consumers for .debug_types which is rarely used by our clients. Right now we have one data extractor with only a virtual destructor.

[jankratochvil]

Just to clarify - I guess DWARFConcatenatingDataExtractor should replace all uses of DWARFDataExtractor in the DWARF/ subdirectory, whenever lldb::offset_t offset is used, right?
As the other option would be to remap just lldb::user_id_t but that would not solve much.
BTW my D40474 patch: DWZ is using DWARF references in both main file .debug_info and in DWZ common file .debug_info so at least one of them always needs to get the read-in DWARF offsets adjusted anyway in DWARFFormValue::Reference(). Then I needed adjustments also in DWARFDebugInfo::GetCompileUnitAranges() as the DWZ common file I had to place from offset 0 so that it can be shared between multiple DWARF files using it, therefore .debug_info of the main files (users of DWZ common file) need to be shifted.
Also the DWZ common file is a separate DWARFDebugInfo so there needs to be some dispatcher (DWARFDebugInfo::DWZRedirect() in my D40474 DWZ patch) for it anyway, which is why I do not find DWARFConcatenatingDataExtractor as much a win for DWZ.
OTOH I do not think the DWARF parsing performance matters too much after .debug_names (or the Apple index which I haven't/cannot test) is used, why it should?

[jankratochvil]

In D32167#1215821, @clayborg wrote:

I have stalled on this due to performance. Anything I do will ruin the performance of 99% of the DWARF consumers for .debug_types which is rarely used by our clients.

I have tried to implement it with acceptable performance as D51578.

[jankratochvil]

BTW I would welcome some direction forward (for our internal yearly team meeting) how to unblock this patch (and assuming DWZ D40474). I can code some other of the solutions described above by Pavel Labath or to performance-improve more D51578 (although then I can come with more DWARF performance improvements, at least re-introduce atomics to aid my mutexes from D40470).

[jankratochvil]

I could not updates this revision so an updated version of this patch is now: D54670

include/lldb/lldb-enumerations.h

   eSectionTypeGoSymtab,
   eSectionTypeAbsoluteAddress, // Dummy section for symbols with absolute
                                // address
+  eSectionTypeDWARFDebugTypes,
   eSectionTypeOther
 };
 

include/lldb/lldb-forward.h

 class DiagnosticManager;
 class Disassembler;
 class DumpValueObjectOptions;
+class DWARFDataExtractor;
 class DynamicCheckerFunctions;
 class DynamicLoader;
 class Editline;

packages/Python/lldbsuite/test/lldbinline.py

         self.buildGModules()
         self.do_test()
 
+    def __test_with_dwarf_type_units(self):
+        self.using_dsym = False
+        self.BuildMakefile()
+        self.buildDwarfTypeUnits()
+        self.do_test()
+
     def execute_user_command(self, __command):
         exec(__command, globals(), locals())
 

             "gmodules", target_platform, configuration.compiler):
         test.test_with_gmodules = ApplyDecoratorsToFunction(
             test._InlineTest__test_with_gmodules, decorators)
+    if test_categories.is_supported_on_platform(
+            "dwarf_type_units", target_platform, configuration.compiler):
+        test.test_with_gmodules = ApplyDecoratorsToFunction(
+            test._InlineTest__test_with_dwarf_type_units, decorators)
 
     # Add the test case to the globals, and hide InlineTest
     __globals.update({test_name: test})

packages/Python/lldbsuite/test/lldbtest.py

                 clean):
             raise Exception("Don't know how to build binary with gmodules")
 
+    def buildDwarfTypeUnits(
+            self,
+            architecture=None,
+            compiler=None,
+            dictionary=None,
+            clean=True):
+        """Platform specific way to build binaries with gmodules info."""
+        module = builder_module()
+        if not module.buildDwarfTypeUnits(
+                self,
+                architecture,
+                compiler,
+                dictionary,
+                clean):
+            raise Exception("Don't know how to build binary with DWARF type units")
+
     def buildGo(self):
         """Build the default go binary.
         """

                     gmodules_test_method.__name__ = gmodules_method_name
                     newattrs[gmodules_method_name] = gmodules_test_method
 
+                if "dwarf_type_units" in supported_categories:
+                    @decorators.add_test_categories(["dwarf_type_units"])
+                    @wraps(attrvalue)
+                    def gmodules_test_method(self, attrvalue=attrvalue):
+                        self.debug_info = "dwarf_type_units"
+                        return attrvalue(self)
+                    gmodules_method_name = attrname + "_dwarf_type_units"
+                    gmodules_test_method.__name__ = gmodules_method_name
+                    newattrs[gmodules_method_name] = gmodules_test_method
+
             else:
                 newattrs[attrname] = attrvalue
         return super(

         temp = os.path.join(os.getcwd(), template)
         with open(temp, 'r') as f:
             content = f.read()
-            
         public_api_dir = os.path.join(
             os.environ["LLDB_SRC"], "include", "lldb", "API")
 

         elif self.debug_info == "gmodules":
             return self.buildGModules(
                 architecture, compiler, dictionary, clean)
+        elif self.debug_info == "dwarf_type_units":
+            return self.buildDwarfTypeUnits(
+                architecture, compiler, dictionary, clean)
         else:
             self.fail("Can't build for debug info: %s" % self.debug_info)
 

packages/Python/lldbsuite/test/make/Makefile.rules

    MODULE_DEBUG_INFO_FLAGS += -gmodules
 endif
 
-DEBUG_INFO_FLAG ?= -g
+ifeq "$(DWARF_TYPE_UNITS)" "YES"
+  DEBUG_INFO_FLAG ?= -gdwarf-4
+else
+  DEBUG_INFO_FLAG ?= -g
+endif
 
 CFLAGS ?= $(DEBUG_INFO_FLAG) -O0 -fno-builtin
 ifeq "$(OS)" "Darwin"

   CFLAGS += -gsplit-dwarf
 endif
 
+ifeq "$(DWARF_TYPE_UNITS)" "YES"
+  CFLAGS += -fdebug-types-section
+endif
+
 ifeq "$(MAKE_GMODULES)" "YES"
   CFLAGS += -fmodules -gmodules
 endif

 
 #----------------------------------------------------------------------
 # From http://blog.melski.net/tag/debugging-makefiles/
-# 
+#
 # Usage: make print-CC print-CXX print-LD
 #----------------------------------------------------------------------
 print-%:

packages/Python/lldbsuite/test/plugins/builder_base.py

     # True signifies that we can handle building dwo.
     return True
 
-
 def buildGModules(
         sender=None,
         architecture=None,

     # True signifies that we can handle building with gmodules.
     return True
 
+def buildDwarfTypeUnits(
+        sender=None,
+        architecture=None,
+        compiler=None,
+        dictionary=None,
+        clean=True):
+    """Build the binaries with type units (type in  a .debug_types section)."""
+    commands = []
+    if clean:
+        commands.append([getMake(), "clean", getCmdLine(dictionary)])
+    commands.append([getMake(), "MAKE_DSYM=NO", "DWARF_TYPE_UNITS=YES",
+                    getArchSpec( architecture), getCCSpec(compiler),
+                    getCmdLine(dictionary)])
 
+    runBuildCommands(commands, sender=sender)
+    # True signifies that we can handle building dwo.
+    return True
+
 def cleanup(sender=None, dictionary=None):
     """Perform a platform-specific cleanup after the test."""
     #import traceback

packages/Python/lldbsuite/test/test_categories.py

 
 
 debug_info_categories = [
-    'dwarf', 'dwo', 'dsym', 'gmodules'
+    'dwarf', 'dwo', 'dsym', 'gmodules', 'dwarf_type_units'
 ]
 
 all_categories = {

     'dwo': 'Tests that can be run with DWO debug information',
     'dsym': 'Tests that can be run with DSYM debug information',
     'gmodules': 'Tests that can be run with -gmodules debug information',
+    'dwarf_type_units' : 'Tests using the DWARF type units (-fdebug-types-section)',
     'expression': 'Tests related to the expression parser',
     'libc++': 'Test for libc++ data formatters',
     'objc': 'Tests related to the Objective-C programming language support',

         if platform not in ["linux", "freebsd", "darwin", "macosx", "ios"]:
             return False
         return gmodules.is_compiler_clang_with_gmodules(compiler_path)
+    elif category == "dwarf_type_units":
+        return platform in ["linux", "freebsd"]
     return True
 
 

source/Core/Section.cpp

     return "dwarf-str";
   case eSectionTypeDWARFDebugStrOffsets:
     return "dwarf-str-offsets";
+  case eSectionTypeDWARFDebugTypes:
+      return "dwarf-types";
   case eSectionTypeELFSymbolTable:
     return "elf-symbol-table";
   case eSectionTypeELFDynamicSymbols:

source/Plugins/ObjectFile/ELF/ObjectFileELF.cpp

       static ConstString g_sect_name_dwarf_debug_str(".debug_str");
       static ConstString g_sect_name_dwarf_debug_str_offsets(
           ".debug_str_offsets");
+      static ConstString g_sect_name_dwarf_debug_types(".debug_types");
       static ConstString g_sect_name_dwarf_debug_abbrev_dwo(
           ".debug_abbrev.dwo");
       static ConstString g_sect_name_dwarf_debug_info_dwo(".debug_info.dwo");

         sect_type = eSectionTypeDWARFDebugRanges;
       else if (name == g_sect_name_dwarf_debug_str)
         sect_type = eSectionTypeDWARFDebugStr;
+      else if (name == g_sect_name_dwarf_debug_types)
+        sect_type = eSectionTypeDWARFDebugTypes;
       else if (name == g_sect_name_dwarf_debug_str_offsets)
         sect_type = eSectionTypeDWARFDebugStrOffsets;
       else if (name == g_sect_name_dwarf_debug_abbrev_dwo)

source/Plugins/ObjectFile/Mach-O/ObjectFileMachO.cpp

           case eSectionTypeDWARFDebugRanges:
           case eSectionTypeDWARFDebugStr:
           case eSectionTypeDWARFDebugStrOffsets:
+          case eSectionTypeDWARFDebugTypes:
           case eSectionTypeDWARFAppleNames:
           case eSectionTypeDWARFAppleTypes:
           case eSectionTypeDWARFAppleNamespaces:

                   static ConstString g_sect_name_dwarf_debug_ranges(
                       "__debug_ranges");
                   static ConstString g_sect_name_dwarf_debug_str("__debug_str");
+                  static ConstString g_sect_name_dwarf_debug_types(
+                      "__debug_types");
                   static ConstString g_sect_name_dwarf_apple_names(
                       "__apple_names");
                   static ConstString g_sect_name_dwarf_apple_types(

                     sect_type = eSectionTypeDWARFDebugRanges;
                   else if (section_name == g_sect_name_dwarf_debug_str)
                     sect_type = eSectionTypeDWARFDebugStr;
+                  else if (section_name == g_sect_name_dwarf_debug_types)
+                    sect_type = eSectionTypeDWARFDebugTypes;
                   else if (section_name == g_sect_name_dwarf_apple_names)
                     sect_type = eSectionTypeDWARFAppleNames;
                   else if (section_name == g_sect_name_dwarf_apple_types)

source/Plugins/ObjectFile/PECOFF/ObjectFilePECOFF.cpp

         static ConstString g_sect_name_dwarf_debug_pubtypes(".debug_pubtypes");
         static ConstString g_sect_name_dwarf_debug_ranges(".debug_ranges");
         static ConstString g_sect_name_dwarf_debug_str(".debug_str");
+        static ConstString g_sect_name_dwarf_debug_types(".debug_types");
         static ConstString g_sect_name_eh_frame(".eh_frame");
         static ConstString g_sect_name_go_symtab(".gosymtab");
         SectionType section_type = eSectionTypeOther;

           section_type = eSectionTypeDWARFDebugRanges;
         else if (const_sect_name == g_sect_name_dwarf_debug_str)
           section_type = eSectionTypeDWARFDebugStr;
+        else if (const_sect_name == g_sect_name_dwarf_debug_types)
+          section_type = eSectionTypeDWARFDebugTypes;
         else if (const_sect_name == g_sect_name_eh_frame)
           section_type = eSectionTypeEHFrame;
         else if (const_sect_name == g_sect_name_go_symtab)

source/Plugins/SymbolFile/DWARF/DIERef.cpp

   if (form_value.IsValid()) {
     const DWARFCompileUnit *dwarf_cu = form_value.GetCompileUnit();
     if (dwarf_cu) {
+      // Replace the compile unit with the type signature compile unit for
+      // type signature attributes.
+      if (form_value.Form() == DW_FORM_ref_sig8) {
+        uint64_t type_sig = form_value.Unsigned();
+        auto debug_info = dwarf_cu->GetSymbolFileDWARF()->DebugInfo();
+        auto type_cu = debug_info->GetTypeUnitForSignature(type_sig);
+        if (type_cu) {
+          cu_offset = type_cu->GetOffset();
+          die_offset = type_cu->GetTypeUnitDIEOffset();
+        }
+        return;
+      }
       if (dwarf_cu->GetBaseObjOffset() != DW_INVALID_OFFSET)
         cu_offset = dwarf_cu->GetBaseObjOffset();
       else

source/Plugins/SymbolFile/DWARF/DWARFASTParserClang.cpp

               if (form_value.BlockData()) {
                 Value initialValue(0);
                 Value memberOffset(0);
-                const DWARFDataExtractor &debug_info_data =
+                const DWARFDataExtractor &debug_info_data = die.GetData();
                     die.GetDWARF()->get_debug_info_data();
                 uint32_t block_length = form_value.Unsigned();
                 uint32_t block_offset =

               if (form_value.BlockData()) {
                 Value initialValue(0);
                 Value memberOffset(0);
-                const DWARFDataExtractor &debug_info_data =
-                    die.GetDWARF()->get_debug_info_data();
+                const DWARFDataExtractor &debug_info_data = die.GetData();
                 uint32_t block_length = form_value.Unsigned();
                 uint32_t block_offset =
                     form_value.BlockData() - debug_info_data.GetDataStart();

source/Plugins/SymbolFile/DWARF/DWARFASTParserGo.cpp

               if (form_value.BlockData()) {
                 Value initialValue(0);
                 Value memberOffset(0);
-                const DWARFDataExtractor &debug_info_data =
-                    die.GetDWARF()->get_debug_info_data();
+                const DWARFDataExtractor &debug_info_data = die.GetData();
                 uint32_t block_length = form_value.Unsigned();
                 uint32_t block_offset =
                     form_value.BlockData() - debug_info_data.GetDataStart();

source/Plugins/SymbolFile/DWARF/DWARFAttribute.cpp

   form_value.SetCompileUnit(cu);
   form_value.SetForm(FormAtIndex(i));
   lldb::offset_t offset = DIEOffsetAtIndex(i);
-  return form_value.ExtractValue(
-      cu->GetSymbolFileDWARF()->get_debug_info_data(), &offset);
+  return form_value.ExtractValue(cu->GetData(), &offset);
 }
 
 uint64_t DWARFAttributes::FormValueAsUnsigned(dw_attr_t attr,

source/Plugins/SymbolFile/DWARF/DWARFCompileUnit.h

   ~DWARFCompileUnit();
 
   bool Extract(const lldb_private::DWARFDataExtractor &debug_info,
-               lldb::offset_t *offset_ptr);
+               lldb::offset_t *offset_ptr, bool is_type_unit);
+  const lldb_private::DWARFDataExtractor &GetData() const;
   size_t ExtractDIEsIfNeeded(bool cu_die_only);
   DWARFDIE LookupAddress(const dw_addr_t address);
   size_t AppendDIEsWithTag(const dw_tag_t tag,

                            DWARFDIECollection &matching_dies,
                            uint32_t depth = UINT32_MAX) const;
   void Clear();
-  bool Verify(lldb_private::Stream *s) const;
   void Dump(lldb_private::Stream *s) const;
   dw_offset_t GetOffset() const { return m_offset; }
+  size_t GetLengthByteSize() const { return m_is_dwarf64 ? 12 : 4; }
   lldb::user_id_t GetID() const;
   uint32_t Size() const {
-    return m_is_dwarf64 ? 23
-                        : 11; /* Size in bytes of the compile unit header */
+    // Size in bytes of the compile or type unit header
+    // Start with the common size between compile and type units
+    uint32_t header_size = m_is_dwarf64 ? 23 : 11;
+    // Add the extra type unit size if needed
+    if (IsTypeUnit())
+      header_size += m_is_dwarf64 ? 16 : 12;
+    return header_size;
   }
   bool ContainsDIEOffset(dw_offset_t die_offset) const {
     return die_offset >= GetFirstDIEOffset() &&

   }
   dw_offset_t GetFirstDIEOffset() const { return m_offset + Size(); }
   dw_offset_t GetNextCompileUnitOffset() const {
-    return m_offset + m_length + (m_is_dwarf64 ? 12 : 4);
+    return m_offset + m_length + GetLengthByteSize();
   }
   size_t GetDebugInfoSize() const {
-    return m_length + (m_is_dwarf64 ? 12 : 4) - Size(); /* Size in bytes of the
-                                                           .debug_info data
-                                                           associated with this
-                                                           compile unit. */
+    // Size in bytes of the .debug_info or .debug_types data associated with
+    // this compile or type unit.
+    return m_length + GetLengthByteSize() - Size();
   }
   uint32_t GetLength() const { return m_length; }
   uint16_t GetVersion() const { return m_version; }

 
   dw_offset_t GetBaseObjOffset() const { return m_base_obj_offset; }
 
+  // Return true if this compile unit is a type unit.
+  bool IsTypeUnit() const { return m_type_offset != DW_INVALID_OFFSET; }
+  // Return the type signature for the type contained in this type unit. This
+  // value will not be valid if this compile unit is not a type unit.
+  uint64_t GetTypeSignature() const { return m_type_signature; }
+  // If this compile unit is a type unit, then return the DWARFDIE that
+  // respresents the type contained in the type unit, else return an invalid
+  // DIE.
+  DWARFDIE GetTypeUnitDIE();
+  // If this compile unit is a type unit, then return the DIE offset for the
+  // type contained in the type unit, else return an invalid DIE offset.
+  dw_offset_t GetTypeUnitDIEOffset() {
+    if (IsTypeUnit())
+      return m_offset + m_type_offset;
+    return DW_INVALID_OFFSET;
+  }
+  
+  // Find the DIE for any given type signature.
+  //
+  // This is a convenience function that allows anyone to use the current
+  // compile unit to access the DWARF and use its debug info to retrieve a DIE
+  // that represents a type given a type signature. This function will cause all
+  // DIEs in the type unit to be parsed, only call if you need the actual DIE
+  // object.
+  DWARFDIE FindTypeSignatureDIE(uint64_t type_sig) const;
+  
+  // Find the DIE offset for any given type signature.
+  //
+  // This is a convenience function that allows anyone to use the current
+  // compile unit to access the DWARF and use its debug info to retrieve a DIE
+  // offset that represents a type given a type signature. This function doesn't
+  // cause any debug information to be parsed, so if clients only need the
+  // DIE offset of a type signature, this function is more efficient than
+  // DWARFDIE FindTypeSignatureDIE(...) above.
+  dw_offset_t FindTypeSignatureDIEOffset(uint64_t type_sig) const;
+  
 protected:
   SymbolFileDWARF *m_dwarf2Data;
   std::unique_ptr<SymbolFileDWARFDwo> m_dwo_symbol_file;

   dw_offset_t m_base_obj_offset; // If this is a dwo compile unit this is the
                                  // offset of the base compile unit in the main
                                  // object file
+  uint64_t m_type_signature;     // Type signature contained in a type unit
+                                 // which will be valid (non-zero) for type
+                                 // units only.
+  dw_offset_t m_type_offset;     // Compile unit relative type offset for type
+                                 // units only.
 
   void ParseProducerInfo();
 

source/Plugins/SymbolFile/DWARF/DWARFCompileUnit.cpp

       m_producer_version_minor(0), m_producer_version_update(0),
       m_language_type(eLanguageTypeUnknown), m_is_dwarf64(false),
       m_is_optimized(eLazyBoolCalculate), m_addr_base(0),
-      m_ranges_base(0), m_base_obj_offset(DW_INVALID_OFFSET) {}
+      m_ranges_base(0), m_base_obj_offset(DW_INVALID_OFFSET),
+      m_type_signature(0), m_type_offset(DW_INVALID_OFFSET)
+{}
 
 DWARFCompileUnit::~DWARFCompileUnit() {}
 

 }
 
 bool DWARFCompileUnit::Extract(const DWARFDataExtractor &debug_info,
-                               lldb::offset_t *offset_ptr) {
+                               lldb::offset_t *offset_ptr, bool is_type_unit) {
   Clear();
 
   m_offset = *offset_ptr;

         m_dwarf2Data->get_debug_abbrev_data().ValidOffset(abbr_offset);
     bool addr_size_OK = ((m_addr_size == 4) || (m_addr_size == 8));
 
+    if (is_type_unit) {
+      m_type_signature = debug_info.GetU64(offset_ptr);
+      m_type_offset = debug_info.GetDWARFOffset(offset_ptr);
+    }
     if (length_OK && version_OK && addr_size_OK && abbr_offset_OK &&
-        abbr != NULL) {
+        abbr != NULL && (!is_type_unit || m_type_offset != DW_INVALID_OFFSET)) {
       m_abbrevs = abbr->GetAbbreviationDeclarationSet(abbr_offset);
       return true;
     }

   uint32_t depth = 0;
   // We are in our compile unit, parse starting at the offset
   // we were told to parse
-  const DWARFDataExtractor &debug_info_data =
-      m_dwarf2Data->get_debug_info_data();
+  const DWARFDataExtractor &debug_info_data = GetData();
   std::vector<uint32_t> die_index_stack;
   die_index_stack.reserve(32);
   die_index_stack.push_back(0);

   return m_abbrevs ? m_abbrevs->GetOffset() : DW_INVALID_OFFSET;
 }
 
-bool DWARFCompileUnit::Verify(Stream *s) const {
-  const DWARFDataExtractor &debug_info = m_dwarf2Data->get_debug_info_data();
-  bool valid_offset = debug_info.ValidOffset(m_offset);
-  bool length_OK = debug_info.ValidOffset(GetNextCompileUnitOffset() - 1);
-  bool version_OK = SymbolFileDWARF::SupportedVersion(m_version);
-  bool abbr_offset_OK =
-      m_dwarf2Data->get_debug_abbrev_data().ValidOffset(GetAbbrevOffset());
-  bool addr_size_OK = ((m_addr_size == 4) || (m_addr_size == 8));
-  if (valid_offset && length_OK && version_OK && addr_size_OK &&
-      abbr_offset_OK) {
-    return true;
-  } else {
-    s->Printf("    0x%8.8x: ", m_offset);
-    DumpDataExtractor(m_dwarf2Data->get_debug_info_data(), s, m_offset,
-                      lldb::eFormatHex, 1, Size(), 32, LLDB_INVALID_ADDRESS, 0,
-                      0);
-    s->EOL();
-    if (valid_offset) {
-      if (!length_OK)
-        s->Printf("        The length (0x%8.8x) for this compile unit is too "
-                  "large for the .debug_info provided.\n",
-                  m_length);
-      if (!version_OK)
-        s->Printf("        The 16 bit compile unit header version is not "
-                  "supported.\n");
-      if (!abbr_offset_OK)
-        s->Printf("        The offset into the .debug_abbrev section (0x%8.8x) "
-                  "is not valid.\n",
-                  GetAbbrevOffset());
-      if (!addr_size_OK)
-        s->Printf("        The address size is unsupported: 0x%2.2x\n",
-                  m_addr_size);
-    } else
-      s->Printf("        The start offset of the compile unit header in the "
-                ".debug_info is invalid.\n");
-  }
-  return false;
-}
-
 void DWARFCompileUnit::Dump(Stream *s) const {
   s->Printf("0x%8.8x: Compile Unit: length = 0x%8.8x, version = 0x%4.4x, "
             "abbr_offset = 0x%8.8x, addr_size = 0x%2.2x (next CU at "

 
 void DWARFCompileUnit::BuildAddressRangeTable(
     SymbolFileDWARF *dwarf2Data, DWARFDebugAranges *debug_aranges) {
+
+  // No addresses are found in a type unit.
+  if (IsTypeUnit())
+    return;
+
   // This function is usually called if there in no .debug_aranges section
   // in order to produce a compile unit level set of address ranges that
   // is accurate.

 lldb::ByteOrder DWARFCompileUnit::GetByteOrder() const {
   return m_dwarf2Data->GetObjectFile()->GetByteOrder();
 }
+
+DWARFDIE DWARFCompileUnit::GetTypeUnitDIE() {
+  if (IsTypeUnit()) {
+    // The type offset is compile unit relative, so we need to add the compile
+    // unit offset to ensure we get the correct DIE.
+    return GetDIE(GetTypeUnitDIEOffset());
+  }
+  return DWARFDIE();
+}
+
+DWARFDIE DWARFCompileUnit::FindTypeSignatureDIE(uint64_t type_sig) const {
+  auto cu = m_dwarf2Data->DebugInfo()->GetTypeUnitForSignature(type_sig);
+  if (cu)
+    return cu->GetTypeUnitDIE();
+  return DWARFDIE();
+}
+
+dw_offset_t DWARFCompileUnit::FindTypeSignatureDIEOffset(uint64_t type_sig) const {
+  auto cu = m_dwarf2Data->DebugInfo()->GetTypeUnitForSignature(type_sig);
+  if (cu)
+    return cu->GetTypeUnitDIEOffset();
+  return DW_INVALID_OFFSET;
+}
+
+const lldb_private::DWARFDataExtractor &DWARFCompileUnit::GetData() const {
+  // In DWARF 5, type units are in the .debug_info section. Prior to DWARF 5
+  // type units are in the .debug_types section.
+  if (IsTypeUnit() && GetVersion() < 5)
+    return m_dwarf2Data->get_debug_types_data();
+  else
+    return m_dwarf2Data->get_debug_info_data();
+}
+

source/Plugins/SymbolFile/DWARF/DWARFDIE.h

   DWARFDIE
   GetContainingDWOModuleDIE() const;
 
+  // Get the data that contains the attribute values for this DIE. Support
+  // for .debug_types means that any DIE can have its data either in the
+  // .debug_info or the .debug_types section so anyone extracting data from
+  // a DIE must use the correct data.
+  //
+  // Clients must validate that this object is valid before calling this.
+  const lldb_private::DWARFDataExtractor &GetData() const;
+
   //----------------------------------------------------------------------
   // Accessing information about a DIE
   //----------------------------------------------------------------------

source/Plugins/SymbolFile/DWARF/DWARFDIE.cpp

 bool operator!=(const DWARFDIE &lhs, const DWARFDIE &rhs) {
   return lhs.GetDIE() != rhs.GetDIE() || lhs.GetCU() != rhs.GetCU();
 }
+
+const DWARFDataExtractor &DWARFDIE::GetData() const {
+  // Clients must check if this DIE is valid before calling this function.
+  assert(IsValid());
+  return m_cu->GetData();
+}

source/Plugins/SymbolFile/DWARF/DWARFDataExtractor.h

   dw_offset_t GetDWARFOffset(lldb::offset_t *offset_ptr) const;
 
   size_t GetDWARFSizeofInitialLength() const { return m_is_dwarf64 ? 12 : 4; }
+  size_t GetDWARFSizeOfOffset() const { return m_is_dwarf64 ? 8 : 4; }
 
   bool IsDWARF64() const { return m_is_dwarf64; }
 
+  //------------------------------------------------------------------
+  /// Slide the data in the buffer so that access to the data will
+  /// start at offset \a offset.
+  ///
+  /// This is currently used to provide access to the .debug_types
+  /// section and pretend it starts at at offset of the size of the
+  /// .debug_info section. This allows a minimally invasive change to
+  /// add support for .debug_types by allowing all DIEs to have unique
+  /// offsets and thus allowing no code to change in the DWARF parser.
+  /// Modifying the offsets in the .debug_types doesn't affect
+  /// anything because since all info in the .debug_types is type unit
+  /// relative and no types within a type unit can refer to any DIEs
+  /// outside of the type unit without using DW_AT_signature. It also
+  /// sets us up to move to DWARF5 where there is no .debug_types
+  /// section as compile units and type units are in the .debug_info.
+  //------------------------------------------------------------------
+  void OffsetData(lldb::offset_t offset)
+  {
+    if (GetByteSize())
+      m_start -= offset;
+  }
+
 protected:
   mutable bool m_is_dwarf64;
 };

source/Plugins/SymbolFile/DWARF/DWARFDataExtractor.cpp

 
 dw_offset_t
 DWARFDataExtractor::GetDWARFOffset(lldb::offset_t *offset_ptr) const {
-  return GetMaxU64(offset_ptr, m_is_dwarf64 ? 8 : 4);
+  return GetMaxU64(offset_ptr, GetDWARFSizeOfOffset());
 }
 }

source/Plugins/SymbolFile/DWARF/DWARFDebugInfo.h

 
 #include <map>
 #include <vector>
+#include <unordered_map>
 
 #include "DWARFDIE.h"
 #include "SymbolFileDWARF.h"

   size_t GetNumCompileUnits();
   bool ContainsCompileUnit(const DWARFCompileUnit *cu) const;
   DWARFCompileUnit *GetCompileUnitAtIndex(uint32_t idx);
+  DWARFCompileUnit *GetTypeUnitForSignature(uint64_t type_sig);
   DWARFCompileUnit *GetCompileUnit(dw_offset_t cu_offset,
                                    uint32_t *idx_ptr = NULL);
   DWARFCompileUnit *GetCompileUnitContainingDIEOffset(dw_offset_t die_offset);

   DWARFDIE GetDIEForDIEOffset(dw_offset_t die_offset);
   DWARFDIE GetDIE(const DIERef &die_ref);
 
-  void Dump(lldb_private::Stream *s, const uint32_t die_offset,
-            const uint32_t recurse_depth);
-  static void Parse(SymbolFileDWARF *parser, Callback callback, void *userData);
-  static void Verify(lldb_private::Stream *s, SymbolFileDWARF *dwarf2Data);
-  static void Dump(lldb_private::Stream *s, SymbolFileDWARF *dwarf2Data,
-                   const uint32_t die_offset, const uint32_t recurse_depth);
-  bool Find(const char *name, bool ignore_case,
-            std::vector<dw_offset_t> &die_offsets) const;
-  bool Find(lldb_private::RegularExpression &re,
-            std::vector<dw_offset_t> &die_offsets) const;
-
   enum {
     eDumpFlag_Verbose = (1 << 0),  // Verbose dumping
     eDumpFlag_ShowForm = (1 << 1), // Show the DW_form type

 
 protected:
   typedef std::shared_ptr<DWARFCompileUnit> DWARFCompileUnitSP;
+  typedef std::unordered_map<uint64_t, uint32_t> TypeSignatureMap;
 
   static bool OffsetLessThanCompileUnitOffset(dw_offset_t offset,
                                               const DWARFCompileUnitSP &cu_sp);

   //----------------------------------------------------------------------
   SymbolFileDWARF *m_dwarf2Data;
   CompileUnitColl m_compile_units;
+  TypeSignatureMap m_type_sig_to_cu_index;
   std::unique_ptr<DWARFDebugAranges>
       m_cu_aranges_ap; // A quick address to compile unit table
 

source/Plugins/SymbolFile/DWARF/DWARFDebugInfo.cpp

   if (m_compile_units.empty()) {
     if (m_dwarf2Data != NULL) {
       lldb::offset_t offset = 0;
-      const DWARFDataExtractor &debug_info_data =
-          m_dwarf2Data->get_debug_info_data();
+      auto debug_info_data = m_dwarf2Data->get_debug_info_data();
       while (debug_info_data.ValidOffset(offset)) {
         DWARFCompileUnitSP cu_sp(new DWARFCompileUnit(m_dwarf2Data));
         // Out of memory?

         if (cu_sp.get() == NULL)
           break;
 
-        if (cu_sp->Extract(debug_info_data, &offset) == false)
+        if (cu_sp->Extract(debug_info_data, &offset, false) == false)
           break;
 
         m_compile_units.push_back(cu_sp);

 
         offset = cu_sp->GetNextCompileUnitOffset();
       }
+      
+      auto debug_types_data = m_dwarf2Data->get_debug_types_data();
+      offset = debug_info_data.GetByteSize();
+      while (debug_types_data.ValidOffset(offset)) {
+        DWARFCompileUnitSP cu_sp(new DWARFCompileUnit(m_dwarf2Data));
+        // Out of memory?
+        if (!cu_sp)
+          break;
+        
+        if (cu_sp->Extract(debug_types_data, &offset, true) == false)
+          break;
+        
+        m_type_sig_to_cu_index[cu_sp->GetTypeSignature()] = m_compile_units.size();
+
+        m_compile_units.push_back(cu_sp);
+        
+        offset = cu_sp->GetNextCompileUnitOffset();
+      }
     }
   }
 }

   return cu;
 }
 
+DWARFCompileUnit *DWARFDebugInfo::GetTypeUnitForSignature(uint64_t type_sig) {
+  auto pos = m_type_sig_to_cu_index.find(type_sig);
+  if (pos != m_type_sig_to_cu_index.end())
+    return GetCompileUnitAtIndex(pos->second);
+  return nullptr;
+}
+
 bool DWARFDebugInfo::ContainsCompileUnit(const DWARFCompileUnit *cu) const {
   // Not a verify efficient function, but it is handy for use in assertions
   // to make sure that a compile unit comes from a debug information file.

     return cu->GetDIE(die_ref.die_offset);
   return DWARFDIE(); // Not found
 }
-
-//----------------------------------------------------------------------
-// Parse
-//
-// Parses the .debug_info section and uses the .debug_abbrev section
-// and various other sections in the SymbolFileDWARF class and calls the
-// supplied callback function each time a compile unit header, or debug
-// information entry is successfully parsed. This function can be used
-// for different tasks such as parsing the file contents into a
-// structured data, dumping, verifying and much more.
-//----------------------------------------------------------------------
-void DWARFDebugInfo::Parse(SymbolFileDWARF *dwarf2Data, Callback callback,
-                           void *userData) {
-  if (dwarf2Data) {
-    lldb::offset_t offset = 0;
-    uint32_t depth = 0;
-    DWARFCompileUnitSP cu(new DWARFCompileUnit(dwarf2Data));
-    if (cu.get() == NULL)
-      return;
-    DWARFDebugInfoEntry die;
-
-    while (cu->Extract(dwarf2Data->get_debug_info_data(), &offset)) {
-      const dw_offset_t next_cu_offset = cu->GetNextCompileUnitOffset();
-
-      depth = 0;
-      // Call the callback function with no DIE pointer for the compile unit
-      // and get the offset that we are to continue to parse from
-      offset = callback(dwarf2Data, cu.get(), NULL, offset, depth, userData);
-
-      // Make sure we are within our compile unit
-      if (offset < next_cu_offset) {
-        // We are in our compile unit, parse starting at the offset
-        // we were told to parse
-        bool done = false;
-        while (!done && die.Extract(dwarf2Data, cu.get(), &offset)) {
-          // Call the callback function with DIE pointer that falls within the
-          // compile unit
-          offset =
-              callback(dwarf2Data, cu.get(), &die, offset, depth, userData);
-
-          if (die.IsNULL()) {
-            if (depth)
-              --depth;
-            else
-              done = true; // We are done with this compile unit!
-          } else if (die.HasChildren())
-            ++depth;
-        }
-      }
-
-      // Make sure the offset returned is valid, and if not stop parsing.
-      // Returning DW_INVALID_OFFSET from this callback is a good way to end
-      // all parsing
-      if (!dwarf2Data->get_debug_info_data().ValidOffset(offset))
-        break;
-
-      // See if during the callback anyone retained a copy of the compile
-      // unit other than ourselves and if so, let whomever did own the object
-      // and create a new one for our own use!
-      if (!cu.unique())
-        cu.reset(new DWARFCompileUnit(dwarf2Data));
-
-      // Make sure we start on a proper
-      offset = next_cu_offset;
-    }
-  }
-}
-
-typedef struct DumpInfo {
-  DumpInfo(Stream *init_strm, uint32_t off, uint32_t depth)
-      : strm(init_strm), die_offset(off), recurse_depth(depth),
-        found_depth(UINT32_MAX), found_die(false), ancestors() {}
-  Stream *strm;
-  const uint32_t die_offset;
-  const uint32_t recurse_depth;
-  uint32_t found_depth;
-  bool found_die;
-  std::vector<DWARFDebugInfoEntry> ancestors;
-
-  DISALLOW_COPY_AND_ASSIGN(DumpInfo);
-} DumpInfo;
-
-//----------------------------------------------------------------------
-// DumpCallback
-//
-// A callback function for the static DWARFDebugInfo::Parse() function
-// that gets called each time a compile unit header or debug information
-// entry is successfully parsed.
-//
-// This function dump DWARF information and obey recurse depth and
-// whether a single DIE is to be dumped (or all of the data).
-//----------------------------------------------------------------------
-static dw_offset_t DumpCallback(SymbolFileDWARF *dwarf2Data,
-                                DWARFCompileUnit *cu, DWARFDebugInfoEntry *die,
-                                const dw_offset_t next_offset,
-                                const uint32_t curr_depth, void *userData) {
-  DumpInfo *dumpInfo = (DumpInfo *)userData;
-  Stream *s = dumpInfo->strm;
-  bool show_parents =
-      s->GetFlags().Test(DWARFDebugInfo::eDumpFlag_ShowAncestors);
-
-  if (die) {
-    // Are we dumping everything?
-    if (dumpInfo->die_offset == DW_INVALID_OFFSET) {
-      // Yes we are dumping everything. Obey our recurse level though
-      if (curr_depth < dumpInfo->recurse_depth)
-        die->Dump(dwarf2Data, cu, *s, 0);
-    } else {
-      // We are dumping a specific DIE entry by offset
-      if (dumpInfo->die_offset == die->GetOffset()) {
-        // We found the DIE we were looking for, dump it!
-        if (show_parents) {
-          s->SetIndentLevel(0);
-          const uint32_t num_ancestors = dumpInfo->ancestors.size();
-          if (num_ancestors > 0) {
-            for (uint32_t i = 0; i < num_ancestors - 1; ++i) {
-              dumpInfo->ancestors[i].Dump(dwarf2Data, cu, *s, 0);
-              s->IndentMore();
-            }
-          }
-        }
-
-        dumpInfo->found_depth = curr_depth;
-
-        die->Dump(dwarf2Data, cu, *s, 0);
-
-        // Note that we found the DIE we were looking for
-        dumpInfo->found_die = true;
-
-        // Since we are dumping a single DIE, if there are no children we are
-        // done!
-        if (!die->HasChildren() || dumpInfo->recurse_depth == 0)
-          return DW_INVALID_OFFSET; // Return an invalid address to end parsing
-      } else if (dumpInfo->found_die) {
-        // Are we done with all the children?
-        if (curr_depth <= dumpInfo->found_depth)
-          return DW_INVALID_OFFSET;
-
-        // We have already found our DIE and are printing it's children. Obey
-        // our recurse depth and return an invalid offset if we get done
-        // dumping all of the children
-        if (dumpInfo->recurse_depth == UINT32_MAX ||
-            curr_depth <= dumpInfo->found_depth + dumpInfo->recurse_depth)
-          die->Dump(dwarf2Data, cu, *s, 0);
-      } else if (dumpInfo->die_offset > die->GetOffset()) {
-        if (show_parents)
-          dumpInfo->ancestors.back() = *die;
-      }
-    }
-
-    // Keep up with our indent level
-    if (die->IsNULL()) {
-      if (show_parents)
-        dumpInfo->ancestors.pop_back();
-
-      if (curr_depth <= 1)
-        return cu->GetNextCompileUnitOffset();
-      else
-        s->IndentLess();
-    } else if (die->HasChildren()) {
-      if (show_parents) {
-        DWARFDebugInfoEntry null_die;
-        dumpInfo->ancestors.push_back(null_die);
-      }
-      s->IndentMore();
-    }
-  } else {
-    if (cu == NULL)
-      s->PutCString("NULL - cu");
-    // We have a compile unit, reset our indent level to zero just in case
-    s->SetIndentLevel(0);
-
-    // See if we are dumping everything?
-    if (dumpInfo->die_offset == DW_INVALID_OFFSET) {
-      // We are dumping everything
-      if (cu) {
-        cu->Dump(s);
-        return cu->GetFirstDIEOffset(); // Return true to parse all DIEs in this
-                                        // Compile Unit
-      } else {
-        return DW_INVALID_OFFSET;
-      }
-    } else {
-      if (show_parents) {
-        dumpInfo->ancestors.clear();
-        dumpInfo->ancestors.resize(1);
-      }
-
-      // We are dumping only a single DIE possibly with it's children and
-      // we must find it's compile unit before we can dump it properly
-      if (cu && dumpInfo->die_offset < cu->GetFirstDIEOffset()) {
-        // Not found, maybe the DIE offset provided wasn't correct?
-        //  *ostrm_ptr << "DIE at offset " << HEX32 << dumpInfo->die_offset << "
-        //  was not found." << endl;
-        return DW_INVALID_OFFSET;
-      } else {
-        // See if the DIE is in this compile unit?
-        if (cu && dumpInfo->die_offset < cu->GetNextCompileUnitOffset()) {
-          return next_offset;
-          //  // We found our compile unit that contains our DIE, just skip to
-          //  dumping the requested DIE...
-          //  return dumpInfo->die_offset;
-        } else {
-          // Skip to the next compile unit as the DIE isn't in the current one!
-          if (cu) {
-            return cu->GetNextCompileUnitOffset();
-          } else {
-            return DW_INVALID_OFFSET;
-          }
-        }
-      }
-    }
-  }
-
-  // Just return the current offset to parse the next CU or DIE entry
-  return next_offset;
-}
-
-//----------------------------------------------------------------------
-// Dump
-//
-// Dump the information in the .debug_info section to the specified
-// ostream. If die_offset is valid, a single DIE will be dumped. If the
-// die_offset is invalid, all the DWARF information will be dumped. Both
-// cases will obey a "recurse_depth" or how deep to traverse into the
-// children of each DIE entry. A recurse_depth of zero will dump all
-// compile unit headers. A recurse_depth of 1 will dump all compile unit
-// headers and the DW_TAG_compile unit tags. A depth of 2 will also
-// dump all types and functions.
-//----------------------------------------------------------------------
-void DWARFDebugInfo::Dump(Stream *s, SymbolFileDWARF *dwarf2Data,
-                          const uint32_t die_offset,
-                          const uint32_t recurse_depth) {
-  DumpInfo dumpInfo(s, die_offset, recurse_depth);
-  s->PutCString(".debug_info contents");
-  if (dwarf2Data->get_debug_info_data().GetByteSize() > 0) {
-    if (die_offset == DW_INVALID_OFFSET)
-      s->PutCString(":\n");
-    else {
-      s->Printf(" for DIE entry at .debug_info[0x%8.8x]", die_offset);
-      if (recurse_depth != UINT32_MAX)
-        s->Printf(" recursing %u levels deep.", recurse_depth);
-      s->EOL();
-    }
-  } else {
-    s->PutCString(": < EMPTY >\n");
-    return;
-  }
-  DWARFDebugInfo::Parse(dwarf2Data, DumpCallback, &dumpInfo);
-}
-
-//----------------------------------------------------------------------
-// Dump
-//
-// Dump the contents of this DWARFDebugInfo object as has been parsed
-// and/or modified after it has been parsed.
-//----------------------------------------------------------------------
-void DWARFDebugInfo::Dump(Stream *s, const uint32_t die_offset,
-                          const uint32_t recurse_depth) {
-  DumpInfo dumpInfo(s, die_offset, recurse_depth);
-
-  s->PutCString("Dumping .debug_info section from internal representation\n");
-
-  CompileUnitColl::const_iterator pos;
-  uint32_t curr_depth = 0;
-  ParseCompileUnitHeadersIfNeeded();
-  for (pos = m_compile_units.begin(); pos != m_compile_units.end(); ++pos) {
-    DWARFCompileUnit *cu = pos->get();
-    DumpCallback(m_dwarf2Data, cu, NULL, 0, curr_depth, &dumpInfo);
-
-    const DWARFDIE die = cu->DIE();
-    if (die)
-      die.Dump(s, recurse_depth);
-  }
-}
-
-//----------------------------------------------------------------------
-// FindCallbackString
-//
-// A callback function for the static DWARFDebugInfo::Parse() function
-// that gets called each time a compile unit header or debug information
-// entry is successfully parsed.
-//
-// This function will find the die_offset of any items whose DW_AT_name
-// matches the given string
-//----------------------------------------------------------------------
-typedef struct FindCallbackStringInfoTag {
-  const char *name;
-  bool ignore_case;
-  RegularExpression *regex;
-  vector<dw_offset_t> &die_offsets;
-} FindCallbackStringInfo;
-
-static dw_offset_t
-FindCallbackString(SymbolFileDWARF *dwarf2Data, DWARFCompileUnit *cu,
-                   DWARFDebugInfoEntry *die, const dw_offset_t next_offset,
-                   const uint32_t curr_depth, void *userData) {
-  FindCallbackStringInfo *info = (FindCallbackStringInfo *)userData;
-
-  if (!die)
-    return next_offset;
-
-  const char *die_name = die->GetName(dwarf2Data, cu);
-  if (!die_name)
-    return next_offset;
-
-  if (info->regex) {
-    if (info->regex->Execute(llvm::StringRef(die_name)))
-      info->die_offsets.push_back(die->GetOffset());
-  } else {
-    if ((info->ignore_case ? strcasecmp(die_name, info->name)
-                           : strcmp(die_name, info->name)) == 0)
-      info->die_offsets.push_back(die->GetOffset());
-  }
-
-  // Just return the current offset to parse the next CU or DIE entry
-  return next_offset;
-}
-
-//----------------------------------------------------------------------
-// Find
-//
-// Finds all DIE that have a specific DW_AT_name attribute by manually
-// searching through the debug information (not using the
-// .debug_pubnames section). The string must match the entire name
-// and case sensitive searches are an option.
-//----------------------------------------------------------------------
-bool DWARFDebugInfo::Find(const char *name, bool ignore_case,
-                          vector<dw_offset_t> &die_offsets) const {
-  die_offsets.clear();
-  if (name && name[0]) {
-    FindCallbackStringInfo info = {name, ignore_case, NULL, die_offsets};
-    DWARFDebugInfo::Parse(m_dwarf2Data, FindCallbackString, &info);
-  }
-  return !die_offsets.empty();
-}
-
-//----------------------------------------------------------------------
-// Find
-//
-// Finds all DIE that have a specific DW_AT_name attribute by manually
-// searching through the debug information (not using the
-// .debug_pubnames section). The string must match the supplied regular
-// expression.
-//----------------------------------------------------------------------
-bool DWARFDebugInfo::Find(RegularExpression &re,
-                          vector<dw_offset_t> &die_offsets) const {
-  die_offsets.clear();
-  FindCallbackStringInfo info = {NULL, false, &re, die_offsets};
-  DWARFDebugInfo::Parse(m_dwarf2Data, FindCallbackString, &info);
-  return !die_offsets.empty();
-}

source/Plugins/SymbolFile/DWARF/DWARFDebugInfoEntry.cpp

 bool DWARFDebugInfoEntry::Extract(SymbolFileDWARF *dwarf2Data,
                                   const DWARFCompileUnit *cu,
                                   lldb::offset_t *offset_ptr) {
-  const DWARFDataExtractor &debug_info_data = dwarf2Data->get_debug_info_data();
+  const DWARFDataExtractor &debug_info_data = cu->GetData();
   //    const DWARFDataExtractor& debug_str_data =
   //    dwarf2Data->get_debug_str_data();
   const uint32_t cu_end_offset = cu->GetNextCompileUnitOffset();

   lldb::ModuleSP module = dwarf2Data->GetObjectFile()->GetModule();
 
   if (abbrevDecl) {
-    const DWARFDataExtractor &debug_info_data =
-        dwarf2Data->get_debug_info_data();
+    const DWARFDataExtractor &debug_info_data = cu->GetData();
 
     if (!debug_info_data.ValidOffset(offset))
       return false;

 void DWARFDebugInfoEntry::Dump(SymbolFileDWARF *dwarf2Data,
                                const DWARFCompileUnit *cu, Stream &s,
                                uint32_t recurse_depth) const {
-  const DWARFDataExtractor &debug_info_data = dwarf2Data->get_debug_info_data();
+  const DWARFDataExtractor &debug_info_data = cu->GetData();
   lldb::offset_t offset = m_offset;
 
   if (debug_info_data.ValidOffset(offset)) {

   }
 
   if (abbrevDecl) {
-    const DWARFDataExtractor &debug_info_data =
-        dwarf2Data->get_debug_info_data();
+    const DWARFDataExtractor &debug_info_data = cu->GetData();
 
     if (fixed_form_sizes.Empty())
       fixed_form_sizes = DWARFFormValue::GetFixedFormSizesForAddressSize(

     uint32_t attr_idx = abbrevDecl->FindAttributeIndex(attr);
 
     if (attr_idx != DW_INVALID_INDEX) {
-      const DWARFDataExtractor &debug_info_data =
-          dwarf2Data->get_debug_info_data();
+      const DWARFDataExtractor &debug_info_data = cu->GetData();
 
       uint32_t idx = 0;
       while (idx < attr_idx)

         // the DWARF data was mmap'ed, the backing file might have been modified
         // which is bad news.
         const uint64_t abbrev_code =
-            dwarf2Data->get_debug_info_data().GetULEB128(&offset);
+            cu->GetData().GetULEB128(&offset);
 
         if (abbrev_decl->Code() == abbrev_code)
           return abbrev_decl;

source/Plugins/SymbolFile/DWARF/DWARFFormValue.cpp

     die_offset += m_cu->GetOffset();
     break;
 
+  case DW_FORM_ref_sig8:
+    // CU must be valid since we will need to back up into the debug info and
+    // find the DIE offset of the type in the type unit.
+    assert(m_cu);
+      die_offset = m_cu->FindTypeSignatureDIEOffset(m_value.value.uval);
+      break;
+
   default:
     break;
   }

     die_offset += base_offset;
     break;
 
+  case DW_FORM_ref_sig8:
+    // CU must be valid since we will need to back up into the debug info and
+    // find the DIE offset of the type in the type unit.
+    assert(m_cu);
+    die_offset = m_cu->FindTypeSignatureDIEOffset(m_value.value.uval) +
+        base_offset;
+    break;
+      
   default:
     break;
   }

source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.h

   const lldb_private::DWARFDataExtractor &get_debug_ranges_data();
   const lldb_private::DWARFDataExtractor &get_debug_str_data();
   const lldb_private::DWARFDataExtractor &get_debug_str_offsets_data();
+  const lldb_private::DWARFDataExtractor &get_debug_types_data();
   const lldb_private::DWARFDataExtractor &get_apple_names_data();
   const lldb_private::DWARFDataExtractor &get_apple_types_data();
   const lldb_private::DWARFDataExtractor &get_apple_namespaces_data();

   DWARFDataSegment m_data_debug_ranges;
   DWARFDataSegment m_data_debug_str;
   DWARFDataSegment m_data_debug_str_offsets;
+  DWARFDataSegment m_data_debug_types;
   DWARFDataSegment m_data_apple_names;
   DWARFDataSegment m_data_apple_types;
   DWARFDataSegment m_data_apple_namespaces;

source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.cpp

 const DWARFDataExtractor &
 SymbolFileDWARF::GetCachedSectionData(lldb::SectionType sect_type,
                                       DWARFDataSegment &data_segment) {
-  llvm::call_once(data_segment.m_flag, [this, sect_type, &data_segment] {
+  llvm::call_once(data_segment.m_flag, [&] {
     this->LoadSectionData(sect_type, std::ref(data_segment.m_data));
+    if (sect_type == eSectionTypeDWARFDebugTypes) {
+      // To add .debug_types support with minimally invasive changes, we pretend
+      // that any DIEs in .debug_types start at the end of the .debug_info
+      // section. All info in .debug_types is relative and has no external DIE
+      // references unless thay are DW_AT_signature references, so the DIE
+      // offset for things in the .debug_types. If we do this, then we can
+      // just add the type units to the compile units collection and treat all
+      // information just as we do for all other information in the DWARF and
+      // everything just works. If we were to try to split this out, we would
+      // end up having to change a TON of code. Also DWARF 5 will have compile
+      // and type units in the .debug_info, so coding it this way will prepare
+      // use for an easy transition to DWARF 5.
+      uint64_t debug_info_size = get_debug_info_data().GetByteSize();
+      data_segment.m_data.OffsetData(debug_info_size);
+    }
   });
   return data_segment.m_data;
 }

                               m_data_debug_str_offsets);
 }
 
+const DWARFDataExtractor &SymbolFileDWARF::get_debug_types_data() {
+  return GetCachedSectionData(eSectionTypeDWARFDebugTypes, m_data_debug_types);
+}
+
 const DWARFDataExtractor &SymbolFileDWARF::get_apple_names_data() {
   return GetCachedSectionData(eSectionTypeDWARFAppleNames, m_data_apple_names);
 }

               location_is_const_value_data = true;
               // The constant value will be either a block, a data value or a
               // string.
-              const DWARFDataExtractor &debug_info_data = get_debug_info_data();
+              auto debug_info_data = die.GetData();
               if (DWARFFormValue::IsBlockForm(form_value.Form())) {
                 // Retrieve the value as a block expression.
                 uint32_t block_offset =

             location_is_const_value_data = false;
             has_explicit_location = true;
             if (DWARFFormValue::IsBlockForm(form_value.Form())) {
-              const DWARFDataExtractor &debug_info_data = get_debug_info_data();
+              const DWARFDataExtractor &debug_info_data = die.GetData();
 
               uint32_t block_offset =
                   form_value.BlockData() - debug_info_data.GetDataStart();

source/Symbol/ObjectFile.cpp

           case eSectionTypeDWARFDebugRanges:
           case eSectionTypeDWARFDebugStr:
           case eSectionTypeDWARFDebugStrOffsets:
+          case eSectionTypeDWARFDebugTypes:
           case eSectionTypeDWARFAppleNames:
           case eSectionTypeDWARFAppleTypes:
           case eSectionTypeDWARFAppleNamespaces: