Fix expression evaluation inside lambda functions for gcc

[tberghammer]

Summary

Fix expression evaluation inside lambda functions for gcc

gcc emits the artificial struct created for lambda expressions inside a
lexical block what is not understood by clang (it emits them at the
compile unit level). This CL changes the dwarf parsing code to move
types defined inside a lexical block to the first parent what is not a
lexical block to avoid this clang limitation. This will make the
experience between clang and gcc consistent as clang moves these types
out of the lexical blocks before emitting the debug info.

[clayborg]

I am going to include Sean Callanan in on this one to see what he thinks. We might just want to fix the AST importer so that it can handle types in a BlockDecl.

[jingham]

This is legal C++:

#include <stdio.h>

int main(int argc, char **argv)
{
  if (argc > 1)
    {
      struct foo
      {
        long int value;
      };
      foo my_foo = {10};
      printf ("Foo: %ld.\n", my_foo.value);

    }
  else
    {
      struct foo
      {
        short int value;
      };
      foo my_foo = {10};
      printf ("Foo: %d.\n", my_foo.value);
    }
  return 0;
}

Isn't this patch going to cause problems with this sort of code?

[jingham]

At present clang produces correct debug info, and lldb correctly debugs this code.

[clayborg]

As Jim said, we can't move stuff around because types will conflict. Marking as request changes.

[spyffe]

How does Clang-emitted debug information express the region in which a type is declared if it has been moved to the containing Decl? I'm thinking specifically of this case:

int foo() {
  return
    []() { struct A { int a; } a1; a1.a = 3; return a1.a; }() +
    []() { struct A { int b; } a2; a2.b = 4; return a2.b; }();
}

As another point, it's not clear to me that the test is validating this. There are no type declarations in the lambda in the test.

I'm not convinced this is all working fine at the moment, though:

(lldb) br s -l 5
(lldb) br s -l 6
Breakpoint 2: 2 locations.
(lldb) r
   3   	int foo() {
   4   	          return
-> 5   	                      []() { struct A { int a; } a1; a1.a = 3; return a1.a; }() +
   6   	                          []() { struct A { int b; } a2; a2.b = 4; return a2.b; }();
   7   	}
   8
(lldb) fr var
(lldb) c
(lldb) fr var
(const (anonymous class) *) this = 0x00007fff5fbff678
(A) a1 = (a = 1606416064)
(lldb) c
(lldb) fr var
(lldb) c
(lldb) fr var
(const (anonymous class) *) this = 0x00007fff5fbff670
(A) a2 = (a = 3)

I suggest we bring @aprantl in to this discussion.

[tberghammer]

+Adrian

Thank you for all the comment. I agree that teaching the AST Importer to be able to handle types inside a DeclBlock would be the best long term solution but I am not sure about its complexity and feasibility (will take a look).

For the test Jim pasted clang ~ToT generates the following (IMO fairly bad) debug info:

 Compilation Unit @ offset 0x0:
  Length:        0xe5 (32-bit)
  Version:       4
  Abbrev Offset: 0x0
  Pointer Size:  8
<0><b>: Abbrev Number: 1 (DW_TAG_compile_unit)
   <c>   DW_AT_producer    : (indirect string, offset: 0x0): clang version 4.0.0 (http://llvm.org/git/clang.git 12dcbf43701c142e8313d322c14b53a6c2957826) (http://llvm.org/git/llvm.git 8ab193a8a5383b6a2ddca138c76cfd43bcff6a09)	
   <10>   DW_AT_language    : 4	(C++)
   <12>   DW_AT_name        : (indirect string, offset: 0xa5): foo.cpp	
   <16>   DW_AT_stmt_list   : 0x0	
   <1a>   DW_AT_comp_dir    : (indirect string, offset: 0xad): /home/tberghammer/tmp	
   <1e>   DW_AT_low_pc      : 0x400510	
   <26>   DW_AT_high_pc     : 0x75	
<1><2a>: Abbrev Number: 2 (DW_TAG_subprogram)
   <2b>   DW_AT_low_pc      : 0x400510	
   <33>   DW_AT_high_pc     : 0x75	
   <37>   DW_AT_frame_base  : 1 byte block: 56 	(DW_OP_reg6 (rbp))
   <39>   DW_AT_name        : (indirect string, offset: 0xc3): main	
   <3d>   DW_AT_decl_file   : 1	
   <3e>   DW_AT_decl_line   : 3	
   <3f>   DW_AT_type        : <0xc2>	
   <43>   DW_AT_external    : 1	
<2><43>: Abbrev Number: 3 (DW_TAG_formal_parameter)
   <44>   DW_AT_location    : 2 byte block: 91 78 	(DW_OP_fbreg: -8)
   <47>   DW_AT_name        : (indirect string, offset: 0xc8): argc	
   <4b>   DW_AT_decl_file   : 1	
   <4c>   DW_AT_decl_line   : 3	
   <4d>   DW_AT_type        : <0xc2>	
<2><51>: Abbrev Number: 3 (DW_TAG_formal_parameter)
   <52>   DW_AT_location    : 2 byte block: 91 70 	(DW_OP_fbreg: -16)
   <55>   DW_AT_name        : (indirect string, offset: 0xcd): argv	
   <59>   DW_AT_decl_file   : 1	
   <5a>   DW_AT_decl_line   : 3	
   <5b>   DW_AT_type        : <0xc9>	
<2><5f>: Abbrev Number: 4 (DW_TAG_lexical_block)
   <60>   DW_AT_low_pc      : 0x40053a	
   <68>   DW_AT_high_pc     : 0x1f	
<3><6c>: Abbrev Number: 5 (DW_TAG_variable)
   <6d>   DW_AT_location    : 2 byte block: 91 68 	(DW_OP_fbreg: -24)
   <70>   DW_AT_name        : (indirect string, offset: 0xd7): my_foo	
   <74>   DW_AT_decl_file   : 1	
   <75>   DW_AT_decl_line   : 11	
   <76>   DW_AT_type        : <0x97>	
<3><7a>: Abbrev Number: 0
<2><7b>: Abbrev Number: 4 (DW_TAG_lexical_block)
   <7c>   DW_AT_low_pc      : 0x400563	
   <84>   DW_AT_high_pc     : 0x1a	
<3><88>: Abbrev Number: 5 (DW_TAG_variable)
   <89>   DW_AT_location    : 2 byte block: 91 60 	(DW_OP_fbreg: -32)
   <8c>   DW_AT_name        : (indirect string, offset: 0xd7): my_foo	
   <90>   DW_AT_decl_file   : 1	
   <91>   DW_AT_decl_line   : 20	
   <92>   DW_AT_type        : <0xac>	
<3><96>: Abbrev Number: 0
<2><97>: Abbrev Number: 6 (DW_TAG_structure_type)
   <98>   DW_AT_name        : (indirect string, offset: 0xda): foo	
   <9c>   DW_AT_byte_size   : 8	
   <9d>   DW_AT_decl_file   : 1	
   <9e>   DW_AT_decl_line   : 7	
<3><9f>: Abbrev Number: 7 (DW_TAG_member)
   <a0>   DW_AT_name        : (indirect string, offset: 0xde): value	
   <a4>   DW_AT_type        : <0xda>	
   <a8>   DW_AT_decl_file   : 1	
   <a9>   DW_AT_decl_line   : 9	
   <aa>   DW_AT_data_member_location: 0	
<3><ab>: Abbrev Number: 0
<2><ac>: Abbrev Number: 6 (DW_TAG_structure_type)
   <ad>   DW_AT_name        : (indirect string, offset: 0xda): foo	
   <b1>   DW_AT_byte_size   : 2	
   <b2>   DW_AT_decl_file   : 1	
   <b3>   DW_AT_decl_line   : 16	
<3><b4>: Abbrev Number: 7 (DW_TAG_member)
   <b5>   DW_AT_name        : (indirect string, offset: 0xde): value	
   <b9>   DW_AT_type        : <0xe1>	
   <bd>   DW_AT_decl_file   : 1	
   <be>   DW_AT_decl_line   : 18	
   <bf>   DW_AT_data_member_location: 0	
<3><c0>: Abbrev Number: 0
<2><c1>: Abbrev Number: 0
<1><c2>: Abbrev Number: 8 (DW_TAG_base_type)
   <c3>   DW_AT_name        : (indirect string, offset: 0xe9): int	
   <c7>   DW_AT_encoding    : 5	(signed)
   <c8>   DW_AT_byte_size   : 4	
<1><c9>: Abbrev Number: 9 (DW_TAG_pointer_type)
   <ca>   DW_AT_type        : <0xce>	
<1><ce>: Abbrev Number: 9 (DW_TAG_pointer_type)
   <cf>   DW_AT_type        : <0xd3>	
<1><d3>: Abbrev Number: 8 (DW_TAG_base_type)
   <d4>   DW_AT_name        : (indirect string, offset: 0xd2): char	
   <d8>   DW_AT_encoding    : 6	(signed char)
   <d9>   DW_AT_byte_size   : 1	
<1><da>: Abbrev Number: 8 (DW_TAG_base_type)
   <db>   DW_AT_name        : (indirect string, offset: 0xe4): long int	
   <df>   DW_AT_encoding    : 5	(signed)
   <e0>   DW_AT_byte_size   : 8	
<1><e1>: Abbrev Number: 8 (DW_TAG_base_type)
   <e2>   DW_AT_name        : (indirect string, offset: 0xed): short	
   <e6>   DW_AT_encoding    : 5	(signed)
   <e7>   DW_AT_byte_size   : 2	
<1><e8>: Abbrev Number: 0

Clang emits the type information for both foo type into the DW_TAG_subprogram without giving any (explicit) clue about the actual scope for these types. The reason LLDB can still debug this code is that it will import the local variable and then look up the type for that specific variable. My change won't effect this behaviour in any way.

If we compile the same code with gcc 4.9 then it will emit the debug info for the types inside a DW_TAG_lexical_block (as expected) and that will cause the problem I am trying to fix here. The goal of my fix is to take the debug info emitted by gcc and make it look like the one emitted by clang.

For the example Sean posted (after adding some white space) I get the following debug info when compiling with ~ToT clang:

 Compilation Unit @ offset 0x0:
  Length:        0x12a (32-bit)
  Version:       4
  Abbrev Offset: 0x0
  Pointer Size:  8
<0><b>: Abbrev Number: 1 (DW_TAG_compile_unit)
   <c>   DW_AT_producer    : (indirect string, offset: 0x0): clang version 4.0.0 (http://llvm.org/git/clang.git 12dcbf43701c142e8313d322c14b53a6c2957826) (http://llvm.org/git/llvm.git 8ab193a8a5383b6a2ddca138c76cfd43bcff6a09)	
   <10>   DW_AT_language    : 4	(C++)
   <12>   DW_AT_name        : (indirect string, offset: 0xa5): a.cpp	
   <16>   DW_AT_stmt_list   : 0x0	
   <1a>   DW_AT_comp_dir    : (indirect string, offset: 0xab): /home/tberghammer/tmp	
   <1e>   DW_AT_low_pc      : 0x4004d0	
   <26>   DW_AT_high_pc     : 0x74	
<1><2a>: Abbrev Number: 2 (DW_TAG_subprogram)
   <2b>   DW_AT_low_pc      : 0x4004d0	
   <33>   DW_AT_high_pc     : 0x31	
   <37>   DW_AT_frame_base  : 1 byte block: 56 	(DW_OP_reg6 (rbp))
   <39>   DW_AT_name        : (indirect string, offset: 0xd0): main	
   <3d>   DW_AT_decl_file   : 1	
   <3e>   DW_AT_decl_line   : 1	
   <3f>   DW_AT_type        : <0x72>	
   <43>   DW_AT_external    : 1	
<2><43>: Abbrev Number: 3 (DW_TAG_class_type)
   <44>   DW_AT_byte_size   : 1	
   <45>   DW_AT_decl_file   : 1	
   <46>   DW_AT_decl_line   : 2	
<3><47>: Abbrev Number: 4 (DW_TAG_subprogram)
   <48>   DW_AT_name        : (indirect string, offset: 0xc1): operator()	
   <4c>   DW_AT_decl_file   : 1	
   <4d>   DW_AT_decl_line   : 2	
   <4e>   DW_AT_type        : <0x72>	
   <52>   DW_AT_declaration : 1	
   <52>   DW_AT_external    : 1	
   <52>   DW_AT_accessibility: 1	(public)
<4><53>: Abbrev Number: 5 (DW_TAG_formal_parameter)
   <54>   DW_AT_type        : <0x79>	
   <58>   DW_AT_artificial  : 1	
<4><58>: Abbrev Number: 0
<3><59>: Abbrev Number: 0
<2><5a>: Abbrev Number: 3 (DW_TAG_class_type)
   <5b>   DW_AT_byte_size   : 1	
   <5c>   DW_AT_decl_file   : 1	
   <5d>   DW_AT_decl_line   : 9	
<3><5e>: Abbrev Number: 4 (DW_TAG_subprogram)
   <5f>   DW_AT_name        : (indirect string, offset: 0xc1): operator()	
   <63>   DW_AT_decl_file   : 1	
   <64>   DW_AT_decl_line   : 9	
   <65>   DW_AT_type        : <0x72>	
   <69>   DW_AT_declaration : 1	
   <69>   DW_AT_external    : 1	
   <69>   DW_AT_accessibility: 1	(public)
<4><6a>: Abbrev Number: 5 (DW_TAG_formal_parameter)
   <6b>   DW_AT_type        : <0xce>	
   <6f>   DW_AT_artificial  : 1	
<4><6f>: Abbrev Number: 0
<3><70>: Abbrev Number: 0
<2><71>: Abbrev Number: 0
<1><72>: Abbrev Number: 6 (DW_TAG_base_type)
   <73>   DW_AT_name        : (indirect string, offset: 0xcc): int	
   <77>   DW_AT_encoding    : 5	(signed)
   <78>   DW_AT_byte_size   : 4	
<1><79>: Abbrev Number: 7 (DW_TAG_pointer_type)
   <7a>   DW_AT_type        : <0x7e>	
<1><7e>: Abbrev Number: 8 (DW_TAG_const_type)
   <7f>   DW_AT_type        : <0x43>	
<1><83>: Abbrev Number: 9 (DW_TAG_subprogram)
   <84>   DW_AT_low_pc      : 0x400510	
   <8c>   DW_AT_high_pc     : 0x14	
   <90>   DW_AT_frame_base  : 1 byte block: 56 	(DW_OP_reg6 (rbp))
   <92>   DW_AT_object_pointer: <0x9e>	
   <96>   DW_AT_linkage_name: (indirect string, offset: 0xd5): _ZZ4mainENK3$_0clEv	
   <9a>   DW_AT_specification: <0x47>	
<2><9e>: Abbrev Number: 10 (DW_TAG_formal_parameter)
   <9f>   DW_AT_location    : 2 byte block: 91 78 	(DW_OP_fbreg: -8)
   <a2>   DW_AT_name        : (indirect string, offset: 0xfd): this	
   <a6>   DW_AT_type        : <0x123>	
   <aa>   DW_AT_artificial  : 1	
<2><aa>: Abbrev Number: 11 (DW_TAG_variable)
   <ab>   DW_AT_location    : 2 byte block: 91 70 	(DW_OP_fbreg: -16)
   <ae>   DW_AT_name        : (indirect string, offset: 0x102): a1	
   <b2>   DW_AT_decl_file   : 1	
   <b3>   DW_AT_decl_line   : 5	
   <b4>   DW_AT_type        : <0xb8>	
<2><b8>: Abbrev Number: 12 (DW_TAG_structure_type)
   <b9>   DW_AT_name        : (indirect string, offset: 0x107): A	
   <bd>   DW_AT_byte_size   : 4	
   <be>   DW_AT_decl_file   : 1	
   <bf>   DW_AT_decl_line   : 3	
<3><c0>: Abbrev Number: 13 (DW_TAG_member)
   <c1>   DW_AT_name        : (indirect string, offset: 0x105): a	
   <c5>   DW_AT_type        : <0x72>	
   <c9>   DW_AT_decl_file   : 1	
   <ca>   DW_AT_decl_line   : 4	
   <cb>   DW_AT_data_member_location: 0	
<3><cc>: Abbrev Number: 0
<2><cd>: Abbrev Number: 0
<1><ce>: Abbrev Number: 7 (DW_TAG_pointer_type)
   <cf>   DW_AT_type        : <0xd3>	
<1><d3>: Abbrev Number: 8 (DW_TAG_const_type)
   <d4>   DW_AT_type        : <0x5a>	
<1><d8>: Abbrev Number: 9 (DW_TAG_subprogram)
   <d9>   DW_AT_low_pc      : 0x400530	
   <e1>   DW_AT_high_pc     : 0x14	
   <e5>   DW_AT_frame_base  : 1 byte block: 56 	(DW_OP_reg6 (rbp))
   <e7>   DW_AT_object_pointer: <0xf3>	
   <eb>   DW_AT_linkage_name: (indirect string, offset: 0xe9): _ZZ4mainENK3$_1clEv	
   <ef>   DW_AT_specification: <0x5e>	
<2><f3>: Abbrev Number: 10 (DW_TAG_formal_parameter)
   <f4>   DW_AT_location    : 2 byte block: 91 78 	(DW_OP_fbreg: -8)
   <f7>   DW_AT_name        : (indirect string, offset: 0xfd): this	
   <fb>   DW_AT_type        : <0x128>	
   <ff>   DW_AT_artificial  : 1	
<2><ff>: Abbrev Number: 11 (DW_TAG_variable)
   <100>   DW_AT_location    : 2 byte block: 91 70 	(DW_OP_fbreg: -16)
   <103>   DW_AT_name        : (indirect string, offset: 0x109): a2	
   <107>   DW_AT_decl_file   : 1	
   <108>   DW_AT_decl_line   : 12	
   <109>   DW_AT_type        : <0x10d>	
<2><10d>: Abbrev Number: 12 (DW_TAG_structure_type)
   <10e>   DW_AT_name        : (indirect string, offset: 0x107): A	
   <112>   DW_AT_byte_size   : 4	
   <113>   DW_AT_decl_file   : 1	
   <114>   DW_AT_decl_line   : 10	
<3><115>: Abbrev Number: 13 (DW_TAG_member)
   <116>   DW_AT_name        : (indirect string, offset: 0x10c): b	
   <11a>   DW_AT_type        : <0x72>	
   <11e>   DW_AT_decl_file   : 1	
   <11f>   DW_AT_decl_line   : 11	
   <120>   DW_AT_data_member_location: 0	
<3><121>: Abbrev Number: 0
<2><122>: Abbrev Number: 0
<1><123>: Abbrev Number: 7 (DW_TAG_pointer_type)
   <124>   DW_AT_type        : <0x7e>	
<1><128>: Abbrev Number: 7 (DW_TAG_pointer_type)
   <129>   DW_AT_type        : <0xd3>	
<1><12d>: Abbrev Number: 0

It doesn't contain any DW_TAG_lexical_block so this change won't have any effect on the behaviour. I tested it with LLDB both before this change and after this change and it seems to work as expected for me (seeing the frame variables and can evaluate expressions).

For the test I opted to try to run an expression while we are stopped inside a lambda expression because that is the original problem I am trying to solve and I think having a test like this is a good idea anyway. It effects the code modified by this CL because both gcc and clang generated an artificial for every lambda created to hold the function call operator and the captured variables. If people think that we should have a test for the case Jim posted I can write a separate test for that.

[tberghammer]

Based on the feedback will try to fix the Clang AS importer instead (don't have any ETA for it)

packages/Python/lldbsuite/test/expression_command/lambda/Makefile

+LEVEL = ../../make
+
+CXX_SOURCES := main.cpp
+
+include $(LEVEL)/Makefile.rules

packages/Python/lldbsuite/test/expression_command/lambda/TestLambda.py

+from __future__ import print_function
+
+import lldb
+from lldbsuite.test.decorators import *
+from lldbsuite.test.lldbtest import *
+from lldbsuite.test import lldbutil
+
+
+class LambdaTestCase(TestBase):
+
+    mydir = TestBase.compute_mydir(__file__)
+
+    def setUp(self):
+        # Call super's setUp().
+        TestBase.setUp(self)
+
+        self.main_source = "main.cpp"
+        self.main_source_spec = lldb.SBFileSpec(self.main_source)
+        self.exe = os.path.join(os.getcwd(), "a.out")
+
+    def test_expression_inside_lambda(self):
+        self.build()
+
+        target = self.dbg.CreateTarget(self.exe)
+        self.assertTrue(target)
+
+        breakpoint = target.BreakpointCreateBySourceRegex(
+            '// Break here 1', self.main_source_spec)
+        self.assertTrue(breakpoint)
+
+        # Launch the process, and do not stop at the entry point.
+        process = target.LaunchSimple(
+            None, None, self.get_process_working_directory())
+        self.assertTrue(process)
+
+        threads = lldbutil.get_threads_stopped_at_breakpoint(
+            process, breakpoint)
+        self.assertEqual(len(threads), 1)
+
+        frame = threads[0].GetFrameAtIndex(0)
+
+        value = frame.EvaluateExpression("a + b")
+        self.assertTrue(value.IsValid())
+        self.assertTrue(value.GetError().Success())
+        self.assertEqual(value.GetValueAsSigned(0), 579)
+
+    @expectedFailureAll(compiler="gcc")
+    def test_expression_call_lambda(self):
+        self.build()
+
+        target = self.dbg.CreateTarget(self.exe)
+        self.assertTrue(target)
+
+        breakpoint = target.BreakpointCreateBySourceRegex(
+            '// Break here 2', self.main_source_spec)
+        self.assertTrue(breakpoint)
+
+        # Launch the process, and do not stop at the entry point.
+        process = target.LaunchSimple(
+            None, None, self.get_process_working_directory())
+        self.assertTrue(process)
+
+        threads = lldbutil.get_threads_stopped_at_breakpoint(
+            process, breakpoint)
+        self.assertEqual(len(threads), 1)
+
+        frame = threads[0].GetFrameAtIndex(0)
+
+        value = frame.EvaluateExpression("l(321, 654)")
+        self.assertTrue(value.IsValid())
+        self.assertTrue(value.GetError().Success())
+        self.assertEqual(value.GetValueAsSigned(0), 975)
+

packages/Python/lldbsuite/test/expression_command/lambda/main.cpp

+int main() {
+  auto l = [](int a, int b) {
+    return a + b; // Break here 1
+  };
+  return l(123, 456); // Break here 2
+}

source/Plugins/SymbolFile/DWARF/DWARFASTParserClang.cpp

DWARFASTParserClang::ResolveNamespaceDIE(const DWARFDIE &die) {
   }
   return nullptr;
 }
 
 clang::DeclContext *DWARFASTParserClang::GetClangDeclContextContainingDIE(
     const DWARFDIE &die, DWARFDIE *decl_ctx_die_copy) {
   SymbolFileDWARF *dwarf = die.GetDWARF();
 
+  bool is_type = false;
+  switch (die.Tag()) {
+    case DW_TAG_base_type:
+    case DW_TAG_class_type:
+    case DW_TAG_const_type:
+    case DW_TAG_pointer_type:
+    case DW_TAG_reference_type:
+    case DW_TAG_restrict_type:
+    case DW_TAG_rvalue_reference_type:
+    case DW_TAG_structure_type:
+    case DW_TAG_typedef:
+    case DW_TAG_union_type:
+    case DW_TAG_unspecified_type:
+    case DW_TAG_volatile_type:
+      is_type = true;
+      break;
+    default:
+      break;
+  }
+
   DWARFDIE decl_ctx_die = dwarf->GetDeclContextDIEContainingDIE(die);
+  if (is_type) {
+    // The clang AST importer can't handle types declared inside a BlockDecl.
+    // Move these declarartions into the parent context of the block. This will
+    // make the behavior from gcc (what emits type information inside lexical
+    // blocks) consistent with the way clang emits debug info for types defined
+    // inside a function (class, namespace or compile unit level).
+    while (decl_ctx_die.Tag() == DW_TAG_lexical_block) {
+      decl_ctx_die = dwarf->GetDeclContextDIEContainingDIE(decl_ctx_die);
+    }
+  }
 
   if (decl_ctx_die_copy)
     *decl_ctx_die_copy = decl_ctx_die;
 
   if (decl_ctx_die) {
     clang::DeclContext *clang_decl_ctx =
         GetClangDeclContextForDIE(decl_ctx_die);
     if (clang_decl_ctx)