lldb: Add support for printing variables with DW_AT_ranges on DW_TAG_subprograms

[dblaikie]

Summary

Finishing out the support (to the best of my knowledge/based on current
testing running the whole check-lldb with a clang forcibly using
DW_AT_ranges on all DW_TAG_subprograms) for this feature.

[labath]

I don't think that simply setting func_lo_pc to zero will be sufficient to make this work. I would expect this to break in more complicated scenarios (like, even if we just have two of these functions). I think the only reason it works in this case is because for this particular function, it's base address (relative to the CU base) *is* zero.

The purpose of func_lo_pc is pretty weird, but it's essentially used for runtime relocation of location lists within the function -- we take the static "base" of the function, and the runtime "base", and the difference between the two produces the load bias. Applying the same bias to all variable location lists "relocates" the variables. (The reason this is so convoluted is reading location lists straight from (unrelocated) .o files on mac. I seriously considered changing this when working on debug_rnglists, but it turned out it wasn't really necessary, so I let it go.)

The "runtime" function address is being taken in https://github.com/llvm/llvm-project/blob/main/lldb/source/Core/ValueObjectVariable.cpp#L159 (and maybe a couple of other places), and these two things need to be in sync. I think this could just be defined as the first address in the first address range of the function. That should be fine for ELF purposes (I have a feeling this thing would completely break down if macho started rearranging parts of the function), but it does bring up another problem.

LLDB's representation of a function (lldb_private::Function) assumes that all functions will be contiguous (Function::GetAddressRange returns a single range). If we make it so that this range matches the first block of the function (maybe that's what happens now), then maybe we could make things work for the case where DW_AT_ranges are used as a size optimization. However, getting things to work for truly discontinuous functions will require more work...

(Regarding the test, I think we could avoid running a binary by testing this via the "image lookup --verbose" command, and checking that the output contains the variables it should.)

[dblaikie]

Use image lookup to make test runnable without executing the test code

[dblaikie]

In D94064#2481925, @labath wrote:

I don't think that simply setting func_lo_pc to zero will be sufficient to make this work. I would expect this to break in more complicated scenarios (like, even if we just have two of these functions). I think the only reason it works in this case is because for this particular function, it's base address (relative to the CU base) *is* zero.

I certainly don't have high confidence in the change, to be sure - but I think it's a bit more robust than that.

Here's at least one experiment where a function is at a non-zero offset relative to the CU:

$ cat test.cpp
void stop() {
}
void f1() {
  int i = 7;
  stop();
}
int main() {
  int j = 12;
  f1();
  stop();
}
$ $ clang++-tot test.cpp -gdwarf-5 -mllvm -always-use-ranges-in-v5=Enable && ~/dev/llvm/build/default/bin/lldb ./a.out
(lldb) target create "./a.out"
Current executable set to '/usr/local/google/home/blaikie/dev/scratch/a.out' (x86_64).
(lldb) r
Process 1039251 launched: '/usr/local/google/home/blaikie/dev/scratch/a.out' (x86_64)
Process 1039251 exited with status = 0 (0x00000000) 
(lldb) b stop
Breakpoint 1: where = a.out`stop() + 4 at test.cpp:2:1, address = 0x0000000000401114
(lldb) r
Process 1039605 launched: '/usr/local/google/home/blaikie/dev/scratch/a.out' (x86_64)
Process 1039605 stopped
* thread #1, name = 'a.out', stop reason = breakpoint 1.1
    frame #0: 0x0000000000401114 a.out`stop() at test.cpp:2:1
   1    void stop() {
-> 2    }
   3    void f1() {
   4      int i = 7;
   5      stop();
   6    }
   7    int main() {
(lldb) up
frame #1: 0x0000000000401134 a.out`f1() at test.cpp:5:3
   2    }
   3    void f1() {
   4      int i = 7;
-> 5      stop();
   6    }
   7    int main() {
   8      int j = 12;
(lldb) p i
(int) $0 = 7
(lldb) c
Process 1039605 resuming
Process 1039605 stopped
* thread #1, name = 'a.out', stop reason = breakpoint 1.1
    frame #0: 0x0000000000401114 a.out`stop() at test.cpp:2:1
   1    void stop() {
-> 2    }
   3    void f1() {
   4      int i = 7;
   5      stop();
   6    }
   7    int main() {
(lldb) up
frame #1: 0x0000000000401159 a.out`main at test.cpp:10:3
   7    int main() {
   8      int j = 12;
   9      f1();
-> 10     stop();
   11   }
(lldb) p j
(int) $1 = 12
$ llvm-dwarfdump a.out
.debug_info contents:
0x00000000: Compile Unit: length = 0x0000005f, format = DWARF32, version = 0x0005, unit_type = DW_UT_compile, abbr_offset = 0x0000, addr_size = 0x08 (next unit at 0x00000063)

0x0000000c: DW_TAG_compile_unit
              DW_AT_producer    ("clang version 12.0.0 ([e-mail]:llvm/llvm-project.git 13740c7d80e6c7b47506fd34cd2ea2a7b658cdfa)")
              DW_AT_language    (DW_LANG_C_plus_plus_14)
              DW_AT_name        ("test.cpp")
              DW_AT_str_offsets_base    (0x00000008)
              DW_AT_stmt_list   (0x00000000)
              DW_AT_comp_dir    ("/usr/local/google/home/blaikie/dev/scratch")
              DW_AT_low_pc      (0x0000000000401110)
              DW_AT_high_pc     (0x0000000000401161)
              DW_AT_addr_base   (0x00000008)
              DW_AT_rnglists_base       (0x0000000c)

0x00000027:   DW_TAG_subprogram
                DW_AT_low_pc    (0x0000000000401110)
                DW_AT_high_pc   (0x0000000000401116)
                DW_AT_frame_base        (DW_OP_reg6 RBP)
                DW_AT_linkage_name      ("_Z4stopv")
                DW_AT_name      ("stop")
                DW_AT_decl_file ("/usr/local/google/home/blaikie/dev/scratch/test.cpp")
                DW_AT_decl_line (1)
                DW_AT_external  (true)

0x00000033:   DW_TAG_subprogram
                DW_AT_ranges    (indexed (0x0) rangelist = 0x00000014
                   [0x0000000000401120, 0x000000000040113a))
                DW_AT_frame_base        (DW_OP_reg6 RBP)
                DW_AT_linkage_name      ("_Z2f1v")
                DW_AT_name      ("f1")
                DW_AT_decl_file ("/usr/local/google/home/blaikie/dev/scratch/test.cpp")
                DW_AT_decl_line (3)
                DW_AT_external  (true)

0x0000003b:     DW_TAG_variable
                  DW_AT_location        (DW_OP_fbreg -4)
                  DW_AT_name    ("i")
                  DW_AT_decl_file       ("/usr/local/google/home/blaikie/dev/scratch/test.cpp")
                  DW_AT_decl_line       (4)
                  DW_AT_type    (0x0000005e "int")

0x00000046:     NULL

0x00000047:   DW_TAG_subprogram
                DW_AT_ranges    (indexed (0x1) rangelist = 0x00000018
                   [0x0000000000401140, 0x0000000000401161))
                DW_AT_frame_base        (DW_OP_reg6 RBP)
                DW_AT_name      ("main")
                DW_AT_decl_file ("/usr/local/google/home/blaikie/dev/scratch/test.cpp")
                DW_AT_decl_line (7)
                DW_AT_type      (0x0000005e "int")
                DW_AT_external  (true)

0x00000052:     DW_TAG_variable
                  DW_AT_location        (DW_OP_fbreg -4)
                  DW_AT_name    ("j")
                  DW_AT_decl_file       ("/usr/local/google/home/blaikie/dev/scratch/test.cpp")
                  DW_AT_decl_line       (8)
                  DW_AT_type    (0x0000005e "int")

0x0000005d:     NULL

0x0000005e:   DW_TAG_base_type
                DW_AT_name      ("int")
                DW_AT_encoding  (DW_ATE_signed)
                DW_AT_byte_size (0x04)

0x00000062:   NULL

The purpose of func_lo_pc is pretty weird, but it's essentially used for runtime relocation of location lists within the function -- we take the static "base" of the function, and the runtime "base", and the difference between the two produces the load bias. Applying the same bias to all variable location lists "relocates" the variables. (The reason this is so convoluted is reading location lists straight from (unrelocated) .o files on mac. I seriously considered changing this when working on debug_rnglists, but it turned out it wasn't really necessary, so I let it go.)

Yep, I figured a bunch of this was for DWARF in unrelocated MachO files - and that they wouldn't be able to/want to use Split DWARF or this feature (which is particularly relevant to Split DWARF). Does any of this logic apply outside that unrelocated MachO scenario?

The "runtime" function address is being taken in https://github.com/llvm/llvm-project/blob/main/lldb/source/Core/ValueObjectVariable.cpp#L159 (and maybe a couple of other places), and these two things need to be in sync. I think this could just be defined as the first address in the first address range of the function. That should be fine for ELF purposes (I have a feeling this thing would completely break down if macho started rearranging parts of the function), but it does bring up another problem.

LLDB's representation of a function (lldb_private::Function) assumes that all functions will be contiguous (Function::GetAddressRange returns a single range). If we make it so that this range matches the first block of the function (maybe that's what happens now),

Actually seems to adjust for discontiguous ranges and not just pick the start of the first (as it appears in the DWARF - which isn't guaranteed to be ordered in any way) range, instead finding the smallest and largest addresses: https://github.com/llvm-mirror/lldb/blob/master/source/Plugins/SymbolFile/DWARF/DWARFASTParserClang.cpp#L2357

then maybe we could make things work for the case where DW_AT_ranges are used as a size optimization. However, getting things to work for truly discontinuous functions will require more work...

(Regarding the test, I think we could avoid running a binary by testing this via the "image lookup --verbose" command, and checking that the output contains the variables it should.)

Ah, sure thing - I've updated it that way.

[labath]

In D94064#2483578, @dblaikie wrote:

In D94064#2481925, @labath wrote:

I don't think that simply setting func_lo_pc to zero will be sufficient to make this work. I would expect this to break in more complicated scenarios (like, even if we just have two of these functions). I think the only reason it works in this case is because for this particular function, it's base address (relative to the CU base) *is* zero.

I certainly don't have high confidence in the change, to be sure - but I think it's a bit more robust than that.

Yes, it seems it is, though just by a bit. :)

The thing which is missing is the location list on the variable DIE -- without it, it does not matter which address gets put here (so long as it does not trigger the assertion), as the value is totally unused. The necessity of the location list is kind of obvious from my description of the problem, though even I did not realize when I was writing it that this requires a more elaborate test case.

Adjusting your test case to produce a location list for i I got this (probably not minimal) snippet (add -O1 to CFLAGS):

#define optnone __attribute__((optnone))
#define noinline __attribute__((noinline))

void optnone noinline stop() {}
void optnone noinline consume(int i) {}
void noinline f1(int x) {
  int i = x;
  stop();
  consume(i);
  i = 5;
  consume(i);
}
int main() {
  int j = 12;
  f1(j);
  stop();
}

Here lldb fails to print the value of i, but if I rearrange the code so that the f1 functions comes first, the value is printed correctly:

#define optnone __attribute__((optnone))
#define noinline __attribute__((noinline))

void optnone noinline stop();
void optnone noinline consume(int i);
void noinline f1(int x) {
  int i = x;
  stop();
  consume(i);
  i = 5;
  consume(i);
}
void optnone noinline stop() {}
void optnone noinline consume(int i) {}
int main() {
  int j = 12;
  f1(j);
  stop();
}

The purpose of func_lo_pc is pretty weird, but it's essentially used for runtime relocation of location lists within the function -- we take the static "base" of the function, and the runtime "base", and the difference between the two produces the load bias. Applying the same bias to all variable location lists "relocates" the variables. (The reason this is so convoluted is reading location lists straight from (unrelocated) .o files on mac. I seriously considered changing this when working on debug_rnglists, but it turned out it wasn't really necessary, so I let it go.)

Yep, I figured a bunch of this was for DWARF in unrelocated MachO files - and that they wouldn't be able to/want to use Split DWARF or this feature (which is particularly relevant to Split DWARF). Does any of this logic apply outside that unrelocated MachO scenario?

Yes, this code is used everywhere, though it's role is less important (and one that could be implemented in an easier manner, were it not for MachO) -- it adjusts the variable address for ASLR, i.e., the variable/function being loaded at a different address than what the static debug info says.

LLDB's representation of a function (lldb_private::Function) assumes that all functions will be contiguous (Function::GetAddressRange returns a single range). If we make it so that this range matches the first block of the function (maybe that's what happens now),

Actually seems to adjust for discontiguous ranges and not just pick the start of the first (as it appears in the DWARF - which isn't guaranteed to be ordered in any way) range, instead finding the smallest and largest addresses: https://github.com/llvm-mirror/lldb/blob/master/source/Plugins/SymbolFile/DWARF/DWARFASTParserClang.cpp#L2357

Ah.. interesting. In that case, I believe things should work if we use the same algorithm to compute func_lo_pc. We may still run into weird issues for functions which are truly discontinuous, but it should be ok for single-range range lists, at least.

[dblaikie]

Retrieve the lowest address in the address ranges, rather than just hardcoding 0

[dblaikie]

In D94064#2485222, @labath wrote:

In D94064#2483578, @dblaikie wrote:

In D94064#2481925, @labath wrote:

I don't think that simply setting func_lo_pc to zero will be sufficient to make this work. I would expect this to break in more complicated scenarios (like, even if we just have two of these functions). I think the only reason it works in this case is because for this particular function, it's base address (relative to the CU base) *is* zero.

I certainly don't have high confidence in the change, to be sure - but I think it's a bit more robust than that.

Yes, it seems it is, though just by a bit. :)

The thing which is missing is the location list on the variable DIE -- without it, it does not matter which address gets put here (so long as it does not trigger the assertion), as the value is totally unused. The necessity of the location list is kind of obvious from my description of the problem, though even I did not realize when I was writing it that this requires a more elaborate test case.

Adjusting your test case to produce a location list for i I got this (probably not minimal) snippet (add -O1 to CFLAGS):

#define optnone __attribute__((optnone))
#define noinline __attribute__((noinline))

void optnone noinline stop() {}
void optnone noinline consume(int i) {}
void noinline f1(int x) {
  int i = x;
  stop();
  consume(i);
  i = 5;
  consume(i);
}
int main() {
  int j = 12;
  f1(j);
  stop();
}

Here lldb fails to print the value of i, but if I rearrange the code so that the f1 functions comes first, the value is printed correctly:

#define optnone __attribute__((optnone))
#define noinline __attribute__((noinline))

void optnone noinline stop();
void optnone noinline consume(int i);
void noinline f1(int x) {
  int i = x;
  stop();
  consume(i);
  i = 5;
  consume(i);
}
void optnone noinline stop() {}
void optnone noinline consume(int i) {}
int main() {
  int j = 12;
  f1(j);
  stop();
}

Ah, thanks - think I figured out a representative test & understand better.
I ended up with this:

__attribute__((nodebug)) volatile int i;
int main() {
  int var = 3;
  i = 1;
  var = 5;
  i = 2;
}

By using volatile writes, I was able to get the 'var' variable live range to start at the start of the function (so that image lookup -v -s main would render the "var" variable since it's now live at the very start of the function (rather than only after the push instruction)). And use nodebug to reduce the DWARF since the 'i' isn't interesting.

Indeed without the fix you suggested to use "lowest address" rather than zero, this test above would/was failing (running the image lookup command would not show the "var" variable).

Also, I figured out how to run the API tests, and that showed a bunch more failures when using ranges-everywhere (actually a more aggressive version of ranges-everywhere - using it no matter the DWARF version and even when it wouldn't reduce the address pool size (eg: using ranges on a subprogram even when low_pc is an address already in the pool (such as for the start of a CU range))) - and with the change to use the lowest address of the ranges, all those failures now go away.

So *fingers crossed* this is ready.

The purpose of func_lo_pc is pretty weird, but it's essentially used for runtime relocation of location lists within the function -- we take the static "base" of the function, and the runtime "base", and the difference between the two produces the load bias. Applying the same bias to all variable location lists "relocates" the variables. (The reason this is so convoluted is reading location lists straight from (unrelocated) .o files on mac. I seriously considered changing this when working on debug_rnglists, but it turned out it wasn't really necessary, so I let it go.)

Yep, I figured a bunch of this was for DWARF in unrelocated MachO files - and that they wouldn't be able to/want to use Split DWARF or this feature (which is particularly relevant to Split DWARF). Does any of this logic apply outside that unrelocated MachO scenario?

Yes, this code is used everywhere, though it's role is less important (and one that could be implemented in an easier manner, were it not for MachO) -- it adjusts the variable address for ASLR, i.e., the variable/function being loaded at a different address than what the static debug info says.

Ah, good to know!

LLDB's representation of a function (lldb_private::Function) assumes that all functions will be contiguous (Function::GetAddressRange returns a single range). If we make it so that this range matches the first block of the function (maybe that's what happens now),

Actually seems to adjust for discontiguous ranges and not just pick the start of the first (as it appears in the DWARF - which isn't guaranteed to be ordered in any way) range, instead finding the smallest and largest addresses: https://github.com/llvm-mirror/lldb/blob/master/source/Plugins/SymbolFile/DWARF/DWARFASTParserClang.cpp#L2357

Ah.. interesting. In that case, I believe things should work if we use the same algorithm to compute func_lo_pc. We may still run into weird issues for functions which are truly discontinuous, but it should be ok for single-range range lists, at least.

Fair.

[dblaikie]

Include some more details about/in the test

[labath]

In D94064#2516883, @dblaikie wrote:

Ah, thanks - think I figured out a representative test & understand better.
I ended up with this:

__attribute__((nodebug)) volatile int i;
int main() {
  int var = 3;
  i = 1;
  var = 5;
  i = 2;
}

By using volatile writes, I was able to get the 'var' variable live range to start at the start of the function (so that image lookup -v -s main would render the "var" variable since it's now live at the very start of the function (rather than only after the push instruction)). And use nodebug to reduce the DWARF since the 'i' isn't interesting.

Indeed without the fix you suggested to use "lowest address" rather than zero, this test above would/was failing (running the image lookup command would not show the "var" variable).

Also, I figured out how to run the API tests, and that showed a bunch more failures when using ranges-everywhere (actually a more aggressive version of ranges-everywhere - using it no matter the DWARF version and even when it wouldn't reduce the address pool size (eg: using ranges on a subprogram even when low_pc is an address already in the pool (such as for the start of a CU range))) - and with the change to use the lowest address of the ranges, all those failures now go away.

So *fingers crossed* this is ready.

/me too

lldb/test/Shell/SymbolFile/DWARF/subprogram_ranges.test

20

btw, the way I've dealt with this in the past is to introduce an additional label in the assembly (look_me_up:), and then give that to the image lookup command.

[dblaikie]

lldb/test/Shell/SymbolFile/DWARF/subprogram_ranges.test

20

Oh, makes sense - will certainly keep that in mind!

lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.cpp

 }
 
 size_t SymbolFileDWARF::ParseVariablesForContext(const SymbolContext &sc) {
   std::lock_guard<std::recursive_mutex> guard(GetModuleMutex());
   if (sc.comp_unit != nullptr) {
     if (sc.function) {
       DWARFDIE function_die = GetDIE(sc.function->GetID());
 
-      const dw_addr_t func_lo_pc = function_die.GetAttributeValueAsAddress(
+      dw_addr_t func_lo_pc = function_die.GetAttributeValueAsAddress(
           DW_AT_low_pc, LLDB_INVALID_ADDRESS);
+      DWARFFormValue form_value;
+      if (func_lo_pc == LLDB_INVALID_ADDRESS &&
+          function_die.GetDIE()->GetAttributeValue(function_die.GetCU(),
+                                                   DW_AT_ranges, form_value))
+        func_lo_pc = 0;
       if (func_lo_pc != LLDB_INVALID_ADDRESS) {
         const size_t num_variables = ParseVariables(
             sc, function_die.GetFirstChild(), func_lo_pc, true, true);
 
         // Let all blocks know they have parse all their variables
         sc.function->GetBlock(false).SetDidParseVariables(true, true);
         return num_variables;
       }

lldb/test/Shell/SymbolFile/DWARF/subprogram_ranges.test

 # UNSUPPORTED: system-windows
 # RUN: %clang_host -g -O0 %S/Inputs/subprogram_ranges.s -o %t.out
-# RUN: not %lldb -b -s %s %t.out 2>&1 | FileCheck %s
+# RUN: %lldb -b -s %s %t.out 2>&1 | FileCheck %s
 
 # Test breaking on symbols and printing variables when a DW_TAG_subprogram uses
 # DW_AT_ranges instead of DW_AT_low_pc/DW_AT_high_pc.  While the assembly here
 # is a bit unrealistic - it's a single-entry range using DWARFv4 which isn't
 # useful for anything (a single-entry range with DWARFv5 can reduce address
 # relocations, and multi-entry ranges can be used for function sections), but
 # it's the simplest thing to test. If anyone's updating this test at some
 # point, feel free to replace it with another equivalent test if it's
 # especially useful, but don't dismiss it as pointless just because it's a bit
 # weird.
 
 b main
 # CHECK: (lldb) b main
 # CHECK-NEXT: Breakpoint 1: where = subprogram_ranges.test.tmp.out`main + 6 at main.c:2:7
 
 run
 # Stopping inside of the stop hook range
 # CHECK: (lldb) run
 
 print var
 # Check that local variable names can be looked up
-# FIXME: This should be: (int) $0 = {{.*}}
 # CHECK: (lldb) print var
-# CHECK-NEXT: error: <user expression 0>:1:1: use of undeclared identifier 'var'
+# CHECK-NEXT: (int) $0 = {{.*}}
 
 q