This is an archive of the discontinued LLVM Phabricator instance.

lldb: Add support for printing variables with DW_AT_ranges on DW_TAG_subprograms
ClosedPublic

Authored by dblaikie on Jan 5 2021, 12:12 AM.

Details

Summary

Finishing out the support (to the best of my knowledge/based on current
testing running the whole check-lldb with a clang forcibly using
DW_AT_ranges on all DW_TAG_subprograms) for this feature.

Diff Detail

Event Timeline

dblaikie requested review of this revision.Jan 5 2021, 12:12 AM
dblaikie created this revision.
Herald added a project: Restricted Project. · View Herald TranscriptJan 5 2021, 12:12 AM
amharc added a subscriber: amharc.Jan 6 2021, 1:39 AM
labath added a comment.Jan 6 2021, 5:19 AM

I don't think that simply setting func_lo_pc to zero will be sufficient to make this work. I would expect this to break in more complicated scenarios (like, even if we just have two of these functions). I think the only reason it works in this case is because for this particular function, it's base address (relative to the CU base) *is* zero.

The purpose of func_lo_pc is pretty weird, but it's essentially used for runtime relocation of location lists within the function -- we take the static "base" of the function, and the runtime "base", and the difference between the two produces the load bias. Applying the same bias to all variable location lists "relocates" the variables. (The reason this is so convoluted is reading location lists straight from (unrelocated) .o files on mac. I seriously considered changing this when working on debug_rnglists, but it turned out it wasn't really necessary, so I let it go.)

The "runtime" function address is being taken in https://github.com/llvm/llvm-project/blob/main/lldb/source/Core/ValueObjectVariable.cpp#L159 (and maybe a couple of other places), and these two things need to be in sync. I think this could just be defined as the first address in the first address range of the function. That should be fine for ELF purposes (I have a feeling this thing would completely break down if macho started rearranging parts of the function), but it does bring up another problem.

LLDB's representation of a function (lldb_private::Function) assumes that all functions will be contiguous (Function::GetAddressRange returns a single range). If we make it so that this range matches the first block of the function (maybe that's what happens now), then maybe we could make things work for the case where DW_AT_ranges are used as a size optimization. However, getting things to work for truly discontinuous functions will require more work...

(Regarding the test, I think we could avoid running a binary by testing this via the "image lookup --verbose" command, and checking that the output contains the variables it should.)

dblaikie updated this revision to Diff 315036.Jan 6 2021, 6:45 PM

Use image lookup to make test runnable without executing the test code

I don't think that simply setting func_lo_pc to zero will be sufficient to make this work. I would expect this to break in more complicated scenarios (like, even if we just have two of these functions). I think the only reason it works in this case is because for this particular function, it's base address (relative to the CU base) *is* zero.

I certainly don't have high confidence in the change, to be sure - but I think it's a bit more robust than that.

Here's at least one experiment where a function is at a non-zero offset relative to the CU:

$ cat test.cpp
void stop() {
}
void f1() {
  int i = 7;
  stop();
}
int main() {
  int j = 12;
  f1();
  stop();
}
$ $ clang++-tot test.cpp -gdwarf-5 -mllvm -always-use-ranges-in-v5=Enable && ~/dev/llvm/build/default/bin/lldb ./a.out
(lldb) target create "./a.out"
Current executable set to '/usr/local/google/home/blaikie/dev/scratch/a.out' (x86_64).
(lldb) r
Process 1039251 launched: '/usr/local/google/home/blaikie/dev/scratch/a.out' (x86_64)
Process 1039251 exited with status = 0 (0x00000000) 
(lldb) b stop
Breakpoint 1: where = a.out`stop() + 4 at test.cpp:2:1, address = 0x0000000000401114
(lldb) r
Process 1039605 launched: '/usr/local/google/home/blaikie/dev/scratch/a.out' (x86_64)
Process 1039605 stopped
* thread #1, name = 'a.out', stop reason = breakpoint 1.1
    frame #0: 0x0000000000401114 a.out`stop() at test.cpp:2:1
   1    void stop() {
-> 2    }
   3    void f1() {
   4      int i = 7;
   5      stop();
   6    }
   7    int main() {
(lldb) up
frame #1: 0x0000000000401134 a.out`f1() at test.cpp:5:3
   2    }
   3    void f1() {
   4      int i = 7;
-> 5      stop();
   6    }
   7    int main() {
   8      int j = 12;
(lldb) p i
(int) $0 = 7
(lldb) c
Process 1039605 resuming
Process 1039605 stopped
* thread #1, name = 'a.out', stop reason = breakpoint 1.1
    frame #0: 0x0000000000401114 a.out`stop() at test.cpp:2:1
   1    void stop() {
-> 2    }
   3    void f1() {
   4      int i = 7;
   5      stop();
   6    }
   7    int main() {
(lldb) up
frame #1: 0x0000000000401159 a.out`main at test.cpp:10:3
   7    int main() {
   8      int j = 12;
   9      f1();
-> 10     stop();
   11   }
(lldb) p j
(int) $1 = 12
$ llvm-dwarfdump a.out
.debug_info contents:
0x00000000: Compile Unit: length = 0x0000005f, format = DWARF32, version = 0x0005, unit_type = DW_UT_compile, abbr_offset = 0x0000, addr_size = 0x08 (next unit at 0x00000063)

0x0000000c: DW_TAG_compile_unit
              DW_AT_producer    ("clang version 12.0.0 (git@github.com:llvm/llvm-project.git 13740c7d80e6c7b47506fd34cd2ea2a7b658cdfa)")
              DW_AT_language    (DW_LANG_C_plus_plus_14)
              DW_AT_name        ("test.cpp")
              DW_AT_str_offsets_base    (0x00000008)
              DW_AT_stmt_list   (0x00000000)
              DW_AT_comp_dir    ("/usr/local/google/home/blaikie/dev/scratch")
              DW_AT_low_pc      (0x0000000000401110)
              DW_AT_high_pc     (0x0000000000401161)
              DW_AT_addr_base   (0x00000008)
              DW_AT_rnglists_base       (0x0000000c)

0x00000027:   DW_TAG_subprogram
                DW_AT_low_pc    (0x0000000000401110)
                DW_AT_high_pc   (0x0000000000401116)
                DW_AT_frame_base        (DW_OP_reg6 RBP)
                DW_AT_linkage_name      ("_Z4stopv")
                DW_AT_name      ("stop")
                DW_AT_decl_file ("/usr/local/google/home/blaikie/dev/scratch/test.cpp")
                DW_AT_decl_line (1)
                DW_AT_external  (true)

0x00000033:   DW_TAG_subprogram
                DW_AT_ranges    (indexed (0x0) rangelist = 0x00000014
                   [0x0000000000401120, 0x000000000040113a))
                DW_AT_frame_base        (DW_OP_reg6 RBP)
                DW_AT_linkage_name      ("_Z2f1v")
                DW_AT_name      ("f1")
                DW_AT_decl_file ("/usr/local/google/home/blaikie/dev/scratch/test.cpp")
                DW_AT_decl_line (3)
                DW_AT_external  (true)

0x0000003b:     DW_TAG_variable
                  DW_AT_location        (DW_OP_fbreg -4)
                  DW_AT_name    ("i")
                  DW_AT_decl_file       ("/usr/local/google/home/blaikie/dev/scratch/test.cpp")
                  DW_AT_decl_line       (4)
                  DW_AT_type    (0x0000005e "int")

0x00000046:     NULL

0x00000047:   DW_TAG_subprogram
                DW_AT_ranges    (indexed (0x1) rangelist = 0x00000018
                   [0x0000000000401140, 0x0000000000401161))
                DW_AT_frame_base        (DW_OP_reg6 RBP)
                DW_AT_name      ("main")
                DW_AT_decl_file ("/usr/local/google/home/blaikie/dev/scratch/test.cpp")
                DW_AT_decl_line (7)
                DW_AT_type      (0x0000005e "int")
                DW_AT_external  (true)

0x00000052:     DW_TAG_variable
                  DW_AT_location        (DW_OP_fbreg -4)
                  DW_AT_name    ("j")
                  DW_AT_decl_file       ("/usr/local/google/home/blaikie/dev/scratch/test.cpp")
                  DW_AT_decl_line       (8)
                  DW_AT_type    (0x0000005e "int")

0x0000005d:     NULL

0x0000005e:   DW_TAG_base_type
                DW_AT_name      ("int")
                DW_AT_encoding  (DW_ATE_signed)
                DW_AT_byte_size (0x04)

0x00000062:   NULL

The purpose of func_lo_pc is pretty weird, but it's essentially used for runtime relocation of location lists within the function -- we take the static "base" of the function, and the runtime "base", and the difference between the two produces the load bias. Applying the same bias to all variable location lists "relocates" the variables. (The reason this is so convoluted is reading location lists straight from (unrelocated) .o files on mac. I seriously considered changing this when working on debug_rnglists, but it turned out it wasn't really necessary, so I let it go.)

Yep, I figured a bunch of this was for DWARF in unrelocated MachO files - and that they wouldn't be able to/want to use Split DWARF or this feature (which is particularly relevant to Split DWARF). Does any of this logic apply outside that unrelocated MachO scenario?

The "runtime" function address is being taken in https://github.com/llvm/llvm-project/blob/main/lldb/source/Core/ValueObjectVariable.cpp#L159 (and maybe a couple of other places), and these two things need to be in sync. I think this could just be defined as the first address in the first address range of the function. That should be fine for ELF purposes (I have a feeling this thing would completely break down if macho started rearranging parts of the function), but it does bring up another problem.

LLDB's representation of a function (lldb_private::Function) assumes that all functions will be contiguous (Function::GetAddressRange returns a single range). If we make it so that this range matches the first block of the function (maybe that's what happens now),

Actually seems to adjust for discontiguous ranges and not just pick the start of the first (as it appears in the DWARF - which isn't guaranteed to be ordered in any way) range, instead finding the smallest and largest addresses: https://github.com/llvm-mirror/lldb/blob/master/source/Plugins/SymbolFile/DWARF/DWARFASTParserClang.cpp#L2357

then maybe we could make things work for the case where DW_AT_ranges are used as a size optimization. However, getting things to work for truly discontinuous functions will require more work...

(Regarding the test, I think we could avoid running a binary by testing this via the "image lookup --verbose" command, and checking that the output contains the variables it should.)

Ah, sure thing - I've updated it that way.

I don't think that simply setting func_lo_pc to zero will be sufficient to make this work. I would expect this to break in more complicated scenarios (like, even if we just have two of these functions). I think the only reason it works in this case is because for this particular function, it's base address (relative to the CU base) *is* zero.

I certainly don't have high confidence in the change, to be sure - but I think it's a bit more robust than that.

Yes, it seems it is, though just by a bit. :)

The thing which is missing is the location list on the variable DIE -- without it, it does not matter which address gets put here (so long as it does not trigger the assertion), as the value is totally unused. The necessity of the location list is kind of obvious from my description of the problem, though even I did not realize when I was writing it that this requires a more elaborate test case.

Adjusting your test case to produce a location list for i I got this (probably not minimal) snippet (add -O1 to CFLAGS):

#define optnone __attribute__((optnone))
#define noinline __attribute__((noinline))

void optnone noinline stop() {}
void optnone noinline consume(int i) {}
void noinline f1(int x) {
  int i = x;
  stop();
  consume(i);
  i = 5;
  consume(i);
}
int main() {
  int j = 12;
  f1(j);
  stop();
}

Here lldb fails to print the value of i, but if I rearrange the code so that the f1 functions comes first, the value is printed correctly:

#define optnone __attribute__((optnone))
#define noinline __attribute__((noinline))

void optnone noinline stop();
void optnone noinline consume(int i);
void noinline f1(int x) {
  int i = x;
  stop();
  consume(i);
  i = 5;
  consume(i);
}
void optnone noinline stop() {}
void optnone noinline consume(int i) {}
int main() {
  int j = 12;
  f1(j);
  stop();
}

The purpose of func_lo_pc is pretty weird, but it's essentially used for runtime relocation of location lists within the function -- we take the static "base" of the function, and the runtime "base", and the difference between the two produces the load bias. Applying the same bias to all variable location lists "relocates" the variables. (The reason this is so convoluted is reading location lists straight from (unrelocated) .o files on mac. I seriously considered changing this when working on debug_rnglists, but it turned out it wasn't really necessary, so I let it go.)

Yep, I figured a bunch of this was for DWARF in unrelocated MachO files - and that they wouldn't be able to/want to use Split DWARF or this feature (which is particularly relevant to Split DWARF). Does any of this logic apply outside that unrelocated MachO scenario?

Yes, this code is used everywhere, though it's role is less important (and one that could be implemented in an easier manner, were it not for MachO) -- it adjusts the variable address for ASLR, i.e., the variable/function being loaded at a different address than what the static debug info says.

LLDB's representation of a function (lldb_private::Function) assumes that all functions will be contiguous (Function::GetAddressRange returns a single range). If we make it so that this range matches the first block of the function (maybe that's what happens now),

Actually seems to adjust for discontiguous ranges and not just pick the start of the first (as it appears in the DWARF - which isn't guaranteed to be ordered in any way) range, instead finding the smallest and largest addresses: https://github.com/llvm-mirror/lldb/blob/master/source/Plugins/SymbolFile/DWARF/DWARFASTParserClang.cpp#L2357

Ah.. interesting. In that case, I believe things should work if we use the same algorithm to compute func_lo_pc. We may still run into weird issues for functions which are truly discontinuous, but it should be ok for single-range range lists, at least.

dblaikie updated this revision to Diff 318654.Jan 22 2021, 2:41 PM

Retrieve the lowest address in the address ranges, rather than just hardcoding 0

I don't think that simply setting func_lo_pc to zero will be sufficient to make this work. I would expect this to break in more complicated scenarios (like, even if we just have two of these functions). I think the only reason it works in this case is because for this particular function, it's base address (relative to the CU base) *is* zero.

I certainly don't have high confidence in the change, to be sure - but I think it's a bit more robust than that.

Yes, it seems it is, though just by a bit. :)

The thing which is missing is the location list on the variable DIE -- without it, it does not matter which address gets put here (so long as it does not trigger the assertion), as the value is totally unused. The necessity of the location list is kind of obvious from my description of the problem, though even I did not realize when I was writing it that this requires a more elaborate test case.

Adjusting your test case to produce a location list for i I got this (probably not minimal) snippet (add -O1 to CFLAGS):

#define optnone __attribute__((optnone))
#define noinline __attribute__((noinline))

void optnone noinline stop() {}
void optnone noinline consume(int i) {}
void noinline f1(int x) {
  int i = x;
  stop();
  consume(i);
  i = 5;
  consume(i);
}
int main() {
  int j = 12;
  f1(j);
  stop();
}

Here lldb fails to print the value of i, but if I rearrange the code so that the f1 functions comes first, the value is printed correctly:

#define optnone __attribute__((optnone))
#define noinline __attribute__((noinline))

void optnone noinline stop();
void optnone noinline consume(int i);
void noinline f1(int x) {
  int i = x;
  stop();
  consume(i);
  i = 5;
  consume(i);
}
void optnone noinline stop() {}
void optnone noinline consume(int i) {}
int main() {
  int j = 12;
  f1(j);
  stop();
}

Ah, thanks - think I figured out a representative test & understand better.
I ended up with this:

__attribute__((nodebug)) volatile int i;
int main() {
  int var = 3;
  i = 1;
  var = 5;
  i = 2;
}

By using volatile writes, I was able to get the 'var' variable live range to start at the start of the function (so that image lookup -v -s main would render the "var" variable since it's now live at the very start of the function (rather than only after the push instruction)). And use nodebug to reduce the DWARF since the 'i' isn't interesting.

Indeed without the fix you suggested to use "lowest address" rather than zero, this test above would/was failing (running the image lookup command would not show the "var" variable).

Also, I figured out how to run the API tests, and that showed a bunch more failures when using ranges-everywhere (actually a more aggressive version of ranges-everywhere - using it no matter the DWARF version and even when it wouldn't reduce the address pool size (eg: using ranges on a subprogram even when low_pc is an address already in the pool (such as for the start of a CU range))) - and with the change to use the lowest address of the ranges, all those failures now go away.

So *fingers crossed* this is ready.

The purpose of func_lo_pc is pretty weird, but it's essentially used for runtime relocation of location lists within the function -- we take the static "base" of the function, and the runtime "base", and the difference between the two produces the load bias. Applying the same bias to all variable location lists "relocates" the variables. (The reason this is so convoluted is reading location lists straight from (unrelocated) .o files on mac. I seriously considered changing this when working on debug_rnglists, but it turned out it wasn't really necessary, so I let it go.)

Yep, I figured a bunch of this was for DWARF in unrelocated MachO files - and that they wouldn't be able to/want to use Split DWARF or this feature (which is particularly relevant to Split DWARF). Does any of this logic apply outside that unrelocated MachO scenario?

Yes, this code is used everywhere, though it's role is less important (and one that could be implemented in an easier manner, were it not for MachO) -- it adjusts the variable address for ASLR, i.e., the variable/function being loaded at a different address than what the static debug info says.

Ah, good to know!

LLDB's representation of a function (lldb_private::Function) assumes that all functions will be contiguous (Function::GetAddressRange returns a single range). If we make it so that this range matches the first block of the function (maybe that's what happens now),

Actually seems to adjust for discontiguous ranges and not just pick the start of the first (as it appears in the DWARF - which isn't guaranteed to be ordered in any way) range, instead finding the smallest and largest addresses: https://github.com/llvm-mirror/lldb/blob/master/source/Plugins/SymbolFile/DWARF/DWARFASTParserClang.cpp#L2357

Ah.. interesting. In that case, I believe things should work if we use the same algorithm to compute func_lo_pc. We may still run into weird issues for functions which are truly discontinuous, but it should be ok for single-range range lists, at least.

Fair.

dblaikie updated this revision to Diff 318711.Jan 22 2021, 6:10 PM

Include some more details about/in the test

labath accepted this revision.Jan 24 2021, 12:07 PM

Ah, thanks - think I figured out a representative test & understand better.
I ended up with this:

__attribute__((nodebug)) volatile int i;
int main() {
  int var = 3;
  i = 1;
  var = 5;
  i = 2;
}

By using volatile writes, I was able to get the 'var' variable live range to start at the start of the function (so that image lookup -v -s main would render the "var" variable since it's now live at the very start of the function (rather than only after the push instruction)). And use nodebug to reduce the DWARF since the 'i' isn't interesting.

Indeed without the fix you suggested to use "lowest address" rather than zero, this test above would/was failing (running the image lookup command would not show the "var" variable).

Also, I figured out how to run the API tests, and that showed a bunch more failures when using ranges-everywhere (actually a more aggressive version of ranges-everywhere - using it no matter the DWARF version and even when it wouldn't reduce the address pool size (eg: using ranges on a subprogram even when low_pc is an address already in the pool (such as for the start of a CU range))) - and with the change to use the lowest address of the ranges, all those failures now go away.

So *fingers crossed* this is ready.

/me too

lldb/test/Shell/SymbolFile/DWARF/subprogram_ranges.test
22

btw, the way I've dealt with this in the past is to introduce an additional label in the assembly (look_me_up:), and then give that to the image lookup command.

This revision is now accepted and ready to land.Jan 24 2021, 12:07 PM
dblaikie added inline comments.Jan 24 2021, 6:22 PM
lldb/test/Shell/SymbolFile/DWARF/subprogram_ranges.test
22

Oh, makes sense - will certainly keep that in mind!

This revision was landed with ongoing or failed builds.Jan 24 2021, 6:39 PM
This revision was automatically updated to reflect the committed changes.