This is an archive of the discontinued LLVM Phabricator instance.

dwarfdump: Add an option to collect debug info quality metrics
ClosedPublic

Authored by aprantl on Aug 11 2017, 11:01 AM.

Details

Summary

At the last LLVM dev meeting we had a debug info for optimized code BoF session. In that session I presented some graphs that showed how the quality of the debug info produced by LLVM changed over the last couple of years. This is a cleaned up version of the patch I used to collect the this data. It is implemented as an extension of llvm-dwarfdump, adding a new --statistics option. The intended use-case is to automatically run this on the debug info produced by, e.g., our bots, to identify eyebrow-raising changes or regressions introduced by new transformations that we could act on.

In the current form, two kinds of data are being collected:

  • the number of variables that have a debug location versus the number of variables in total (this takes into account inlined instances of the same function, so if a variable is completely missing form only one instance it will be found)
  • the PC range covered by variable location descriptions versus the PC range of all variables' containing lexical scopes

The output format is versioned and extensible, so I'm looking forward to both bug fixes and ideas for other data that would be interesting to track.

Diff Detail

Event Timeline

aprantl created this revision.Aug 11 2017, 11:01 AM
dblaikie added inline comments.Aug 12 2017, 11:11 PM
lib/DebugInfo/DWARF/DWARFDebugLoc.cpp
123–124

I believe C++11 lower_bound supports heterogenous lookup, so you don't need to create a LocationList to query - you can write an asymmetric comparison (unsigned < LocationList) & use that.

I believe we use that to find a CU at an offset - the function's in DWARFContext, I think (just search for lower_bound in lib/DebugInfo/DWARF and you should find this example easily enough, probably)

126

Might be worth adding a range-based lower_bound to STLExtras.h? I think there are a few uses of std::lower_bound that could be switched to that.

128

No need for the "-> bool" here, if you want to skip it.

tools/llvm-dwarfdump/Statistics.cpp
65

Name wouldn't be unambiguous - due to shadowing?

68

Should we gather statistics on that? (honestly there might be cases where it's more efficient to describe a variable as being in one location with one address range - even though the scope has a hole in it, for example (eg: scope says [a, b)+[c,d)+[e,f) but the variable location says '[a,b) it's in register foo, [c,f) it's in register bar' - I /think/ that's probably OK to do, the debugger/consumer shouldn't consult the variable if it's looking at an instruction in [d,e), right?)

179

Worth having two formats?

It seems especially strange/uncommon to have both outputs generated in a single run and need to filter them out with different pipes - is there any precedent for that in other LLVM tools?

I'd say if, as the comment above suggests, the stats are only really interesting in comparison with stats from another run, then maybe only the machine-readable form is relevant?

194–195

that'll get rounded down, if I recall correctly - is that desirable (eg: 49.999% will render as 49% not 50%?)

197

This might not be the best statistic.

Consider code like this

void f() {
  g();
  h j = k();
  return j;
}

optimized, 'j' should never have a location range that covers the entire scope of 'f'. There's no value for 'j' during 'g()'.

So maybe it'd be necessary to track which bytes of a scope are covered by a location list, after the earliest byte in the scope that is covered.

Separately you could track this statistic (which bytes of a scope are covered by the variable) - but not expect this statistic to approach 100%.

aprantl marked 4 inline comments as done.Aug 16 2017, 12:02 PM
aprantl added inline comments.
lib/DebugInfo/DWARF/DWARFDebugLoc.cpp
123–124

I added an operator< which is not quite what you suggested, but what all the other DWARF lookups were doing.

tools/llvm-dwarfdump/Statistics.cpp
65

Yes, this doesn't handle

void f() {
  { int i = 1; }
  { int i = 2; }
}

correctly right now, and I'm also not quite sure how to fix this right now. We could give all lexical scopes unique names based on their nesting levels, but if there are two inlined instances of the same function:

void f(bool b) {
  if (b) {
    int i = 1;
  } else {
    int i = 2;
  }
}

one inlined as f(true), and one as f(false) with only one lexical block remaining in each inlined instance, I don't know how to handle that.

68

Yeah, it's not something I would collect always, but as a one-off it would be interesting. I recently fixed a bug related to this in r305599 and there's probably more opportunities.

197

This case even appears in my test case. The point of this number was to look at the delta between two compilers, and not for the ratio to ever approach 100%.
I really like the idea of only starting at the first byte of coverage though, so changed the implementation to collect number instead.

aprantl updated this revision to Diff 111395.Aug 16 2017, 12:06 PM
aprantl marked an inline comment as done.

Address review feedback. Thanks!

aprantl updated this revision to Diff 111416.Aug 16 2017, 1:38 PM

Fix typo and refactor.

rnk added a reviewer: rnk.Sep 7 2017, 11:03 AM
rnk edited edge metadata.Sep 7 2017, 11:09 AM

You probably want to rebase this to pick up rL312042.

tools/llvm-dwarfdump/Statistics.cpp
65

Numbering the scopes reminds me of what MSVC does for name mangling static locals. I don't think it's terribly important to worry about local variable shadowing in the first iteration of this. I suspect it mostly comes up for induction variables (I and E).

aprantl updated this revision to Diff 116089.Sep 20 2017, 3:22 PM

Rebased and addressed additional review feedback.

rnk accepted this revision.Oct 6 2017, 10:24 AM

pinging @dblaikie.

ping.

@dblaikie is at a conference. I think we should go ahead and land this and iterate. It's a testing tool anyway. Looks good!

tools/llvm-dwarfdump/Statistics.cpp
175–176

This sentence needs editting, I think it's supposed to say, "It's impossible to reduce ..."

This revision is now accepted and ready to land.Oct 6 2017, 10:24 AM

I think we should go ahead and land this and iterate.

Yes, I would also like to point out that this is meant to be a starting point and an invitation to the community to improve / add other (hopefully even better) metrics to the tool!

This revision was automatically updated to reflect the committed changes.