This is an archive of the discontinued LLVM Phabricator instance.

[AST] Print a<b<c>> without extra spaces in C++11 or later.
ClosedPublic

Authored by sammccall on Mar 25 2020, 1:57 PM.

Details

Diff Detail

Event Timeline

sammccall created this revision.Mar 25 2020, 1:57 PM

@kadircet sending this to you as I noticed it as a clangd hover bug :-) The highest impact is probably in diagnostic messages though.

AFAICT the state before this patch is:

  • the type printer always printed the space
  • the stmt printer calls into the type printer to print arg lists
  • the decl printer never prints the space

The best, most consistent thing would be to also fix the decl printer to respect the setting. But I haven't done that in this patch. At least the default configuration (C++14) is now consistent between type/stmt/decl.

sammccall updated this revision to Diff 252671.Mar 25 2020, 2:19 PM

update/move comment

kadircet accepted this revision.Mar 26 2020, 12:53 AM
kadircet marked an inline comment as done.

LGTM, thanks !

clang/unittests/AST/DeclPrinterTest.cpp
1161–1162

testing scheme here seems to be annoying, had to search for TestTemplateArgumentList8 for a pre-cxx11 case :D

but nothing much to do ...

This revision is now accepted and ready to land.Mar 26 2020, 12:53 AM
This revision was automatically updated to reflect the committed changes.

We ran into three distinct issues with this change.

We updated the compiler and break some VS std::map visualizers:
https://crbug.com/1068394
I haven't 100% confirmed that this caused the issue, but looking at the code suggests that the names in our codeview output changed, which I would consider to be a regression. This should've been caught by clang's test suite. We already have codeview tests that try to ensure clang prints debug info names following MSVC's exact spacing, but apparently none broke, so this area was under-tested. I wonder if gdb or other DWARF consumers might be affected by this formatting change.

Two issues in the Chrome codebase where tests relied on the output of clang's diagnostics:
https://crbug.com/1064986
The compiler's diagnostic output isn't stable, so this is expected.

While I appreciate the effort towards modernization, I want to register that it has created work for folks, and this might come up again in six months during the next release. @tstellar @hans

Sorry about the problems here, and thanks for letting me know...

In D76801#1989421, @rnk wrote:

We updated the compiler and break some VS std::map visualizers:
https://crbug.com/1068394
I haven't 100% confirmed that this caused the issue, but looking at the code suggests that the names in our codeview output changed, which I would consider to be a regression. This should've been caught by clang's test suite. We already have codeview tests that try to ensure clang prints debug info names following MSVC's exact spacing, but apparently none broke, so this area was under-tested. I wonder if gdb or other DWARF consumers might be affected by this formatting change.

I've got a sinking feeling that I misunderstood the implications of updating CodeGenCXX/debug-info-template-explicit-specialization.cpp and the Modules/*DebugInfo.cpp tests.
I assumed that these were human-readable names, not things that tools rely on. As you say maybe the dwarf changes broke things.
I don't really know much about debug info, @dblaikie do you think the DWARF changes in those tests are safe? If not, should I be documenting/mitigating this or trying to undo it?

Regarding CodeView it sounds like we do need to undo this change. PrintingPolicy has:

/// Use whitespace and punctuation like MSVC does. In particular, this prints
/// anonymous namespaces as `anonymous namespace' and does not insert spaces
/// after template arguments.
unsigned MSVCFormatting : 1;

and this is set in CGDebugInfo::getPrintingPolicy() in CodeView mode.

That seems like the right knob to change this behavior.
I'm not sure I know enough to write the codeview test, but I'll give it a shot and send that to you.

Two issues in the Chrome codebase where tests relied on the output of clang's diagnostics:
https://crbug.com/1064986
The compiler's diagnostic output isn't stable, so this is expected.

Yeah, obviously this changed a bunch of clang's own diagnostics tests too.
I'm not sure there's much to be done about this - even with perfect knowledge of such downstream tests, I don't think shying away from marginal improvements in diagnostics would be a great tradeoff.
Sorry about the churn though :-(

rnk added a subscriber: amccarth.Apr 17 2020, 1:48 PM

I've got a sinking feeling that I misunderstood the implications of updating CodeGenCXX/debug-info-template-explicit-specialization.cpp and the Modules/*DebugInfo.cpp tests.
I assumed that these were human-readable names, not things that tools rely on. As you say maybe the dwarf changes broke things.
I don't really know much about debug info, @dblaikie do you think the DWARF changes in those tests are safe? If not, should I be documenting/mitigating this or trying to undo it?

Regarding CodeView it sounds like we do need to undo this change. PrintingPolicy has:

/// Use whitespace and punctuation like MSVC does. In particular, this prints
/// anonymous namespaces as `anonymous namespace' and does not insert spaces
/// after template arguments.
unsigned MSVCFormatting : 1;

and this is set in CGDebugInfo::getPrintingPolicy() in CodeView mode.

That seems like the right knob to change this behavior.
I'm not sure I know enough to write the codeview test, but I'll give it a shot and send that to you.

Yep, that sounds like the right fix. I was imagining some policy method useMSVCFormatting that flips both options together.

Hang on, I may have been too quick. We *do* have a test for this, at the end of clang/test/CodeGenCXX/debug-info-codeview-display-names.cpp:
https://github.com/llvm/llvm-project/blob/f8452ddfcc3336e42544a35481507f0b3bae423e/clang/test/CodeGenCXX/debug-info-codeview-display-name.cpp#L98

The test passes for me locally, although I'm not sure why.

I would leave things a bit longer to give @amccarth more time to investigate and confirm. Thanks for the offer to help, though.

In D76801#1989718, @rnk wrote:

Hang on, I may have been too quick. We *do* have a test for this, at the end of clang/test/CodeGenCXX/debug-info-codeview-display-names.cpp:
https://github.com/llvm/llvm-project/blob/f8452ddfcc3336e42544a35481507f0b3bae423e/clang/test/CodeGenCXX/debug-info-codeview-display-name.cpp#L98

The test passes for me locally, although I'm not sure why.

After this patch the default depends on the language version, and that test specifies c++98 :-(

I would leave things a bit longer to give @amccarth more time to investigate and confirm. Thanks for the offer to help, though.

Sure thing, let me know if it's useful for me to pick this up. I'm sure the debugging was the worst of it though, sorry.

Sorry about the problems here, and thanks for letting me know...

In D76801#1989421, @rnk wrote:

We updated the compiler and break some VS std::map visualizers:
https://crbug.com/1068394
I haven't 100% confirmed that this caused the issue, but looking at the code suggests that the names in our codeview output changed, which I would consider to be a regression. This should've been caught by clang's test suite. We already have codeview tests that try to ensure clang prints debug info names following MSVC's exact spacing, but apparently none broke, so this area was under-tested. I wonder if gdb or other DWARF consumers might be affected by this formatting change.

I've got a sinking feeling that I misunderstood the implications of updating CodeGenCXX/debug-info-template-explicit-specialization.cpp and the Modules/*DebugInfo.cpp tests.
I assumed that these were human-readable names, not things that tools rely on. As you say maybe the dwarf changes broke things.
I don't really know much about debug info, @dblaikie do you think the DWARF changes in those tests are safe? If not, should I be documenting/mitigating this or trying to undo it?

( @aprantl @JDevlieghere @probinson @cmtice @tamur @labath )

In general DWARF doesn't specify this at all. In reality, at least GDB does some fuzzy matching - I don't know about LLDB and SCE (Sony's debugger).

GDB's fuzzy matching seems sufficient to make this work/not be a problem here.

Here's one example of testing that. Compile these two files with different compilers (either Clang<with this patch>+GCC, or Clang<with>+Clang<without>):

template<typename T>
struct tmpl {
  int member = 42;
};

const tmpl<tmpl<int>> *get_obj() {
  static tmpl<tmpl<int>> t;
  return &t;
}
int get_mem(const tmpl<tmpl<int>> *t) {
  return t->member;
}
template<typename T>
struct tmpl;

const tmpl<tmpl<int>> *get_obj();
int get_mem(const tmpl<tmpl<int>> *t);

int main() {
  const tmpl<tmpl<int>> *x = get_obj();
  return get_mem(x);
}

Then launch that in a debugger & stop in main, print the type of 'x' - it should include the int member named 'member':

type = const struct tmpl<tmpl<int> > [with T = tmpl<int>] {
    int member;
} *

If the debugger can't associate the type in the two files together, then the type of 'x' would print out as an incomplete type (you can test this by compiling the first file without debug info):

type = const struct tmpl<tmpl<int>> {
    <incomplete type>
} *

David's example does work with gdb without -Wl,--gdb-index (the member variable is shown), presumably due to the aforementioned fuzzy matching. However, it does not work if gdb-index is enabled, nor with lldb (as lldb is always very index-oriented and assumes equality everywhere). That is definitely not ideal, though I'm not sure that means about this patch. This is definitely not the only difference in the formatting of DW_AT_names of templates. For example, template<typename T> operator<<(T, T) will come out as operator<< <A> with gcc, but as operator<<<A> with clang (with or without this patch).
OTOH, differences in type names are more likely to cause problems than is the case for functions/operators.

David's example does work with gdb without -Wl,--gdb-index (the member variable is shown), presumably due to the aforementioned fuzzy matching. However, it does not work if gdb-index is enabled, nor with lldb (as lldb is always very index-oriented and assumes equality everywhere). That is definitely not ideal, though I'm not sure that means about this patch. This is definitely not the only difference in the formatting of DW_AT_names of templates. For example, template<typename T> operator<<(T, T) will come out as operator<< <A> with gcc, but as operator<<<A> with clang (with or without this patch).
OTOH, differences in type names are more likely to cause problems than is the case for functions/operators.

That is concerning. Any idea if that's only with lld's gdb-indexx implementation, or also gold's? This isn't the only naming divergence between GCC and Clang, though, so if gdb-index doesn't support any divergence, that's a problem... (I think non-type template parameters diverge too, perhaps? well, GCC 6 used to print "foo<3u>" (for an unsigned non-type template parameter) and clang/more recent GCC versions just use "foo<3>" now, for instance - enums, GCC uses foo<(EnumType)0> whereas Clang uses foo<enumeratorName> if it's named and foo<3> if it's unnamed))

Differences in function names could manifest as differences in type names - due to local types nested inside functions, but they're less likely/highly unlikely to have a decl/def resolution between CUs, so maybe nested types are less problematic.

aprantl added a subscriber: shafik.Apr 20 2020, 4:32 PM

David's example does work with gdb without -Wl,--gdb-index (the member variable is shown), presumably due to the aforementioned fuzzy matching. However, it does not work if gdb-index is enabled, nor with lldb (as lldb is always very index-oriented and assumes equality everywhere). That is definitely not ideal, though I'm not sure that means about this patch. This is definitely not the only difference in the formatting of DW_AT_names of templates. For example, template<typename T> operator<<(T, T) will come out as operator<< <A> with gcc, but as operator<<<A> with clang (with or without this patch).
OTOH, differences in type names are more likely to cause problems than is the case for functions/operators.

That is concerning. Any idea if that's only with lld's gdb-indexx implementation, or also gold's?

I was using gold for my initial test. However, the same thing happens with lld too...

This isn't the only naming divergence between GCC and Clang, though, so if gdb-index doesn't support any divergence, that's a problem...

It becomes a gdb-index problem because with an index the debugger will do a (hashed?) string lookup and expect the string to be there. If the strings differ, the lookup won't find anything. In the no-index scenario, the debugger has to trawl the debug info itself, and so it has some opportunities to do fuzzy matching.

David's example does work with gdb without -Wl,--gdb-index (the member variable is shown), presumably due to the aforementioned fuzzy matching. However, it does not work if gdb-index is enabled, nor with lldb (as lldb is always very index-oriented and assumes equality everywhere). That is definitely not ideal, though I'm not sure that means about this patch. This is definitely not the only difference in the formatting of DW_AT_names of templates. For example, template<typename T> operator<<(T, T) will come out as operator<< <A> with gcc, but as operator<<<A> with clang (with or without this patch).
OTOH, differences in type names are more likely to cause problems than is the case for functions/operators.

That is concerning. Any idea if that's only with lld's gdb-indexx implementation, or also gold's?

I was using gold for my initial test. However, the same thing happens with lld too...

Good to know! Thanks for checking!

This isn't the only naming divergence between GCC and Clang, though, so if gdb-index doesn't support any divergence, that's a problem...

It becomes a gdb-index problem because with an index the debugger will do a (hashed?) string lookup and expect the string to be there. If the strings differ, the lookup won't find anything. In the no-index scenario, the debugger has to trawl the debug info itself, and so it has some opportunities to do fuzzy matching.

That surprises me a bit - given that one of the things debuggers provide is autocomplete (I'm unlikely to write out a type name exactly as the debug info contains it - if two compilers can't agree, it's much less likely that all users will agree with any compiler rendering), which I'd have thought would be facilitated by the index too - in which case lookup via exact match wouldn't be viable, you'd still want a way to list anything with a matching substring (or at least prefix), etc. Which could help facilitate lookup fuzzy matching like in this sort of case.

It becomes a gdb-index problem because with an index the debugger will do a (hashed?) string lookup and expect the string to be there. If the strings differ, the lookup won't find anything. In the no-index scenario, the debugger has to trawl the debug info itself, and so it has some opportunities to do fuzzy matching.

That surprises me a bit - given that one of the things debuggers provide is autocomplete (I'm unlikely to write out a type name exactly as the debug info contains it - if two compilers can't agree, it's much less likely that all users will agree with any compiler rendering), which I'd have thought would be facilitated by the index too - in which case lookup via exact match wouldn't be viable, you'd still want a way to list anything with a matching substring (or at least prefix), etc. Which could help facilitate lookup fuzzy matching like in this sort of case.

That is kind of true, and I don't really have a definitive reply to that. I suppose there is a difference between the lookups done for type completion and those done e.g. in expression evaluation. The latter are probably more frequent and are assumed to be correct. Maybe they shouldn't be, but in that cases, then there would probably be no use for the hash tables in indexes (I am not very familiar with the gdb index, but I know it has hash tables similar to debug_names/apple_names). I don't know what gdb does for tab completion, but it probably bypasses the hash table (though the index could still be useful even then as it has a concise list of all strings in the debug info), or it gets the list of strings-to-complete from a completely different source (demangled function names?).

Tab completion is always a bit dodgy. E.g., in your example ptype tmpl<TAB> completes to ptype tmpl<tmpl<int>>, but then running that produces an error: No symbol "tmpl" in current context.

In lldb, tab completion in expressions works by hooking into the regular clang tab-completion machinery used by editors. The up- and down-side of that is that it uses the same code path used for actual expression evaluation -- i.e. all the types will be looked up the same (exact) way.

Speaking of templates and indexes, the thing we would really like in lldb would be to have just the bare names of templated class in the indexes -- that way we could reliably look up all instantiations of a template and do the filtering ourselves. However, this runs afoul of the dwarf specification, which says that the index names should match the DW_AT_names of relevant DIEs (and these contain the template arguments for other reasons...). This means that currently we have outstanding issues when looking up templated types, but we haven't really figured out what to do about that...

It becomes a gdb-index problem because with an index the debugger will do a (hashed?) string lookup and expect the string to be there. If the strings differ, the lookup won't find anything. In the no-index scenario, the debugger has to trawl the debug info itself, and so it has some opportunities to do fuzzy matching.

That surprises me a bit - given that one of the things debuggers provide is autocomplete (I'm unlikely to write out a type name exactly as the debug info contains it - if two compilers can't agree, it's much less likely that all users will agree with any compiler rendering), which I'd have thought would be facilitated by the index too - in which case lookup via exact match wouldn't be viable, you'd still want a way to list anything with a matching substring (or at least prefix), etc. Which could help facilitate lookup fuzzy matching like in this sort of case.

That is kind of true, and I don't really have a definitive reply to that. I suppose there is a difference between the lookups done for type completion and those done e.g. in expression evaluation. The latter are probably more frequent and are assumed to be correct. Maybe they shouldn't be, but in that cases, then there would probably be no use for the hash tables in indexes (I am not very familiar with the gdb index, but I know it has hash tables similar to debug_names/apple_names). I don't know what gdb does for tab completion, but it probably bypasses the hash table (though the index could still be useful even then as it has a concise list of all strings in the debug info), or it gets the list of strings-to-complete from a completely different source (demangled function names?).

Tab completion is always a bit dodgy. E.g., in your example ptype tmpl<TAB> completes to ptype tmpl<tmpl<int>>, but then running that produces an error: No symbol "tmpl" in current context.

In lldb, tab completion in expressions works by hooking into the regular clang tab-completion machinery used by editors. The up- and down-side of that is that it uses the same code path used for actual expression evaluation -- i.e. all the types will be looked up the same (exact) way.

Speaking of templates and indexes, the thing we would really like in lldb would be to have just the bare names of templated class in the indexes -- that way we could reliably look up all instantiations of a template and do the filtering ourselves. However, this runs afoul of the dwarf specification, which says that the index names should match the DW_AT_names of relevant DIEs (and these contain the template arguments for other reasons...). This means that currently we have outstanding issues when looking up templated types, but we haven't really figured out what to do about that...

Yeah, points all taken - as for this actual issue... I'm kind of inclined to say "hey, our template names already diverge somewhat - and this divergence is in the realm of acceptable by gdb (without an index) so... *thumbs up*/let's stick with it" & as I think you mentioned earlier, this is more a problem for the index support, than for the debug info producer. I don't think it's a reasonable goal that Clang & GCC produce /exactly/ the same names for templates like this... I mean, we /could/ try, but I don't think it's worthwhile (especially given that gdb's got support intended to do fuzzy matching/allow some divergence)

rnk added a comment.Apr 23 2020, 11:13 AM

Thanks David, I'm comfortable with that stance for DWARF.

I know @amccarth is looking into the recent VS visualizer issue, and I would like him to confirm if it was this change or not when he gets a chance.

Yeah, points all taken - as for this actual issue... I'm kind of inclined to say "hey, our template names already diverge somewhat - and this divergence is in the realm of acceptable by gdb (without an index) so... *thumbs up*/let's stick with it"

Another interesting aspect here is that the DW_AT_name outputs depend on the c++ standard versions used. This means we could get mismatches even with the same compiler if some compile units use -std=c++98, and others -std>=c++11 (hardly a recommended practice but it does work if one knows what he is doing). Compatibility with another compiler is one thing, but maybe self-compatibility is more important (and easier to achieve) ?

One way to achieve that would be by printing all type names in c++98 mode, which (IIUC) is the same thing as what the windows folks are requesting..

Yeah, points all taken - as for this actual issue... I'm kind of inclined to say "hey, our template names already diverge somewhat - and this divergence is in the realm of acceptable by gdb (without an index) so... *thumbs up*/let's stick with it"

Another interesting aspect here is that the DW_AT_name outputs depend on the c++ standard versions used.

Yeah, I find that at least "weird", though not necessarily wrong.

This means we could get mismatches even with the same compiler if some compile units use -std=c++98, and others -std>=c++11 (hardly a recommended practice but it does work if one knows what he is doing). Compatibility with another compiler is one thing, but maybe self-compatibility is more important (and easier to achieve) ?

Yeah, I don't disagree with that - again, not quite sure I'd say it goes as far as "wrong" (I mean, DWARF doesn't spec this - so wrongness in our own judgment, not any authoritative sense) but yeah, perhaps insufficiently motivated quirkiness.

One way to achieve that would be by printing all type names in c++98 mode, which (IIUC) is the same thing as what the windows folks are requesting..

Fair enough - I wouldn't object to that change being made & just having it the same for MSVC and DWARF in this case. (it's a /bit/ less good for C++11 and above users to get names with the extra space, but seems minor enough perhaps not to bother trying to preserve it given the complicaitons)

dyung added a subscriber: dyung.May 6 2020, 12:04 PM

Hi, we noticed an issue with the GDB test suite that was bisected back to this change and I have put the details in PR46052. Can you take a look?

Hi, we noticed an issue with the GDB test suite that was bisected back to this change and I have put the details in PR46052. Can you take a look?

Sorry this never got clearly resolved.
I think the conclusion is while this is all best-effort, we might as well undo the debuginfo changes here. Sent D80554