This is an archive of the discontinued LLVM Phabricator instance.

[lldb][WIP] Use forward decls with redeclared definitions instead of minimal import for records
Needs ReviewPublic

Authored by teemperor on May 5 2021, 3:20 PM.

Details

Reviewers
shafik
Summary

This patch is rewriting the way we move Clang types between different ASTS in Clang.

The motivation is that the current approach for 'completing types' in LLDB just doesn't work. We have all kinds of situations where we either have Clang crash due to for example having a Record without DefinitionData but with FieldDecls in it.

The reason for that is that at the moment we create record types ([CXX]RecordDecls and ObjCInterfaceDecls) and we always pretend that a type potentially has a definition that somehow could be lazily pulled in. However, Clang doesn't have any interface (at least none that is consistently called) that turns a RecordDecl without DefinitionData into a RecordDecl with DefinitionData. The only relevant API in the ExternalASTSource is CompleteType is suffering from the fact that it's essentially not used by generic Clang code.

The part of the ExternalASTSource API that is consistently called to pull in a potential definition is CompleteRedeclChain (which is for example automatically called when calling getDefinition on a RecordDecl). The problem with CompleteRedeclChain however is that it's not supposed to add definition data to the Decl its called on, but instead it should provide a redeclaration that is defining the record. That's a very different system than what we currently have and we can't just swap it out under the hood.

To make it short: We probably need to rewrite that part of LLDB.

So the solution here is essentially rewriting all our completion code to do this:

  1. Instead of creating these weirdly defined records that have fields but maybe not definition and so on, we only create forward decls at the start.
  2. We then bump the GenerationCounter of the ExternalASTSource of the current AST to force rebuilding of the redeclaration chain.
  3. When Clang asks for the definition of the record, it will ask the ExternalASTSource to complete the redeclaration chain. At this point we build the definition and add it as a second decl.

The ASTImporter can now also stop using the minimal mode for records as there is no reason anymore. It just imports first the forward declaration and then the definition when needed.

Diff Detail

Event Timeline

teemperor created this revision.May 5 2021, 3:20 PM
teemperor requested review of this revision.May 5 2021, 3:20 PM
teemperor edited the summary of this revision. (Show Details)May 5 2021, 3:28 PM
teemperor edited the summary of this revision. (Show Details)
teemperor updated this revision to Diff 343396.May 6 2021, 7:07 AM
teemperor edited the summary of this revision. (Show Details)
  • Add a second DIEtoX map from DIEs to records (and their definitions once we have them).
  • Avoid some calls that try to find the 'interesting' TagDecl redeclaration.
martong added a subscriber: martong.May 6 2021, 7:39 AM

This is a great initiative! Finally, the CTU analysis code and LLDB could share a simplified ASTImporter. In the past years we've seen that the minimal import mode caused many corner cases in the ASTImporter and complicated the code pretty much.

lldb/source/Plugins/ExpressionParser/Clang/ClangASTImporter.h
272

Macro celebration_balloons:

Just to clarify because Jan abandoned the linked ASTImporter patch: I don't have a timeline for this. If I find the time to work on this then this might land quite soon or maybe it will only land in a few months. Or I'll run into something serious that just won't work with this approach and we have to abandon this.

  • All C tests are passing.
  • Nearly all Obj-C tests are passing.
  • about 70% of the C++ tests are passing now.
teemperor edited the summary of this revision. (Show Details)Jul 12 2021, 10:16 AM
teemperor planned changes to this revision.Aug 19 2021, 7:15 AM

Just moving this out of the review queue. Also as there are a bunch of Clang changes (that probably trigger a ton of Herald rules), I keep pushing new updates to https://github.com/teemperor/llvm-project/tree/Rewrite until I can split all those changes out into their own smaller patches.

Just want to say this sounds like a really great direction - putting lldb in a codepath that's much more robustly tested (by lazy loading modules during normal compilation) than the "let's create interesting incomplete types in ways Clang Sema never expects, and then try to patch up all the holes where Clang assumes the type must be complete". Really looking forward to this & hope it provides the sort of robustness improvements it sounds like it should.

jgorbe added a subscriber: jgorbe.Oct 25 2021, 10:16 AM
teemperor updated this revision to Diff 386866.Nov 12 2021, 8:41 AM
teemperor edited the summary of this revision. (Show Details)
  • Update from git
martong added a subscriber: labath.Mar 31 2022, 2:54 AM

@JDevlieghere @labath Do you know if someone took it over from @teemperor (since AFAIK he is no longer working on lldb) ? Are there any plans to continue this effort? I am interested as a clang::ASTImporter maintainer, we could have so many simplifications in clang::ASTImporter if this patch will have been landed sometime.

Herald added a project: Restricted Project. · View Herald TranscriptMar 31 2022, 2:54 AM

@JDevlieghere @labath Do you know if someone took it over from @teemperor (since AFAIK he is no longer working on lldb) ? Are there any plans to continue this effort? I am interested as a clang::ASTImporter maintainer, we could have so many simplifications in clang::ASTImporter if this patch will have been landed sometime.

Yup. We've hired someone new to work on the expression evaluator in LLDB but that person hasn't started yet and will need some time to ramp up. I'll sync with @aprantl and see if this is something we can prioritize.

rnk added a subscriber: rnk.Oct 3 2022, 2:58 PM
cmtice added a subscriber: cmtice.Oct 3 2022, 4:03 PM