This is an archive of the discontinued LLVM Phabricator instance.

Decouple ClangASTContext from DWARF
ClosedPublic

Authored by zturner on Mar 22 2016, 3:22 PM.

Details

Summary

This patch attempts to remove the coupling between ClangASTContext and DWARF debug information. Previously, TypeSystem exposed a method called GetDWARFParser, which means that any TypeSystem had to have some concept of DWARF. If we want to implement a PDB TypeSystem though, it doesn't make sense to support DWARF, and we don't want to go adding something like GetPDBParser to the generic TypeSystem either.

This patch removes all of this code from TypeSystem and implements it specifically on ClangASTContext. SymbolFileDWARF and related classes (which are all DWARF specific anyway) can now use llvm casting mechanics to check if the TypeSystem is a ClangASTContext, and if so it can use GetDWARFParser.

In a followup patch, this will allow PDB and DWARF debug information to in theory live side by side for the same module.

This patch doesn't introduce a PDBASTParser yet, it just lays the framework to make this possible in a followup patch.

Diff Detail

Repository
rL LLVM

Event Timeline

zturner updated this revision to Diff 51356.Mar 22 2016, 3:22 PM
zturner retitled this revision from to Decouple ClangASTContext from DWARF.
zturner updated this object.
zturner added a reviewer: clayborg.
zturner added a subscriber: lldb-commits.
clayborg requested changes to this revision.Mar 22 2016, 4:22 PM
clayborg edited edge metadata.

After our last conversation I must have missed that both Go and Java need to be able to provide custom DWARF parsing. Any language, and any TypeSystem, should be able to provide a DWARFASTParser. They don't have to, but they can. So we will need to revert ALL the changes that remove the GetDWARFParser from the TypeSystem and restore them.

This patch them becomes a method to factor the clang::ExternalASTSource stuff only so that you will be able to use it in the SymbolFilePDB. Sorry for the confusion..

include/lldb/Symbol/ClangASTContext.h
37–39 ↗(On Diff #51356)

Revert. We will need to restore this since Go and Java TypeSystem subclasses need to provide DWARF parsers that translate the DWARF into types for those languages in their type systems. This is my fault.. So just leave this as is.

69–71 ↗(On Diff #51356)

Revert. We will need to restore this since Go and Java TypeSystem subclasses need to provide DWARF parsers that translate the DWARF into types for those languages in their type systems. This is my fault.. So just leave this as is.

521–526 ↗(On Diff #51356)

We will need to restore this since Go and Java TypeSystem subclasses need to provide DWARF parsers that translate the DWARF into types for those languages in their type systems. This is my fault.. So just leave this as is.

1228 ↗(On Diff #51356)

Revert. We will need to restore this since Go and Java TypeSystem subclasses need to provide DWARF parsers that translate the DWARF into types for those languages in their type systems. This is my fault.. So just leave this as is.

include/lldb/Symbol/GoASTContext.h
63–64 ↗(On Diff #51356)

Revert. We will need to restore this since Go and Java TypeSystem subclasses need to provide DWARF parsers that translate the DWARF into types for those languages in their type systems. This is my fault.. So just leave this as is.

390 ↗(On Diff #51356)

Revert. We will need to restore this since Go and Java TypeSystem subclasses need to provide DWARF parsers that translate the DWARF into types for those languages in their type systems. This is my fault.. So just leave this as is.

include/lldb/Symbol/JavaASTContext.h
61–63 ↗(On Diff #51356)

Revert. We will need to restore this since Go and Java TypeSystem subclasses need to provide DWARF parsers that translate the DWARF into types for those languages in their type systems. This is my fault.. So just leave this as is.

371 ↗(On Diff #51356)

Revert. We will need to restore this since Go and Java TypeSystem subclasses need to provide DWARF parsers that translate the DWARF into types for those languages in their type systems. This is my fault.. So just leave this as is.

include/lldb/Symbol/TypeSystem.h
31 ↗(On Diff #51356)

This line can still go as it isn't needed.

32 ↗(On Diff #51356)

Revert. We will need to restore this since Go and Java TypeSystem subclasses need to provide DWARF parsers that translate the DWARF into types for those languages in their type systems. This is my fault.. So just leave this as is.

102–107 ↗(On Diff #51356)

Revert. We will need to restore this since Go and Java TypeSystem subclasses need to provide DWARF parsers that translate the DWARF into types for those languages in their type systems. This is my fault.. So just leave this as is.

source/Plugins/SymbolFile/DWARF/DWARFDIE.cpp
12–30 ↗(On Diff #51356)

Revert all changes in this file. We will need to restore this since Go and Java TypeSystem subclasses need to provide DWARF parsers that translate the DWARF into types for those languages in their type systems. This is my fault.. So just leave this as is.

408–415 ↗(On Diff #51356)

Revert. We will need to restore this since Go and Java TypeSystem subclasses need to provide DWARF parsers that translate the DWARF into types for those languages in their type systems. This is my fault.. So just leave this as is.

source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.cpp
57 ↗(On Diff #51356)

Revert. We will need to restore this since Go and Java TypeSystem subclasses need to provide DWARF parsers that translate the DWARF into types for those languages in their type systems. This is my fault.. So just leave this as is.

997–1000 ↗(On Diff #51356)

Revert. We will need to restore this since Go and Java TypeSystem subclasses need to provide DWARF parsers that translate the DWARF into types for those languages in their type systems. This is my fault.. So just leave this as is.

1465–1469 ↗(On Diff #51356)

Revert. We will need to restore this since Go and Java TypeSystem subclasses need to provide DWARF parsers that translate the DWARF into types for those languages in their type systems. This is my fault.. So just leave this as is.

1617–1625 ↗(On Diff #51356)

Revert unless you need to route this to your new Clang external source changes you made.

1635–1638 ↗(On Diff #51356)

revert.

3828–3846 ↗(On Diff #51356)

revert

source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.h
246 ↗(On Diff #51356)

Revert. Whitespace only change

source/Symbol/GoASTContext.cpp
1642–1649 ↗(On Diff #51356)

revert

source/Symbol/JavaASTContext.cpp
499–506 ↗(On Diff #51356)

revert

This revision now requires changes to proceed.Mar 22 2016, 4:22 PM

Should we go back to my original patch then? It left the parser accessible
from TypeSystem, but made it a DebugInfoParser instead of a DWARFParser.

The whole idea is to remove knowledge of a specific debug info format from
the generic TypeSystem, so it still seems wrong to add DWARF specific stuff
to TypeSystem.

One option is to add back the SymbolFormat enum and make a method in
TypeSystem like

DebugInfoASTParser *
GetASTParserForDebugFormat(SymbolFormat format)

This way it's not DWARF specific.

Another option is to add a method for pdb and a method for DWARF. I think
the former is more flexible though.

No. Keep GetDWARFParser() on TypeSystem and all that code as is. You shouldn't need to add GetPDBParser() to TypeSystem as you should get a type system for language from within your SymbolFilePDB (with a language like C or C++) and then make a PDBParser with a reference to your ClangASTContext. So no one else should need to know. The main reason we get the DWARF parser from the type system is we have 2 ways to have DWARF with references to other DWARF files like the MacOSX DWARF in .o files with a debug map where each .o file is a separate SymbolFileDWARF, but they all need to use the common ClangASTContext, so we can't just have each SymbolFileDWARF create its own DWARFASTParser, we need to share a single one. You shouldn't have this problem with PDB since everything is within a single file. We will need to make sure the MacOSX test suite still runs clean with any changes.

I guess my point was just that it seems a little weird to treat DWARF specially. It's just another debug info format (albeit one that most compilers support), so by baking into the supposedly generic interface the requirement that "every type system must be able to support DWARF" seems a little strange.

I agree it solves the problem and everything will work, it just seems more correct from a design standpoint to have an enum for debug info type and expose one method called GetDebugInfoParser that takes the enum.

I

I guess my point was just that it seems a little weird to treat DWARF specially. It's just another debug info format (albeit one that most compilers support), so by baking into the supposedly generic interface the requirement that "every type system must be able to support DWARF" seems a little strange.

We can change the TypeSystem version of this function to not be pure virtual, and have a default implementation that returns nullptr. I do think it is valid to have a function that says "if you want to support parsing DWARF with this type system, please return a DWARFASTParser subclass that can convert DWARF into your types". I see no problem with this.

I agree it solves the problem and everything will work, it just seems more correct from a design standpoint to have an enum for debug info type and expose one method called GetDebugInfoParser that takes the enum.

I don't agree. DWARFASTParser has very specific DWARF parsing functions on it that make no sense to anyone else. Making a enum doesn't help with this, nor does making a common base class that can't do anything for anyone. So I firmly believe this is valid. Mostly because DWARF supports so many languages and each language might have its own type system. A single DWARF file could have Java, C, C++, and Go all in the same DWARF file. Each type system can provide its own DWARFASTParser. Take a look at the DWARFASTParser, most of it is very DWARF specific.

zturner updated this revision to Diff 51476.Mar 23 2016, 2:25 PM
zturner edited edge metadata.

This patch has gotten both bigger and smaller. Smaller in the sense that after taking into consideration the most recent comments, there was really only one piece left of my patch. It is the piece that abstracted out CanComplete, CompleteType, and LayoutRecord into a new class called ClangTypeImportHelper so that the same code could be re-used by PDBASTParser.

But upon closer inspection, this code ultimately just contained multiple levels of indirection to do what seems to be better housed inside of ClangASTImporter. I mean, ClangASTImporter::Import(), makes sense right? Physically, there's no real different in doing it this way versus the other way. In both cases the DWARFASTParserClang had an instance of the thing. Both the ClangASTImporter and the ClangTypeImportHelper with my first patch, and just a ClangASTImporter with my second patch. But the code is much cleaner and more straightforward now, with fewer ping-ponging of API calls to get to the right place now that those 3 functions live directly isnide of ClangASTImporter.

In doing so though, I had some difficulty with the circular dependency relationship between ClangASTImporter and ClangASTContext. They were each calling into each other in some cases, and this was leaking into header files causing compiler errors. To address this I identified the code that needed to be shared (a couple of static methods in ClangASTContext) and moved them into a new file called ClangUtil.

Unfortunately this dirties up the CL and makes it look really large, even though a lot of the changes are just namespace fixes.

There are still many many instances of this kind of thing remaining in ClangASTContext which i did not address (because it quickly would have made the CL un-reviewable), but I would like to continue cleaning this up in subsequent patches. ClangASTContext.cpp is a massive file, and anything we can do to split this code up will help people maintain this code (not to mention help new people grok it). Plus removing circular dependencies is always a win.

clayborg requested changes to this revision.Mar 23 2016, 2:49 PM
clayborg edited edge metadata.

A few things in inlined comments, but this is very close.

source/Plugins/SymbolFile/DWARF/DWARFASTParser.h
36 ↗(On Diff #51476)

Can probably do this later when there are changes to this file other than clang format only changes.

42 ↗(On Diff #51476)

Can probably do this later when there are changes to this file other than clang format only changes.

source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.cpp
1614–1622 ↗(On Diff #51476)

This can be reverted since GetDWARFParser is stying in TypeSystem

This revision now requires changes to proceed.Mar 23 2016, 2:49 PM
zturner added inline comments.Mar 23 2016, 3:11 PM
source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.cpp
1614–1622 ↗(On Diff #51476)

Ahh, you've actually alerted me to a subtle bug. CanCompleteType and CompleteType are not even implemented by DWARFASTParserClang anymore, because they were just forwards into the ClangASTImporter. So instead people are expected to call ast_parser->GetClangASTImporter().CanImport() or ast_parser->GetClangASTImporter().Import(). So the code here is actually calling the CanCompleteType method in the base class DWARFASTParser which simply returns false.

So I'll need to change this to delete the empty implementations from the base class, and call straight into the ClangASTImporter.

zturner updated this revision to Diff 51480.Mar 23 2016, 3:22 PM
zturner edited edge metadata.

Fixes issue with CanCompleteType being unimplemented. This update deletes CanCompleteType and CompleteType from DWARFASTParser base class, and users of that function now call GetClangASTImporter().CanImport() and GetClangASTImporter().CompleteType().

I will need to run the MacOSX test suite on this before I can give it the OK. I won't be able to do this today, but might be able to get to it tomorrow. If you can run the MacOSX test suite, let me know.

It's been a while since I've used my Mac. I'll fire it up and see if I can get it going, but no promises. I'll update tomorrow

Having trouble building on OSX.

ERROR:root:Unable to find swig executable: 'module' object has no attribute
'OSError'

Command /bin/sh failed with exit code 250

But "which swig" finds it just fine. It's been a long time since I've
built on OSX so I don't know if something has changed.

In any case, let me know if there's something I should be doing.

zturner updated this revision to Diff 51605.Mar 24 2016, 2:43 PM
zturner edited edge metadata.

Update patch.

zturner updated this revision to Diff 51705.Mar 25 2016, 5:52 PM

Rebased against ToT

Hi Greg, are you going to have a chance to try this out today?

This revision was automatically updated to reflect the committed changes.