This is an archive of the discontinued LLVM Phabricator instance.

Give the SymbolFile plugin enough information to efficiently do exact match lookups
AcceptedPublic

Authored by zturner on Oct 24 2018, 11:55 AM.

Details

Summary

Previously the Module would parse a type name into scope and base name during a lookup and only give the SymbolFile plugin the base name, then it would filter the results down into the match set based on whether it was supposed to be an exact match.

However, if this is supposed to be an exact match, a SymbolFile can do this lookup in O(1) time by using its internal accelerator tables, but it can't do this unless it actually receives the full name as well as the information that it is supposed to be an exact match. Rather than being too smart, we should just let the symbol file plugin be the arbiter to decide how it wants to do lookups, so we should pass it the full name.

I'm trying to keep this as NFC for the DWARF plugin, so I took the code that was in Module.cpp and tried to rewrite it to be equivalent at the top of SymbolFileDWARF::FindTypes

Diff Detail

Event Timeline

zturner created this revision.Oct 24 2018, 11:55 AM

Note that AFAICT, native PDB plugin is now the only plugin where you can do type lookup -- NS::Struct::Local and have it return instantly. On the regular PDB plugin it doesn't work at all (returns no results). On the DWARF plugin, I haven't tested, but it will either not work at all, or take a potentially long time if you have a lot of debug info).

I worry that your patch changes the behavior when you add the type_class explicitly in the lookup (i.e. look up "struct Struct" not "Struct". That should work...

Note, this doesn't currently work in type lookup:

(lldb) type lookup "struct Foo"
no type was found matching '"struct Foo"'

which I'll have to fix (grr...), but does work correctly in the SB API's:

(lldb) script
Python Interactive Interpreter. To exit, type 'quit()', 'exit()' or Ctrl-D.
>>> module = lldb.target.modules[0]
>>> print module
(x86_64) /tmp/structs
>>> result_list = module.FindTypes("struct Foo")
>>> print result_list.GetSize()
1
>>> print result_list.GetTypeAtIndex(0)
struct Foo {
    int First;
}

Other than that this looks correct to me.

lldb/source/Core/Module.cpp
1038

GetTypeScopeAndBasename's behavior is not well documented. It has a bool return which should mean some kind of failure. The code you are replacing checks for type_basename.empty() and the type_class not being set, and falls back on the input name if it is. You unconditionally use type_basename.

The old code's test doesn't actually accord with the current behavior of GetTypeScopeAndBasename, which will only leave basename empty if the input name was empty, so far as I can see. So I don't think your version is wrong, but it is right only by accident. If we are going to rely on this behavior then it's probably a good idea to document it in the definition of GetTypeScopeAndBasename. Or go back to checking whether type_basename is empty...

Also, by not calling GetTypeScopeAndBasename before you call FindTypes_Impl, your version of the code would pass it "struct Foo" if that's what the user typed, whereas the old code would pass "Foo" (the struct would get stripped by GetTypeScopeAndBasename). I'm not sure whether that matters, did you try 'type lookup "struct Struct"' or something like that? It doesn't look like you do that in your test cases.

Also your "exact" would call "struct ::Foo" not exact, whereas the old code would call that exact.

I worry that your patch changes the behavior when you add the type_class explicitly in the lookup (i.e. look up "struct Struct" not "Struct". That should work...

Note, this doesn't currently work in type lookup:

(lldb) type lookup "struct Foo"
no type was found matching '"struct Foo"'

Ahh, nice catch. Luckily it should be easy to add a couple of tests for those alongside the other ones I added, so I'll try that and upload a new version.

zturner updated this revision to Diff 171134.Oct 25 2018, 10:22 AM

Fixed issues pointed out by @jingham and added some test coverage for this.

jingham accepted this revision.Oct 25 2018, 10:56 AM

This looks good to me.

Looking at the addition of Type::ConsumeTypeClass makes it really clear that both this function and Type::GetTypeScopeAndBasename need to dispatch to the CompilerSystem to do this work. The actual code is way too C-family-ish to be in the generic Type.cpp. That will require passing in a language because I don't think you know enough from just the symfile and name to know which language the user was looking up types for.

But this change doesn't make it harder to get that right, so fixing it doesn't need to be part of this patch.

So far as I can tell, you can't do an O(1) lookup of an exact name in DWARF from the dwarf_names table (or the proceeding apple tables). The tables are organized by base name (or really whatever in the DW_AT_name attribute of the die). So you will always have to pull the base name out, find all the base name matches and then run up the parent dies to build up the fully qualified name. Each name entry has the parent DIE, so building up the full name is pretty efficient. But still, the best you can do O(number_of_occurances_of_name) and knowing the name is exact doesn't help the search speed. If I'm right about that (Greg will surely know) then we should remove the FIXME comment in SymbolFileDWARF since it refers to an unfixable fix.

This revision is now accepted and ready to land.Oct 25 2018, 10:56 AM
clayborg added a comment.EditedOct 25 2018, 11:09 AM

The current SymbolFile::FindTypes(...) in was designed with type base name only due to how DWARF stores it type information. It has a "const CompilerDeclContext *parent_decl_ctx" which can be specified in order to limit what we find. So we might be able to think of this as a type lookup by basename. It might be handy to add another type lookup that does lookups by fully qualified name only. and leave the current infrastructure alone? A default implementation can be added that returns false for all symbol file unless they support name lookup.

The current implementation allows you to under specify a type. For code like:

struct Pt { char x; char y; };
namespace a {
  struct Pt { short x; short y; };
  namespace b {
    struct Pt { int x; int y; };
  }
}

We can find all of them using:

(lldb) type lookup Pt
struct Pt {
    short x;
    short y;
}
struct Pt {
    int x;
    int y;
}
struct Pt {
    char x;
    char y;
}

Or each one individually using:

(lldb) type lookup a::b::Pt
struct Pt {
    int x;
    int y;
}
(lldb) type lookup a::Pt
struct Pt {
    short x;
    short y;
}
(lldb) type lookup ::Pt
struct Pt {
    char x;
    char y;
}

Or under specify the namespace:

(lldb) type lookup b::Pt
struct Pt {
    int x;
    int y;
}

That is the current behavior. Not saying it is right. Note we also don't display the decl context for a type when dumping it which would be nice to fix. So we probably need to make sure there are tests for all of the cases above once we hone in on the approach we want all symbol plug-ins to use.

lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.cpp
2443

There is no O(1) lookup for types in DWARF. Accelerator tables do not have fully qualified type names, only the identifier for the type basename only.

So the current approach relies on the ability to sniff the name to determine the context in which the user intends to find it. It does (and always did) use the presence of an initial "::" to tell whether a symbol is exact. That's obviously also inappropriate for a generic Type method. But OTOH, there needs to be a funnel point where you take in a string you know nothing about (from the user either in type lookup or in an expression) and find it as best you can. I don't think we want to force command line users to say "type lookup --exact " so we've got to figure it out internally.

Internally, it might be helpful to do some initial analysis of the input type string, and then dispatch it to an exact or inexact search. But given I don't think you can get away without a FindTypes that takes a string that you don't know whether it is exact or not, I don't feel strongly about that.

We should abstract "is exact" and ask the various type systems to handle that request, and we also need to abstract "remove type class" and again ask the various type systems to handle that. That seems to me the main ugliness that this patch reveals.

So the current approach relies on the ability to sniff the name to determine the context in which the user intends to find it. It does (and always did) use the presence of an initial "::" to tell whether a symbol is exact. That's obviously also inappropriate for a generic Type method. But OTOH, there needs to be a funnel point where you take in a string you know nothing about (from the user either in type lookup or in an expression) and find it as best you can. I don't think we want to force command line users to say "type lookup --exact " so we've got to figure it out internally.

Internally, it might be helpful to do some initial analysis of the input type string, and then dispatch it to an exact or inexact search. But given I don't think you can get away without a FindTypes that takes a string that you don't know whether it is exact or not, I don't feel strongly about that.

We should abstract "is exact" and ask the various type systems to handle that request, and we also need to abstract "remove type class" and again ask the various type systems to handle that. That seems to me the main ugliness that this patch reveals.

Just to clarify, is the consensus then that I can submit this as is? Or that it needs some modification? Greg's suggestion of making a separate method called FindExactType could work, but it doesn't seem that much different than passing a boolean call exact_match, which is what I've done here. On the extreme end, we could just make Module::FindTypes do absolutely nothing except call the SymbolFile plugin. I don't feel too strongly either way. The current patch seems NFC for DWARF and strictly better for PDB, but I'm willing to make changes if anyone feels like there's an architecturally more sound approach.

jingham added a comment.EditedOct 25 2018, 2:59 PM

It seemed to me like Greg thought you were changing the behavior of lookups, which this patch doesn't do, it just makes it more efficient. I don't know if that alters his objections or not.

The Module and higher layer of FindTypes calls are awkward. For instance Module::FindTypes takes an "exact_match" parameter, but it doesn't actually mean "exact match". If exact_match is passed in as true, then it does mean exactly match the name. But if it is false it means "check if this name is fully qualified and if so make it an exact match otherwise do a partial match".

The header comment for the function is wrong, too. It says:

/// @param[in] exact_match
///     If \b true, \a type_name is fully qualified and must match
///     exactly. If \b false, \a type_name is a partially qualified
///     name where the leading namespaces or classes can be
///     omitted to make finding types that a user may type
///     easier.

So we should change the variable name (or even just the docs) at this level to better reflect what the parameter actually does. You still need to turn off the guessing because we don't have a 100% certain way to tell whether a type name is fully qualified, and the alternative of prepending a "::" to force the guessing algorithm's hand is gross (and very C++ specific). But with that proviso, I don't think passing this as a parameter is any better/worse than having Module:FindTypesExact and Module::FindTypesAutoDetect or some better name that I can't think up right now.

But I don't think the logical third option: "FindTypesPartial" that forces a partial match is useful. I can't see anybody typing "::foo" and wanting that to match "bar::foo". So expressing this as a bool parameter seems fine to me.

OTOH, I don't think it is desirable to have different versions of the Symfile and lower calls for Exact and not Exact, passing the exact_match parameter through there is fine IMO. After all, in all these calls exact_match means exactly what it says, you are requiring an exact match. So you don't gain clarity by having different function names.

BTW, you should remove the O(1) FIXME comment so you don't send some future eager contributor on a wild goose chase to figure out how to do this thing you can't do in DWARF.

My issue with adding "exact_match" is it renders a few of the arguments useless:

size_t FindTypes(
  const SymbolContext &sc, 
  llvm::StringRef name,
  const CompilerDeclContext *parent_decl_ctx, 
  bool exact_match,
  ...);

With exact_match "parent_decl_ctx" is useless and possible the symbol context too. The FindTypes as written today was used for two cases which evolved over time:

  • finding types for users without needing to be fully qualified
  • finding types for the expression parser in a specific context

To fulfill these two cases, arguments have been added over time. As we keep adding arguments we make the function even harder to implement for SymbolFile subclasses. So as long as we are cleaning things up, it might be time to factor this out by making changes now.

Another complexity is that we can either search a single module or multiple modules when doing searches, and that is why we have Module::FindTypes_Impl(...) in since there is some work that needs to be done when we are searching by basename only (strip the context and search by basename, then filters matches afterward.

That is why I was thinking we might want to make changes. Seems like having:

size_t FindTypesByBasename(
  const SymbolContext &sc, 
  llvm::StringRef name,
  const CompilerDeclContext *parent_decl_ctx
  ...);

size_t FindTypesByFullname(llvm::StringRef name, ...);

Clients of the first call must strip a typename to its identifier name prior to calling, and clients of the second can call with a fully qualified typename.

Ok, that reasoning makes sense, I’ll see what i can do

So I started looking into this, and things get tricky. If we're doing a lookup by fully qualified name, I would expect there to never be more than one match, but LLDB doesn't seem to hold this same assumption. For example, in ItaniumABILanguageRuntime::GetTypeInfoFromVTableAddress there is code that tries to get an exact match from a single module, and if it doesn't find one, then searches the entire module list for multiple matches on exact name, then specifically handles the case where more than one match was found. Most of this code is all just for the purposes of logging, but if we cut through all the logging and just get down to the code, it basically amounts to this:

  type_info.SetName(class_name);
  TypeList class_types;

  uint32_t num_matches = 0;
  // First look in the module that the vtable symbol came from and
  // look for a single exact match.
  llvm::DenseSet<SymbolFile *> searched_symbol_files;
  if (sc.module_sp) {
    num_modules = sc.module_sp->FindTypes(sc, ConstString(lookup_name),
      exact_match, 1, searched_symbol_files, class_types);
  }

  // If we didn't find a symbol, then move on to the entire module
  // list in the target and get as many unique matches as possible
  if (num_matches == 0) {
    num_matches = target.GetImages().FindTypes(
        sc, ConstString(lookup_name), exact_match, UINT32_MAX,
        searched_symbol_files, class_types);
  }

  lldb::TypeSP type_sp;
  if (num_matches == 0)
    return TypeAndOrName();
  if (num_matches == 1) {
    type_sp = class_types.GetTypeAtIndex(0);
    if (type_sp) {
      if (ClangASTContext::IsCXXClassType(
              type_sp->GetForwardCompilerType())) {
        type_info.SetTypeSP(type_sp);
    }
  } else if (num_matches > 1) {
    size_t i;

    for (i = 0; i < num_matches; i++) {
      type_sp = class_types.GetTypeAtIndex(i);
      if (type_sp) {
        if (ClangASTContext::IsCXXClassType(
                type_sp->GetForwardCompilerType())) {
          type_info.SetTypeSP(type_sp);
        }
      }
    }

  }
  if (type_info)
    SetDynamicTypeInfo(vtable_addr, type_info);
  return type_info;
}

Why would we ever get more than one type on a fully qualifed exact match name lookup? And even if we did, why would we care which one we chose?

Exact matches aren't a C++ specific thing. An exact match is useful for mixed C/C++ since you might want to say Foo and not Bar::Foo. IIRC a couple of the places where exact was dialed up explicitly in FindTypes were for C types. But since C and ObjC allow multiple types with the same name, that's one way you might see multiple matches for "exact match". Moreover C+ used to be fuzzy about whether non-exported classes from different linkage uses had to follow the ODR. I haven't followed whether this got nailed down but it still seems Quixotic to expect you could enforce this, since you are asking two totally unrelated library vendors to avoid each other's names for private classes... So having several classes with the same name is still a possibility IRL with C++.

Why we care is that if somebody runs the command:

(lldb) expression (::SomeClass *) 0x12345

We're going to try our best to make that work. If SomeClass is not visible in the module/comp_unit/function in which we are currently stopped, then we will go looking far and wide for SomeClass. If we find one result somewhere out in the world, the expression will succeed, but if we find two results which are different, then we want to give an error about ambiguous type.

Exact matches aren't a C++ specific thing. An exact match is useful for mixed C/C++ since you might want to say Foo and not Bar::Foo. IIRC a couple of the places where exact was dialed up explicitly in FindTypes were for C types. But since C and ObjC allow multiple types with the same name, that's one way you might see multiple matches for "exact match". Moreover C+ used to be fuzzy about whether non-exported classes from different linkage uses had to follow the ODR. I haven't followed whether this got nailed down but it still seems Quixotic to expect you could enforce this, since you are asking two totally unrelated library vendors to avoid each other's names for private classes... So having several classes with the same name is still a possibility IRL with C++.

Why we care is that if somebody runs the command:

(lldb) expression (::SomeClass *) 0x12345

We're going to try our best to make that work. If SomeClass is not visible in the module/comp_unit/function in which we are currently stopped, then we will go looking far and wide for SomeClass. If we find one result somewhere out in the world, the expression will succeed, but if we find two results which are different, then we want to give an error about ambiguous type.

So when designing this new FindTypesByFullyQualifiedName API, is it safe to assume, for the purposes of the API signature, that at the Module level, it will either return 0 or 1 match (e.g. the return value shoudl be a TypeSP, but at the ModuleList level, it can return arbitrarily many (so the return value should be a TypeList?

That depends on the definition of "fully qualified name". If you can ensure that it means "name of C++ class" - or other ODR enforcing type system, then you could make that assumption. In C you are free to redefine types on a per-function basis if you so desire; and sadly some interface generators (including MIG the one for generating Mach message handlers) do just this...) So you would need to see a way to restrict the inputs to this function. That doesn't seem like it would be straightforward to me.

You could also get multiple matches if you had classes in anonymous namespaces with the same name in multiple compile units.

That depends on the definition of "fully qualified name". If you can ensure that it means "name of C++ class" - or other ODR enforcing type system, then you could make that assumption. In C you are free to redefine types on a per-function basis if you so desire; and sadly some interface generators (including MIG the one for generating Mach message handlers) do just this...) So you would need to see a way to restrict the inputs to this function. That doesn't seem like it would be straightforward to me.

After I've changed up the codebase there is only one usage of it that I'm not sure is used in an ODR-enforcing context, which is ObjCLanguageRuntime::LookupInCompleteClassCache. All other usages are C++ or Java specific. Anyway, I guess I'll do it that way for now and upload the patch. This already turned out to be more invasvive than I was hoping for, so if this doesn't work out or someone doesn't like the approach, maybe I can just go back to the original patch and submit that

Yes, that usage is exactly the sort of use I was talking about. When we get an ObjC object and want to find its dynamic type, we read the type name by following the object's ISA pointer to the Class object. Then we go to look up the type from that name. We know we want an exact match to the name we found, but there's nothing that says the same module can't have an C++ class with the same name as an ObjC class. So that code needs to get all the types found, and sort through them for the one that is an ObjC interface type, and discard all the others. You will need to support that behavior somehow.

Yes, that usage is exactly the sort of use I was talking about. When we get an ObjC object and want to find its dynamic type, we read the type name by following the object's ISA pointer to the Class object. Then we go to look up the type from that name. We know we want an exact match to the name we found, but there's nothing that says the same module can't have an C++ class with the same name as an ObjC class. So that code needs to get all the types found, and sort through them for the one that is an ObjC interface type, and discard all the others. You will need to support that behavior somehow.

So just to make sure I understand, The C++ and ObjC classes are both potentially in the same Module, both potentially with the same name, correct?

Yes. You can't put them in the same CompileUnit because declaring variables of the type would be ambiguous (though all other references would be okay since ObjC method dispatch looks totally different from C++ method calling). But there's nothing keeping the same module from having a C++ and an ObjC class of the same name.

labath resigned from this revision.Aug 9 2019, 2:03 AM