This is an archive of the discontinued LLVM Phabricator instance.

[clangd] Implement findReferences function
ClosedPublic

Authored by sammccall on Aug 20 2018, 1:44 AM.

Details

Summary

clangd will use findReferences to provide LSP's reference feature.

Event Timeline

hokein created this revision.Aug 20 2018, 1:44 AM
hokein updated this revision to Diff 161441.Aug 20 2018, 1:50 AM

More cleanups.

This patch is incomplete, and it is a prototype based on our discussion last week. You can patch it locally, and play around with VSCode.

Mostly just reviewed the references() implementation.

For the infrastructure I have some high level questions:

  • why are we combining SymbolSlab and occurrences?
  • what's the motivation for the separate preamble/main-file indexes
  • why put more functionality into SymbolCollector?

Let's discuss offline a bit if that works?

clangd/TUScheduler.h
54 ↗(On Diff #161441)

(we're replacing a std::function, why not just a struct with two std::functions? saves boilerplate in a bunch of places...)

clangd/XRefs.cpp
662

I'm not sure unpacking the reference options into individual bools is going to scale well. I'd suggest passing the whole struct instead.

671

"local" is ambiguous here (this function cares both about file-local references and function-local symbols).
Consider ExternallyVisibleIDs

677

(This saves us hitting the index to look up references for one symbol, not sure if it's worth it at all).

I wouldn't particularly trust the helpers in clang::index::. Indeed the implementation looks wrong here (e.g. it would treat lambda captures as global, I think?)

I'd prefer D->getParentFunctionOrMethod() here.

684

I'm not sure this is the best interpretation of LSP includeDeclaration.

The LSP spec is very vague here, and we can usually assume it's written with typescript in mind :-) where the declaration/definition distinction doesn't really exist.
It appears the distinction they're trying to draw is declaration vs "only" reference, rather than definition vs "only" declaration. So I think if we're going to respect this flag, we should *exclude* definitions when the flag is false.

Alternatively we could punt on this and just ignore the flag for now, and add it in a followup.

690

Reusing symbolcollector here seems odd.

It forces us to go through SymbolID rather than just using the Decl*. This is certainly reliable for global symbols, but I've got no idea how robust USRs are for local symbols. It also ends up building the Symbol objects, which we then throw away.

How much of SymbolCollector are we really reusing, vs a custom IndexDataConsumer? How is this different from document-highlights? Maybe we can reuse the consumer.

695

This comment is surprising as it stands. Maybe

// Look in the AST for references from the current file.

(and hoist the commentto the top of the code that deals with AST, currently line 686)

709

This is also a bit confusing - the symbols are non-local (in the function-scope sense) and the references are non-local (in the file sense).
Consider:

// Consult the index for references in other files.
// We only need to consider symbols visible to other files.
718

in the usual case, SeenURIs is the current file (if there were any local usages), or empty (if there weren't).

This means if there are refs from the current file in the index but not locally (they were deleted), you'll return them when it would be better not to.

Why not just obtain the URI for the main file directly and compare that, instead of extracting it from the local references found?

clangd/index/Index.h
368 ↗(On Diff #161441)

why are we moving occurrences into the symbol slab?

ilya-biryukov added a comment.EditedAug 24 2018, 2:43 AM

Parts of this patch were landed as D50847 and D50889, happy to discuss the design decisions ("merged" dynamic index, two callbacks), but that's probably out of scope of this patch.

Parts of this patch were landed as D50847 and D50889, happy to discuss the design decisions ("merged" dynamic index, two callbacks), but that's probably out of scope of this patch.

Ah ok, that explains some of the unrelated changes.
I do have some questions there that would be good to discuss. Meanwhile, @hokein can you rebase this patch against head?

Thanks for the comments.

I do have some questions there that would be good to discuss. Meanwhile, @hokein can you rebase this patch against head?

Yes, I'd love to, but since this patch is quite large, I need split it into smaller patches. I will update this patch (focus on the references implementation), it depends on the following patches

hokein updated this revision to Diff 163284.Aug 30 2018, 2:11 AM
hokein marked 8 inline comments as done.
  • address review comments
  • rescope the patch only focus on findReferences implementation.
hokein retitled this revision from [clangd] WIP: xrefs for dynamic index. to [clangd] Implement findReferences function.Aug 30 2018, 2:15 AM
hokein edited the summary of this revision. (Show Details)
hokein added inline comments.
clangd/XRefs.cpp
662

Removed this parameter.

677

I think it is worth the effort here. At least for function-local symbols, traversing AST is sufficient to get complete xref results.

684

Ignore this flag, and always return all kinds.

690

I refined the highlight IndexDataConsumer, and now the common code is shared.

hokein updated this revision to Diff 163654.Sep 2 2018, 11:16 PM
hokein edited the summary of this revision. (Show Details)

Rebase

Looking forward to using this!

Unless you object, I'd like to address the remaining comments (and get a review), so you can make the most of your leave!

clangd/XRefs.cpp
354

why is this a map keyed by D? the keys are never used, the result is always flattened into a vector.

363

as above, why does this need to be a subclass

364

why is this a subclass rather than just using OccurrenceCollector and putting the finish() code in findReferences()?

381

out of curiosity: how do we actually get duplicates?

666

(comment just echoes code)

669

visiable -> visible

672

we can check isExternallyVisible, I think?
maybe it's not worth it, but the comment as it stands doesn't seem accurate

unittests/clangd/XRefsTests.cpp
1192

can we use a simple real implementation TU.index() instead of mocking here?
I think this is an artifact of the fact that TestTU doesn't actually expose occurrences in the index, can we fix that?

hokein added a comment.Sep 4 2018, 5:22 AM

Looking forward to using this!

Unless you object, I'd like to address the remaining comments (and get a review), so you can make the most of your leave!

Thanks, feel free to pick it up. The comments make sense to me. Just let you know I have one more xref patch: https://reviews.llvm.org/D50896.

clangd/XRefs.cpp
354

Using map can group results by the Decl, but it doesn't matter here.

381

Some AST nodes will be visited more than once, so we might have duplicated locations.

672

I'm not sure whether isExternallyVisible suits our cases. For example, if a header defines a static const int MAX, and the header is widely included in the project, we want to query it in the index, but the linkage of the symbol MAX is internal.

So I'd prefer to be conservative here (only for function-local symbols).

unittests/clangd/XRefsTests.cpp
1192

Yes, we can. But the test here is only to verify whether the index has been queried. The results from index are not important.

OK thanks, I'll steal this one.

clangd/XRefs.cpp
672

Agree, this sounds like the right place to start.

unittests/clangd/XRefsTests.cpp
1192

ah, that makes sense. But we should have some actual tests that use the index content too!

sammccall commandeered this revision.Sep 4 2018, 8:07 AM
sammccall edited reviewers, added: hokein; removed: sammccall.
sammccall updated this revision to Diff 163834.Sep 4 2018, 8:24 AM

Address comments and tighten up highlights/refs reuse in XRefs.cpp a bit.

Still to do:

  • test interaction with index, including actual data
  • maybe add the ClangdServer/LSP binding and lit test, if it's just boilerplate

Oops, and I rebased on head (Occurrences -> Refs) of course.

@ilya-biryukov @ioeric, any interest in taking over review here?

hokein accepted this revision.Sep 4 2018, 11:55 AM

Thanks, looks good from my side.

clangd/XRefs.cpp
328

nit: we can initialize TargetDecls like Targets(TargetDecls.begin(), TargetDecls.end()).

676

nit: use const auto & indicating the variable is immutable.

This revision is now accepted and ready to land.Sep 4 2018, 11:55 AM
sammccall updated this revision to Diff 164004.Sep 5 2018, 3:30 AM

Finish tests, fix TestTU::index() to include occurrences.

sammccall added inline comments.
clangd/XRefs.cpp
328

In fact SmallSet is missing this constructor :-(

sammccall closed this revision.Sep 5 2018, 3:59 AM

Oops, I forgot to associate my patch with this review. It landed as rL341458.