This is an archive of the discontinued LLVM Phabricator instance.

[clangd] Add xref for macro to static index.
ClosedPublic

Authored by usaxena95 on Nov 20 2019, 6:22 AM.

Details

Summary

This adds the references for macros to the SymbolCollector (used for static index).

After this patch:
Num refs increases by 5.7%
Disk usage increases by: 2.5%

Diff Detail

Event Timeline

usaxena95 created this revision.Nov 20 2019, 6:22 AM
Herald added a project: Restricted Project. · View Herald TranscriptNov 20 2019, 6:22 AM

Build result: pass - 60166 tests passed, 0 failed and 730 were skipped.
Log files: console-log.txt, CMakeCache.txt

hokein added inline comments.Nov 20 2019, 7:05 AM
clang-tools-extra/clangd/index/SymbolCollector.cpp
379

I think we should avoid running the code below if this is a mere reference.

532

I'm curious of the behavior getMacroDefinition -- from its implementation, if II doesn't have macro definition, it just returns empty.

could you check whether it works at the below case (or even with a same macro name defined/undef multiple times)?

#define abc 1
#undef abc

// not at the EOF, I guess, II for `abc` doesn't have macro definition at this point because it has been undef?
clang-tools-extra/clangd/index/SymbolCollector.h
76

This option is for macro symbols, I'd not use it for collecting macro references. I think the whether to collect macro references is judged by RefFilter and RefsInHeaders.

154

since we now use it for macros, the name is not valid any more, maybe rename it to SymbolRef?

usaxena95 updated this revision to Diff 231648.Dec 2 2019, 1:51 AM
usaxena95 marked 5 inline comments as done.

Addressed comments.

usaxena95 marked an inline comment as done.Dec 2 2019, 1:52 AM
usaxena95 added inline comments.
clang-tools-extra/clangd/index/SymbolCollector.cpp
379

Interesting. Just to understand it better can you give an example where the Role is just a reference ?

532

I had a FIXME in the tests for this case. I see why there was no macro definition at that point. I guess it would be better to keep the symbol id instead of the II.

clang-tools-extra/clangd/index/SymbolCollector.h
76

I see. RefFilter sounds right but I am don't think RefsInHeaders is the correct one. It just tells whether to collect references from the included header (i.e. outside mainfile) or not.
I think it would be better to have a separate flag for macro references in such case.
WDYT ?

Build result: pass - 60166 tests passed, 0 failed and 730 were skipped.

Log files: console-log.txt, CMakeCache.txt

usaxena95 marked 2 inline comments as done.Dec 2 2019, 2:47 AM
usaxena95 added inline comments.
clang-tools-extra/clangd/index/SymbolCollector.cpp
379

My bad. Got this one. The part below it is just for the symbol slab so it's fine to omit a reference.

usaxena95 updated this revision to Diff 231660.Dec 2 2019, 3:08 AM

Added correct documentation.

Build result: pass - 60166 tests passed, 0 failed and 730 were skipped.

Log files: console-log.txt, CMakeCache.txt

hokein added a comment.Dec 2 2019, 8:08 AM

btw, could you measure the increasing size of the index with this patch?

You could run

./bin/clangd-indexer -format=binary -executor=all-TUs . > static-index.idx
./bin/dexp static-index.idx
# you will see the memory usage.
clang-tools-extra/clangd/index/SymbolCollector.cpp
532

sorry, I didn't see a FIXME in the test, am I missing anything? Maybe move the FIXME to here?

clang-tools-extra/clangd/index/SymbolCollector.h
76

I don't think we need an extra flag merely for macro references. macro references should be treated in the same way of decl references.

usaxena95 updated this revision to Diff 231834.Dec 3 2019, 12:14 AM
usaxena95 marked 5 inline comments as done.

Addressed comments.
Will update the increase in size of index shortly.

clang-tools-extra/clangd/index/SymbolCollector.cpp
532

Instead of storing the II, I now store the SymbolID. So we do not need to call getMacroDefinition at the end of TU.
Therefore the problem with undef and multiple declaration of macros was resolved and I removed the FIXME too.
There is a test of the form

// Macro defined multiple times.
#define $ud1[[UD]] 1
int ud_1 = $ud1[[UD]];
#undef UD

#define $ud2[[UD]] 2
int ud_2 = $ud2[[UD]];
#undef UD
clang-tools-extra/clangd/index/SymbolCollector.h
76

Got it. Done.

Build result: pass - 60166 tests passed, 0 failed and 730 were skipped.

Log files: console-log.txt, CMakeCache.txt

Before this patch
CollectMacro is disabled by default

Loaded Dex from static-index.idx with estimated memory usage 354410880 bytes
  - number of symbols: 462463
  - number of refs: 6361763
  - numnber of relations: 20322

Before this patch
CollectMacros=true enabled in IndexerMain.cpp
Num symbols increases by 3%
Disk increase: 1.2%

Loaded Dex from no-macro-xref.idx with estimated memory usage 358596403 bytes
  - number of symbols: 476056
  - number of refs: 6361764
  - numnber of relations: 20322

After this patch
CollectMacros=true enabled in IndexerMain.cpp
Num symbols increases by 3%
Num refs increases by 5.7%
Disk increase: 3.7%

Loaded Dex from with-macro-xref.idx with estimated memory usage 367648065 bytes
  - number of symbols: 476069
  - number of refs: 6727042
  - numnber of relations: 20324
usaxena95 edited the summary of this revision. (Show Details)Dec 3 2019, 2:34 AM
hokein added a comment.Dec 3 2019, 1:33 PM

thanks, mostly good, my main concern is that the patch still relies on the CollectMacro and CollectMainFileSymbols option, see my comments below.

clang-tools-extra/clangd/index/SymbolCollector.cpp
349

again, CollectMacro just tells whether we should collect macro symbols, we use RefFilter to control whether we collect macro references. I think we'd need to move this code after the code of collecting macro refs.

362

and this as well.

472

this code should not be modified, since we don't use the CollectMacro option.

clang-tools-extra/clangd/unittests/SymbolCollectorTests.cpp
615

I think we don't need these flags if we change the symbolcollector as my other comments mention.

usaxena95 updated this revision to Diff 232078.Dec 4 2019, 3:40 AM
usaxena95 marked 4 inline comments as done.

Addressed comments.

usaxena95 updated this revision to Diff 232079.Dec 4 2019, 3:45 AM

Explicitly turn off CollectMacro in test.

#Updatin D70489: [clangd] Add xref for macro to static index.

Build result: fail - 60403 tests passed, 1 failed and 726 were skipped.

failed: LLVM.Bindings/Go/go.test

Log files: console-log.txt, CMakeCache.txt

Build result: fail - 60403 tests passed, 1 failed and 726 were skipped.

failed: LLVM.Bindings/Go/go.test

Log files: console-log.txt, CMakeCache.txt

hokein added inline comments.Dec 4 2019, 6:23 AM
clang-tools-extra/clangd/index/SymbolCollector.cpp
472

this comment seems undone yet.

clang-tools-extra/clangd/indexer/IndexerMain.cpp
45 ↗(On Diff #232079)

I think this change is unintended?

usaxena95 updated this revision to Diff 232125.Dec 4 2019, 6:47 AM
usaxena95 marked 3 inline comments as done.

Addressed comments.

usaxena95 added inline comments.Dec 4 2019, 6:48 AM
clang-tools-extra/clangd/index/SymbolCollector.cpp
472

I moved the Collecting references for Macros out of the if. This part of code just seems to be moved.
It's now moved to the original place.

usaxena95 updated this revision to Diff 232126.Dec 4 2019, 6:51 AM

Removed unintended space.

Build result: fail - 60403 tests passed, 1 failed and 726 were skipped.

failed: LLVM.Bindings/Go/go.test

Log files: console-log.txt, CMakeCache.txt

Build result: fail - 60403 tests passed, 1 failed and 726 were skipped.

failed: LLVM.Bindings/Go/go.test

Log files: console-log.txt, CMakeCache.txt

hokein accepted this revision.Dec 4 2019, 7:05 AM

thanks, looks good with a few nits. I think the benchmark data doesn't correct any more with the latest patch, we don't increase number of symbols.

clang-tools-extra/clangd/index/SymbolCollector.cpp
373

nit: symbols => macros, and other places as well.

523

nit: I'd just inline the ID here.

no {} needed for single-body for statement.

548–549

nit: remove the {}.

clang-tools-extra/clangd/unittests/SymbolCollectorTests.cpp
671

could you add a comment why we use _ here rather than the macro name?

700

nit: this variable is not used.

This revision is now accepted and ready to land.Dec 4 2019, 7:05 AM
usaxena95 updated this revision to Diff 232250.Dec 4 2019, 6:45 PM
usaxena95 marked 5 inline comments as done.

Addressed comments.

Build result: fail - 60403 tests passed, 1 failed and 726 were skipped.

failed: LLVM.Bindings/Go/go.test

Log files: console-log.txt, CMakeCache.txt

usaxena95 added a comment.EditedDec 4 2019, 7:13 PM

Updated benchmarks:
Before this patch

Loaded Dex from static-index.idx with estimated memory usage 354410880 bytes
  - number of symbols: 462463
  - number of refs: 6361763
  - numnber of relations: 20322

After this patch
Num refs increases by 5.7%
Disk increase: 2.5%

Loaded Dex from with-macro-xref.idx with estimated memory usage 363390426 bytes
  - number of symbols: 462482
  - number of refs: 6727051
  - numnber of relations: 20325
usaxena95 edited the summary of this revision. (Show Details)Dec 4 2019, 7:14 PM
usaxena95 edited the summary of this revision. (Show Details)Dec 4 2019, 7:27 PM
This revision was automatically updated to reflect the committed changes.