[clangd] Add removeFile interface in FileIndex.

[hokein]

Summary

Currently we don't cleanup dynamic index if a file is closed in clangd,
which may result in large memory consumption in clangd (if we open many files
and closes them).

This patch is the first step.

[hokein]

clang-format.

[ioeric]

clangd/index/FileIndex.h

78

should we use this somewhere? E.g. when file is closed in ClangdServer?

[hokein]

clangd/index/FileIndex.h

78

Yes, the usage of this method is not included in this patch.

We probably use this function in TUScheduler::remove, the code path will be like

TUScheduler::remove => ParsingCallback::onASTRemove => FileIndex::removeFile.

We need to add a similar interface to ParsingCallback.

[ioeric]

clangd/index/FileIndex.h

78

Or maybe in ClangdServer::removeDocument?

[hokein]

clangd/index/FileIndex.h

78

It seems that we will have problems if calling it in ClangdServer::removeDocument.

When removing a file, we schedule a remove task in TUScheduler, and the AST thread will wait for all pending tasks to be finished and then exit.

An example like:

1. The AST thread has a pending request of update AST task (which will update the FileIndex) in its task queue.
2. We remove the documentation, FileIndex will be cleanup immediately in ClangdServer::removeDocument, and clangd schedules a remove task.
3. The AST thread processes all pending tasks (including update AST), and exits, Now the symbol comes back again.

clangd/index/FileIndex.h

private:
 
   /// Stores the latest symbol snapshots for all active files.
   llvm::StringMap<std::shared_ptr<SymbolSlab>> FileToSymbols;
   /// Stores the latest ref snapshots for all active files.
   llvm::StringMap<std::shared_ptr<RefSlab>> FileToRefs;
 };
 
 /// This manages symbols from files and an in-memory index on all symbols.
-/// FIXME: Expose an interface to remove files that are closed.
 class FileIndex : public MergedIndex {
 public:
   /// If URISchemes is empty, the default schemes in SymbolCollector will be
   /// used.
   FileIndex(std::vector<std::string> URISchemes = {});
 
   /// Update preamble symbols of file \p Path with all declarations in \p AST
   /// and macros in \p PP.
   void updatePreamble(PathRef Path, ASTContext &AST,
                       std::shared_ptr<Preprocessor> PP);
 
   /// Update symbols and references from main file \p Path with
   /// `indexMainDecls`.
   void updateMain(PathRef Path, ParsedAST &AST);
 
+  /// Remove all index data associated with the file \p Path.
+  void removeFile(PathRef Path);
+
 private:
   std::vector<std::string> URISchemes;
 
   // Contains information from each file's preamble only.
   // These are large, but update fairly infrequently (preambles are stable).
   // Missing information:
   //  - symbol refs (these are always "from the main file")
   //  - definition locations in the main file

clangd/index/FileIndex.cpp

 void FileIndex::updateMain(PathRef Path, ParsedAST &AST) {
   auto Contents = indexMainDecls(AST, URISchemes);
   MainFileSymbols.update(
       Path, llvm::make_unique<SymbolSlab>(std::move(Contents.first)),
       llvm::make_unique<RefSlab>(std::move(Contents.second)));
   MainFileIndex.reset(MainFileSymbols.buildMemIndex());
 }
 
+void FileIndex::removeFile(PathRef Path) {
+  PreambleSymbols.update(Path, nullptr, nullptr);
+  PreambleIndex.reset(PreambleSymbols.buildMemIndex());
+  MainFileSymbols.update(Path, nullptr, nullptr);
+  MainFileIndex.reset(MainFileSymbols.buildMemIndex());
+}
+
 } // namespace clangd
 } // namespace clang

unittests/clangd/FileIndexTests.cpp

 #include "clang/Frontend/CompilerInvocation.h"
 #include "clang/Frontend/PCHContainerOperations.h"
 #include "clang/Index/IndexSymbol.h"
 #include "clang/Lex/Preprocessor.h"
 #include "clang/Tooling/CompilationDatabase.h"
 #include "gtest/gtest.h"
 
 using testing::_;
+using testing::IsEmpty;
 using testing::AllOf;
 using testing::ElementsAre;
 using testing::Pair;
 using testing::UnorderedElementsAre;
 
 MATCHER_P(RefRange, Range, "") {
   return std::tie(arg.Location.Start.Line, arg.Location.Start.Column,
                   arg.Location.End.Line, arg.Location.End.Column) ==
TEST(FileIndexTest, IndexAST) {
   update(M, "f1", "namespace ns { void f() {} class X {}; }");
 
   FuzzyFindRequest Req;
   Req.Query = "";
   Req.Scopes = {"ns::"};
   EXPECT_THAT(match(M, Req), UnorderedElementsAre("ns::f", "ns::X"));
 }
 
+TEST(FileIndexTest, RemoveFile) {
+  FileIndex M;
+  update(M, "f1", "class Foo {};");
+
+  FuzzyFindRequest Req;
+  EXPECT_THAT(match(M, Req), UnorderedElementsAre("Foo"));
+
+  M.removeFile("f1.cpp");
+  EXPECT_THAT(match(M, Req), IsEmpty());
+}
+
 TEST(FileIndexTest, NoLocal) {
   FileIndex M;
   update(M, "f1", "namespace ns { void f() { int local = 0; } class X {}; }");
 
   FuzzyFindRequest Req;
   Req.Query = "";
   EXPECT_THAT(match(M, Req), UnorderedElementsAre("ns", "ns::f", "ns::X"));
 }