[clangd] Allow consumption of DocIDs without overhead

[kbobyrev]

Summary

This patch allows processing DocIDs from iterator using callback so that they are not stored in a vector if actual DocIDs are not needed.

Such overhead is the case for D50337 patch: fuzzyFindLongQuery stores SymbolDocIDs but they are thrown away later, because what the index really needs is std::vector<std::pair<float, const Symbol *>> Scores; and it can be filled on-the-fly in matchSymbols.

[kbobyrev]

As discussed, this is not a very clean API for the user interaction and it's OK to leave consume() as is.

clang-tools-extra/clangd/index/dex/Iterator.h

friend llvm::raw_ostream &operator<<(llvm::raw_ostream &OS,
                                        const Iterator &Iterator) {
     return Iterator.dump(OS);
   }
 
 private:
   virtual llvm::raw_ostream &dump(llvm::raw_ostream &OS) const = 0;
 };
 
-/// Advances the iterator until it is either exhausted or the number of
-/// requested items is reached. The result contains sorted DocumentIDs.
+/// Advances given iterator until it is exhausted or the requested number of
+/// symbols is already seen while using callback on each processed DocID.
+void matchSymbols(Iterator &It,
+                  llvm::function_ref<void(const DocID &)> Callback,
+                  size_t Limit = std::numeric_limits<size_t>::max());
+
+/// Advances given iterator until it is either exhausted or the number of
+/// requested items is reached and returns all seen DocIDs.
 std::vector<DocID> consume(Iterator &It,
                            size_t Limit = std::numeric_limits<size_t>::max());
 
 /// Returns a document iterator over given PostingList.
 std::unique_ptr<Iterator> create(PostingListRef Documents);
 
 /// Returns AND Iterator which performs the intersection of the PostingLists of
 /// its children.

clang-tools-extra/clangd/index/dex/Iterator.cpp

private:
   // FIXME(kbobyrev): Would storing Children in min-heap be faster?
   std::vector<std::unique_ptr<Iterator>> Children;
 };
 
 } // end namespace
 
 std::vector<DocID> consume(Iterator &It, size_t Limit) {
   std::vector<DocID> Result;
+  matchSymbols(It, [&](const DocID &ID) { Result.push_back(ID); }, Limit);
+  return Result;
+}
+
+void matchSymbols(Iterator &It,
+                  llvm::function_ref<void(const DocID &)> Callback,
+                  size_t Limit) {
   for (size_t Retrieved = 0; !It.reachedEnd() && Retrieved < Limit;
        It.advance(), ++Retrieved)
-    Result.push_back(It.peek());
-  return Result;
+    Callback(It.peek());
 }
 
 std::unique_ptr<Iterator> create(PostingListRef Documents) {
   return llvm::make_unique<DocumentIterator>(Documents);
 }
 
 std::unique_ptr<Iterator>
 createAnd(std::vector<std::unique_ptr<Iterator>> Children) {