[clangd] Make sure project-aware index is up-to-date for estimateMemoryUsage()

[sammccall]

Why do we want to do this?
The existing estimate covers the existing loaded data.
This change prompts it to load data that we think we might need soon before estimating, but I'm not sure why that's a good thing.

(actually I don't think it'll work as if it causes new indexes to be loaded, they'll load asynchronously and report 0 here)

[kbobyrev]

Hm, yes, this makes sense. I just found it awkward that memory usage estimate will spike right after the first request but that's actually the reality (loading the index is deferred until the first actual index request). My confusion here is that I don't really understand whether deferring the index loading is a good thing because we're practically moving the delay from the warm-up to the working state which is suboptimal. I was trying to cause the index to load in the initialization stage for D92198 but couldn't figure out where exactly the config is loaded and what could be the best place for triggering index loading.

[sammccall]

In D92201#2419933, @kbobyrev wrote:

Hm, yes, this makes sense. I just found it awkward that memory usage estimate will spike right after the first request but that's actually the reality (loading the index is deferred until the first actual index request).

Exactly. The good news is: I don't think we have any particular use case for probing the memory usage right after startup.

My confusion here is that I don't really understand whether deferring the index loading is a good thing because we're practically moving the delay from the warm-up to the working state which is suboptimal.

It's not great that we offer partial functionality without signaling this somehow, and we could consider making this more transparent.
However this happens in lots of places in clangd, especially w.r.t index (completion before preamble, background index async load, actual background indexing itself, the cached-build-artifacts we have internally at google) and in every case feedback has been positive when we introduced it. Blocking the user for any one feature is rarely a good trade, if we can recover even half-gracefully instead.

I was trying to cause the index to load in the initialization stage for D92198 but couldn't figure out where exactly the config is loaded and what could be the best place for triggering index loading.

Fair point, will have a look at that. Forcing an index load might be a nice side-effect of this patch, but I think we can find a better way to do it.

[kadircet]

In D92201#2421520, @sammccall wrote:

In D92201#2419933, @kbobyrev wrote:

I was trying to cause the index to load in the initialization stage for D92198 but couldn't figure out where exactly the config is loaded and what could be the best place for triggering index loading.

Fair point, will have a look at that. Forcing an index load might be a nice side-effect of this patch, but I think we can find a better way to do it.

I suppose the most natural place would be under ClangdServer::addDocument, but we don't have any nice way to trigger this warmup conceptually. All we can do is perform some other index operation that'll trigger index loading as a side-effect, which makes it quite weird to trigger in any "direct" place :/

clang-tools-extra/clangd/index/ProjectAware.cpp

mutable llvm::DenseMap<Config::ExternalIndexSpec,
                          std::unique_ptr<SymbolIndex>>
       IndexForSpec;
   mutable AsyncTaskRunner Tasks;
 
   const IndexFactory Gen;
 };
 
 size_t ProjectAwareIndex::estimateMemoryUsage() const {
+  // Ensure IndexForSpec is up-to-date.
+  getIndex(); // Unused.
   size_t Total = 0;
   std::lock_guard<std::mutex> Lock(Mu);
   for (auto &Entry : IndexForSpec)
     Total += Entry.second->estimateMemoryUsage();
   return Total;
 }
 
 void ProjectAwareIndex::lookup(