[clangd] Rewrite of logic to rebuild the background index serving structures.
Previously it was rebuilding every 5s by default, which was much too frequent
in the long run - the goal was to provide an early build. There were also some
bugs. There were also some bugs, and a dedicated thread was used in production
but not tested.
- rebuilds are triggered by #TUs built, rather than time. This should scale more sensibly to fast vs slow machines.
- there are two separate indexed-TU thresholds to trigger index build: 5 TUs for the first build, 100 for subsequent rebuilds.
- rebuild is always done on the regular indexing threads, and is affected by blockUntilIdle. This means unit/lit tests run the production configuration.
- fixed a bug where we'd rebuild after attempting to load shards, even if there were no shards.
- the BackgroundIndexTests don't really test the subtleties of the rebuild policy (for determinism, we call blockUntilIdle, so rebuild-on-idle is enough to pass the tests). Instead, we expose the rebuilder as a separate class and have fine-grained tests for it.
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, jfb, llvm-commits
Differential Revision: https://reviews.llvm.org/D64291