[clangd] Use xxh3_64bits for background index file digests

[MaskRay]

Summary

Many sources show that xxh3 is much better than xxh64. This particular
instance may or may not have noticeable difference, but this change
moves us toward removing xxHash64.

[sammccall]

These hashes are used in the filenames of index shards, and in the metadata describing dependencies between shards.

So this is an incompatible change to indexing format, you also need to increment constexpr static uint32_t Version in Serialization.cpp:460.

(I think this is fine, but not stamping just yet as I want to sync with Kadir on monday - I don't think we have anything that GCs the old shards but probably that shouldn't block this change)

[MaskRay]

bump version

[kadircet]

thanks for doing this!

as Sam pointed out this will result in invalidation of all the index shards, but that's not something new. we already don't clean up non-relevant index shards when people delete sources over time and rely on people having new checkouts or clearing things up manually. this will be somewhat more severe, as we'll double the index size all of a sudden.

but i'd like to keep this patch from landing until llvm-17 is cut (which should be tomorrow). we didn't have any changes to our index shard serialization since llvm-16 release, so forcing everyone to face this invalidation purely for the sake of a clean up that might never happen in the codebase doesn't feel enough of a justification. whereas it's more likely that we'll have changes to serialization format throughout the next release cycle and can hit users only once with big cache invalidations.

[sammccall]

Agree - LGTM but please land once the branch release/17.x exists. Thanks!

[MaskRay]

In D155992#4527081, @kadircet wrote:

thanks for doing this!

as Sam pointed out this will result in invalidation of all the index shards, but that's not something new. we already don't clean up non-relevant index shards when people delete sources over time and rely on people having new checkouts or clearing things up manually. this will be somewhat more severe, as we'll double the index size all of a sudden.

but i'd like to keep this patch from landing until llvm-17 is cut (which should be tomorrow). we didn't have any changes to our index shard serialization since llvm-16 release, so forcing everyone to face this invalidation purely for the sake of a clean up that might never happen in the codebase doesn't feel enough of a justification. whereas it's more likely that we'll have changes to serialization format throughout the next release cycle and can hit users only once with big cache invalidations.

Sounds good. I'll wait after the branch (today) :)

clang-tools-extra/clangd/SourceCode.cpp

TextEdit toTextEdit(const FixItHint &FixIt, const SourceManager &M,
   TextEdit Result;
   Result.range =
       halfOpenToRange(M, Lexer::makeFileCharRange(FixIt.RemoveRange, M, L));
   Result.newText = FixIt.CodeToInsert;
   return Result;
 }
 
 FileDigest digest(llvm::StringRef Content) {
-  uint64_t Hash{llvm::xxHash64(Content)};
+  uint64_t Hash{llvm::xxh3_64bits(Content)};
   FileDigest Result;
   for (unsigned I = 0; I < Result.size(); ++I) {
     Result[I] = uint8_t(Hash);
     Hash >>= 8;
   }
   return Result;
 }
 

clang-tools-extra/clangd/index/Background.cpp

 
 static llvm::StringRef filenameWithoutExtension(llvm::StringRef Path) {
   Path = llvm::sys::path::filename(Path);
   return Path.drop_back(llvm::sys::path::extension(Path).size());
 }
 
 BackgroundQueue::Task BackgroundIndex::indexFileTask(std::string Path) {
   std::string Tag = filenameWithoutExtension(Path).str();
-  uint64_t Key = llvm::xxHash64(Path);
+  uint64_t Key = llvm::xxh3_64bits(Path);
   BackgroundQueue::Task T([this, Path(std::move(Path))] {
     std::optional<WithContext> WithProvidedContext;
     if (ContextProvider)
       WithProvidedContext.emplace(ContextProvider(Path));
     auto Cmd = CDB.getCompileCommand(Path);
     if (!Cmd)
       return;
     if (auto Error = index(std::move(*Cmd)))

clang-tools-extra/clangd/index/Serialization.cpp

 //   - srcs: information related to include graph
 //   - stri: string table
 //   - symb: symbols
 //   - refs: references to symbols
 
 // The current versioning scheme is simple - non-current versions are rejected.
 // If you make a breaking change, bump this version number to invalidate stored
 // data. Later we may want to support some backward compatibility.
-constexpr static uint32_t Version = 18;
+constexpr static uint32_t Version = 19;
 
 llvm::Expected<IndexFileIn> readRIFF(llvm::StringRef Data,
                                      SymbolOrigin Origin) {
   auto RIFF = riff::readFile(Data);
   if (!RIFF)
     return RIFF.takeError();
   if (RIFF->Type != riff::fourCC("CdIx"))
     return error("wrong RIFF filetype: {0}", riff::fourCCStr(RIFF->Type));