[clangd] Extend SymbolOrigin, stop serializing it

[sammccall]

Summary

New values:

• Split Dynamic into Open/Preamble
• Add Background (previously was just Unknown)
• Soon: stdlib index

This requires extending to 16 bits, which fits within the padding of Symbol.
Unfortunately we're also *serializing* SymbolOrigin as a fixed 8 bits.

Stop serializing SymbolOrigin:

• conceptually, the source is whoever indexes or *deserializes* a symbol
• deserialization takes SymbolOrigin as a parameter and stamps it on each sym
• this is a breaking format change

[kadircet]

I think we also need to update index/remote/Server.cpp && FileSymbols (and FileIndex too).

Regarding updates to loadIndex, I actually think it makes sense for that index the always retrieve symbols as Static origin, then whoever makes use of that (we always have either a FileSymbol or a complete remote-index marshaling on top) should overwrite the origin while either building the final merged index or during marshaling. That would get rid of lots of extra plumbing.
WDYT?

clang-tools-extra/clangd/index/SymbolOrigin.cpp

17

s/P/9 && s/6/P

clang-tools-extra/clangd/index/SymbolOrigin.h

26

do we think this still provides a meaningful signal ?
It only provides value only when multiple indices of same type was involved in providing the result (and only that single type of indices). I suppose after this patch it can only happen for SM (as there are certain remote-index implementations that mark themselves as static) or RM (well this is not possible today, but some day we might have a stack of remote-indices).
As soon as there's a different type of index is involved M no longer has any meanings, as it's clear that there was some sort of merge happening (or am I missing something obvious?)

---

Before this patch situation is a little bit different since FileIndex just said D for both main file and preamble, but that's changing.

30

thanks for remembering that! :D

[sammccall]

In D115243#3176741, @kadircet wrote:

I think we also need to update index/remote/Server.cpp && FileSymbols (and FileIndex too).

Regarding updates to loadIndex, I actually think it makes sense for that index the always retrieve symbols as Static origin, then whoever makes use of that (we always have either a FileSymbol or a complete remote-index marshaling on top) should overwrite the origin while either building the final merged index or during marshaling. That would get rid of lots of extra plumbing.
WDYT?

The problem is these methods yield SymbolSlabs, and the symbols within them are frozen/const. There's no provision to "just tweak some bitfield" - we'd have to copy the whole slab.
In general the const-ness is useful safety: it ensures strings are interned and point to the slab's storage. But it's annoying here.
(The amount of extra plumbing seemed modest enough that I didn't try very hard to hack around this, there may be a neat solution).

clang-tools-extra/clangd/index/SymbolOrigin.cpp

17

Whoops, good catch!

clang-tools-extra/clangd/index/SymbolOrigin.h

26

Hmm, we can also get M when merging happens at indexing time due to multiple visits of the same symbol (DuplicateHandling::Merge, which is used by background indexing).

I don't have a strong opinion on its usefulness though. IIRC we added it (and origins) when trying to understand the paths various symbols took through indexes. As a debugging tool some redundancy maybe has some sanity-check value, but we could also drop it.

I don't think it has so much to do with this patch (unless you think we should reuse the bit instead of expanding to 16)...

Before this patch situation is a little bit different since FileIndex just said D for both main file and preamble

...it makes sense that this could have been a reason for the flag, but I don't think it actually was :-)

[sammccall]

fix sigils

[sammccall]

Update remote index server and FileIndex

[sammccall]

I think we also need to update index/remote/Server.cpp && FileSymbols (and FileIndex too).

Done. I wonder why premerge tests didn't fail for FileIndex not being updated?

[kadircet]

Thanks, LGTM!

---

The problem is these methods yield SymbolSlabs, and the symbols within them are frozen/const. There's no provision to "just tweak some bitfield" - we'd have to copy the whole slab.

Yes and no. As mentioned SymbolSlabs are frozen once created and I totally agree with the reasons there. But AFAICT in none of the indexes (apart from monolithic FileIndex) we ever have a single SymbolSlab.
We always have multiple of those and periodically build a single slab out of them, by copying/merging each symbol so that we can build a SymbolIndex on top of the final merged SymbolSlab. I was suggesting to perform ultimate Origin tweaking there (we can even respect the Origins of each individual slabs too).

Couple reasons:

• Definitely less plumbing, we just let FileSymbol now what it should be called during construction. Background/RemoteIndex already has this knowledge hard coded.
• In theory those indexes that operate on top of a multiple SymbolSlabs should be inserting their own "origin" into the final merged slab as it contains some extra logic, just like MergeIndex.

Downside is the possible discrepancy between the invocation of SymbolCollector and the serving of the SymbolSlab acquired from it.
The discrepancy is only possible when we serialize/deserialize the slab between collection and serving, I'd say in those cases the symbol origin should just be Static (and the server should add any extra semantics).

No matter where we do the "overriding" this discrepancy will become an issue today anyways, the consumer of the serialized slabs needs to overwrite them.
I just feel like keeping that semantic to FileSymbol/Background/RemoteIndex isolates the logic better.

---

I don't feel too strongly about it though, feel free to roll with this version if you don't think that isolation is worth the changes.

clang-tools-extra/clangd/index/SymbolOrigin.h

26

I don't think it has so much to do with this patch (unless you think we should reuse the bit instead of expanding to 16)...

Nope it isn't directly related to this patch (also, we soon plan to introduce the stdlib kind here, so doesn't really let us get back to 8).
Mostly checking if we can get rid of it and some code governing it.

Hmm, we can also get M when merging happens at indexing time due to multiple visits of the same symbol (DuplicateHandling::Merge, which is used by background indexing).
I don't have a strong opinion on its usefulness though. IIRC we added it (and origins) when trying to understand the paths various symbols took through indexes. As a debugging tool some redundancy maybe has some sanity-check value, but we could also drop it.

I see, yeah I am not sure about its usefulness either, sounds like there might still be rare cases this provides value (like debugging some behaviour in background index).
Thanks for the thoughts, definitely no action needed here.

[sammccall]

In D115243#3185393, @kadircet wrote:

Yes and no. As mentioned SymbolSlabs are frozen once created and I totally agree with the reasons there. But AFAICT in none of the indexes (apart from monolithic FileIndex) we ever have a single SymbolSlab.
We always have multiple of those and periodically build a single slab out of them, by copying/merging each symbol so that we can build a SymbolIndex on top of the final merged SymbolSlab. I was suggesting to perform ultimate Origin tweaking there (we can even respect the Origins of each individual slabs too).

Discussed this offline a bit.

We could pass around pairs of (SymbolSlab, Origin) and make it the index's responsibility to set the origin on each symbol. But I don't think this communicates so clearly that the origin reflects where the symbol came from/passed through, rather than what's serving it. (Given that Symbol::Origin exists in the slabs too).

• Definitely less plumbing, we just let FileSymbol now what it should be called during construction. Background/RemoteIndex already has this knowledge hard coded.

Origin for an index isn't monolithic. Even a slab can contain symbols that are the result of merging, but FileIndex will soon have symbols from stdlib index + preamble index too.
Given this, I'm not sure it's much less plumbing - it's not just passing a constructor parameter to the index.

• In theory those indexes that operate on top of a multiple SymbolSlabs should be inserting their own "origin" into the final merged slab as it contains some extra logic, just like MergeIndex.

This makes sense to me if it's useful, but I think this is *additional* to the origin reflecting the source of the slabs.

[sammccall]

Rebase

[sammccall]

(Our plan is to land this after new year some time, so that remote index servers don't get disrupted by format changes while people are on holiday)

clang-tools-extra/clangd/index/BackgroundIndexStorage.cpp

public:
 
   std::unique_ptr<IndexFileIn>
   loadShard(llvm::StringRef ShardIdentifier) const override {
     const std::string ShardPath =
         getShardPathFromFilePath(DiskShardRoot, ShardIdentifier);
     auto Buffer = llvm::MemoryBuffer::getFile(ShardPath);
     if (!Buffer)
       return nullptr;
-    if (auto I = readIndexFile(Buffer->get()->getBuffer()))
+    if (auto I =
+            readIndexFile(Buffer->get()->getBuffer(), SymbolOrigin::Background))
       return std::make_unique<IndexFileIn>(std::move(*I));
     else
       elog("Error while reading shard {0}: {1}", ShardIdentifier,
            I.takeError());
     return nullptr;
   }
 
   llvm::Error storeShard(llvm::StringRef ShardIdentifier,

clang-tools-extra/clangd/index/FileIndex.cpp

 #include "index/SymbolOrigin.h"
 #include "index/dex/Dex.h"
 #include "support/Logger.h"
 #include "support/MemoryTree.h"
 #include "support/Path.h"
 #include "clang/AST/ASTContext.h"
 #include "clang/Index/IndexingAction.h"
 #include "clang/Index/IndexingOptions.h"
-#include "clang/Lex/MacroInfo.h"
 #include "clang/Lex/Preprocessor.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/StringMap.h"
 #include "llvm/ADT/StringRef.h"
-#include "llvm/Support/Error.h"
 #include <algorithm>
 #include <memory>
 #include <tuple>
 #include <utility>
 #include <vector>
 
 namespace clang {
 namespace clangd {
 namespace {
 
 SlabTuple indexSymbols(ASTContext &AST, Preprocessor &PP,
                        llvm::ArrayRef<Decl *> DeclsToIndex,
                        const MainFileMacros *MacroRefsToIndex,
                        const CanonicalIncludes &Includes, bool IsIndexMainAST,
                        llvm::StringRef Version, bool CollectMainFileRefs) {
   SymbolCollector::Options CollectorOpts;
   CollectorOpts.CollectIncludePath = true;
   CollectorOpts.Includes = &Includes;
   CollectorOpts.CountReferences = false;
-  CollectorOpts.Origin = SymbolOrigin::Dynamic;
+  CollectorOpts.Origin =
+      IsIndexMainAST ? SymbolOrigin::Open : SymbolOrigin::Preamble;
   CollectorOpts.CollectMainFileRefs = CollectMainFileRefs;
   // We want stdlib implementation details in the index only if we've opened the
   // file in question. This does means xrefs won't work, though.
   CollectorOpts.CollectReserved = IsIndexMainAST;
 
   index::IndexingOptions IndexOpts;
   // We only need declarations, because we don't count references.
   IndexOpts.SystemSymbolFilter =

clang-tools-extra/clangd/index/Serialization.h

struct IndexFileIn {
   llvm::Optional<RefSlab> Refs;
   llvm::Optional<RelationSlab> Relations;
   // Keys are URIs of the source files.
   llvm::Optional<IncludeGraph> Sources;
   // This contains only the Directory and CommandLine.
   llvm::Optional<tooling::CompileCommand> Cmd;
 };
 // Parse an index file. The input must be a RIFF or YAML file.
-llvm::Expected<IndexFileIn> readIndexFile(llvm::StringRef);
+llvm::Expected<IndexFileIn> readIndexFile(llvm::StringRef, SymbolOrigin);
 
 // Specifies the contents of an index file to be written.
 struct IndexFileOut {
   const SymbolSlab *Symbols = nullptr;
   const RefSlab *Refs = nullptr;
   const RelationSlab *Relations = nullptr;
   // Keys are URIs of the source files.
   const IncludeGraph *Sources = nullptr;

 llvm::raw_ostream &operator<<(llvm::raw_ostream &OS, const IndexFileOut &O);
 
 // Convert a single symbol to YAML, a nice debug representation.
 std::string toYAML(const Symbol &);
 std::string toYAML(const std::pair<SymbolID, ArrayRef<Ref>> &);
 std::string toYAML(const Relation &);
 std::string toYAML(const Ref &);
 
-// Deserialize a single symbol from YAML.
-llvm::Expected<clangd::Symbol> symbolFromYAML(StringRef YAML,
-                                              llvm::UniqueStringSaver *Strings);
-llvm::Expected<clangd::Ref> refFromYAML(StringRef YAML,
-                                        llvm::UniqueStringSaver *Strings);
-
 // Build an in-memory static index from an index file.
 // The size should be relatively small, so data can be managed in memory.
 std::unique_ptr<SymbolIndex> loadIndex(llvm::StringRef Filename,
-                                       bool UseDex = true);
+                                       SymbolOrigin Origin, bool UseDex = true);
 
 } // namespace clangd
 } // namespace clang
 
 #endif

clang-tools-extra/clangd/index/Serialization.cpp

void writeSymbol(const Symbol &Sym, const StringTableOut &Strings,
   OS.write(static_cast<uint8_t>(Sym.SymInfo.Lang));
   writeVar(Strings.index(Sym.Name), OS);
   writeVar(Strings.index(Sym.Scope), OS);
   writeVar(Strings.index(Sym.TemplateSpecializationArgs), OS);
   writeLocation(Sym.Definition, Strings, OS);
   writeLocation(Sym.CanonicalDeclaration, Strings, OS);
   writeVar(Sym.References, OS);
   OS.write(static_cast<uint8_t>(Sym.Flags));
-  OS.write(static_cast<uint8_t>(Sym.Origin));
   writeVar(Strings.index(Sym.Signature), OS);
   writeVar(Strings.index(Sym.CompletionSnippetSuffix), OS);
   writeVar(Strings.index(Sym.Documentation), OS);
   writeVar(Strings.index(Sym.ReturnType), OS);
   writeVar(Strings.index(Sym.Type), OS);
 
   auto WriteInclude = [&](const Symbol::IncludeHeaderWithReferences &Include) {
     writeVar(Strings.index(Include.IncludeHeader), OS);
     writeVar(Include.References, OS);
   };
   writeVar(Sym.IncludeHeaders.size(), OS);
   for (const auto &Include : Sym.IncludeHeaders)
     WriteInclude(Include);
 }
 
-Symbol readSymbol(Reader &Data, llvm::ArrayRef<llvm::StringRef> Strings) {
+Symbol readSymbol(Reader &Data, llvm::ArrayRef<llvm::StringRef> Strings,
+                  SymbolOrigin Origin) {
   Symbol Sym;
   Sym.ID = Data.consumeID();
   Sym.SymInfo.Kind = static_cast<index::SymbolKind>(Data.consume8());
   Sym.SymInfo.Lang = static_cast<index::SymbolLanguage>(Data.consume8());
   Sym.Name = Data.consumeString(Strings);
   Sym.Scope = Data.consumeString(Strings);
   Sym.TemplateSpecializationArgs = Data.consumeString(Strings);
   Sym.Definition = readLocation(Data, Strings);
   Sym.CanonicalDeclaration = readLocation(Data, Strings);
   Sym.References = Data.consumeVar();
   Sym.Flags = static_cast<Symbol::SymbolFlag>(Data.consume8());
-  Sym.Origin = static_cast<SymbolOrigin>(Data.consume8());
+  Sym.Origin = Origin;
   Sym.Signature = Data.consumeString(Strings);
   Sym.CompletionSnippetSuffix = Data.consumeString(Strings);
   Sym.Documentation = Data.consumeString(Strings);
   Sym.ReturnType = Data.consumeString(Strings);
   Sym.Type = Data.consumeString(Strings);
   if (!Data.consumeSize(Sym.IncludeHeaders))
     return Sym;
   for (auto &I : Sym.IncludeHeaders) {

 //   - srcs: information related to include graph
 //   - stri: string table
 //   - symb: symbols
 //   - refs: references to symbols
 
 // The current versioning scheme is simple - non-current versions are rejected.
 // If you make a breaking change, bump this version number to invalidate stored
 // data. Later we may want to support some backward compatibility.
-constexpr static uint32_t Version = 16;
+constexpr static uint32_t Version = 17;
 
-llvm::Expected<IndexFileIn> readRIFF(llvm::StringRef Data) {
+llvm::Expected<IndexFileIn> readRIFF(llvm::StringRef Data,
+                                     SymbolOrigin Origin) {
   auto RIFF = riff::readFile(Data);
   if (!RIFF)
     return RIFF.takeError();
   if (RIFF->Type != riff::fourCC("CdIx"))
     return error("wrong RIFF filetype: {0}", riff::fourCCStr(RIFF->Type));
   llvm::StringMap<llvm::StringRef> Chunks;
   for (const auto &Chunk : RIFF->Chunks)
     Chunks.try_emplace(llvm::StringRef(Chunk.ID.data(), Chunk.ID.size()),
if (Chunks.count("srcs")) {
     if (SrcsReader.err())
       return error("malformed or truncated include uri");
   }
 
   if (Chunks.count("symb")) {
     Reader SymbolReader(Chunks.lookup("symb"));
     SymbolSlab::Builder Symbols;
     while (!SymbolReader.eof())
-      Symbols.insert(readSymbol(SymbolReader, Strings->Strings));
+      Symbols.insert(readSymbol(SymbolReader, Strings->Strings, Origin));
     if (SymbolReader.err())
       return error("malformed or truncated symbol");
     Result.Symbols = std::move(Symbols).build();
   }
   if (Chunks.count("refs")) {
     Reader RefsReader(Chunks.lookup("refs"));
     RefSlab::Builder Refs;
     while (!RefsReader.eof()) {
void writeRIFF(const IndexFileOut &Data, llvm::raw_ostream &OS) {
 
   OS << RIFF;
 }
 
 } // namespace
 
 // Defined in YAMLSerialization.cpp.
 void writeYAML(const IndexFileOut &, llvm::raw_ostream &);
-llvm::Expected<IndexFileIn> readYAML(llvm::StringRef);
+llvm::Expected<IndexFileIn> readYAML(llvm::StringRef, SymbolOrigin Origin);
 
 llvm::raw_ostream &operator<<(llvm::raw_ostream &OS, const IndexFileOut &O) {
   switch (O.Format) {
   case IndexFileFormat::RIFF:
     writeRIFF(O, OS);
     break;
   case IndexFileFormat::YAML:
     writeYAML(O, OS);
     break;
   }
   return OS;
 }
 
-llvm::Expected<IndexFileIn> readIndexFile(llvm::StringRef Data) {
+llvm::Expected<IndexFileIn> readIndexFile(llvm::StringRef Data,
+                                          SymbolOrigin Origin) {
   if (Data.startswith("RIFF")) {
-    return readRIFF(Data);
+    return readRIFF(Data, Origin);
   }
-  if (auto YAMLContents = readYAML(Data)) {
+  if (auto YAMLContents = readYAML(Data, Origin)) {
     return std::move(*YAMLContents);
   } else {
     return error("Not a RIFF file and failed to parse as YAML: {0}",
                  YAMLContents.takeError());
   }
 }
 
 std::unique_ptr<SymbolIndex> loadIndex(llvm::StringRef SymbolFilename,
-                                       bool UseDex) {
+                                       SymbolOrigin Origin, bool UseDex) {
   trace::Span OverallTracer("LoadIndex");
   auto Buffer = llvm::MemoryBuffer::getFile(SymbolFilename);
   if (!Buffer) {
     elog("Can't open {0}: {1}", SymbolFilename, Buffer.getError().message());
     return nullptr;
   }
 
   SymbolSlab Symbols;
   RefSlab Refs;
   RelationSlab Relations;
   {
     trace::Span Tracer("ParseIndex");
-    if (auto I = readIndexFile(Buffer->get()->getBuffer())) {
+    if (auto I = readIndexFile(Buffer->get()->getBuffer(), Origin)) {
       if (I->Symbols)
         Symbols = std::move(*I->Symbols);
       if (I->Refs)
         Refs = std::move(*I->Refs);
       if (I->Relations)
         Relations = std::move(*I->Relations);
     } else {
       elog("Bad index file: {0}", I.takeError());

clang-tools-extra/clangd/index/SymbolOrigin.h

 #include <cstdint>
 
 namespace clang {
 namespace clangd {
 
 // Describes the source of information about a symbol.
 // Mainly useful for debugging, e.g. understanding code completion results.
 // This is a bitfield as information can be combined from several sources.
-enum class SymbolOrigin : uint8_t {
+enum class SymbolOrigin : uint16_t {
   Unknown = 0,
   AST = 1 << 0,        // Directly from the AST (indexes should not set this).
-  Dynamic = 1 << 1,    // From the dynamic index of opened files.
-  Static = 1 << 2,     // From the static, externally-built index.
+  Open = 1 << 1,       // From the dynamic index of open files.
+  Static = 1 << 2,     // From a static, externally-built index.
   Merge = 1 << 3,      // A non-trivial index merge was performed.
   Identifier = 1 << 4, // Raw identifiers in file.
   Remote = 1 << 5,     // Remote index.
-  // Remaining bits reserved for index implementations.
+  Preamble = 1 << 6,   // From the dynamic index of preambles.
+                       // 7 reserved
+  Background = 1 << 8, // From the automatic project index.
 };
 
 inline SymbolOrigin operator|(SymbolOrigin A, SymbolOrigin B) {
-  return static_cast<SymbolOrigin>(static_cast<uint8_t>(A) |
-                                   static_cast<uint8_t>(B));
+  return static_cast<SymbolOrigin>(static_cast<uint16_t>(A) |
+                                   static_cast<uint16_t>(B));
 }
 inline SymbolOrigin &operator|=(SymbolOrigin &A, SymbolOrigin B) {
   return A = A | B;
 }
 inline SymbolOrigin operator&(SymbolOrigin A, SymbolOrigin B) {
-  return static_cast<SymbolOrigin>(static_cast<uint8_t>(A) &
-                                   static_cast<uint8_t>(B));
+  return static_cast<SymbolOrigin>(static_cast<uint16_t>(A) &
+                                   static_cast<uint16_t>(B));
 }
 
 llvm::raw_ostream &operator<<(llvm::raw_ostream &, SymbolOrigin);
 
 } // namespace clangd
 } // namespace clang
 
 #endif // LLVM_CLANG_TOOLS_EXTRA_CLANGD_INDEX_SYMBOLORIGIN_H

clang-tools-extra/clangd/index/SymbolOrigin.cpp

 //===--- SymbolOrigin.cpp ----------------------------------------*- C++-*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "SymbolOrigin.h"
 
 namespace clang {
 namespace clangd {
 
 llvm::raw_ostream &operator<<(llvm::raw_ostream &OS, SymbolOrigin O) {
   if (O == SymbolOrigin::Unknown)
     return OS << "unknown";
-  constexpr static char Sigils[] = "ADSMIR67";
+  constexpr static char Sigils[] = "AOSMIRP7B9012345";
   for (unsigned I = 0; I < sizeof(Sigils); ++I)
-    if (static_cast<uint8_t>(O) & 1u << I)
+    if (static_cast<uint16_t>(O) & 1u << I)
       OS << Sigils[I];
   return OS;
 }
 
 } // namespace clangd
 } // namespace clang

clang-tools-extra/clangd/index/YAMLSerialization.cpp

 using clang::clangd::IncludeGraphNode;
 using clang::clangd::Ref;
 using clang::clangd::RefKind;
 using clang::clangd::Relation;
 using clang::clangd::RelationKind;
 using clang::clangd::Symbol;
 using clang::clangd::SymbolID;
 using clang::clangd::SymbolLocation;
-using clang::clangd::SymbolOrigin;
 using clang::index::SymbolInfo;
 using clang::index::SymbolKind;
 using clang::index::SymbolLanguage;
 using clang::tooling::CompileCommand;
 
 // Helper to (de)serialize the SymbolID. We serialize it as a hex string.
 struct NormalizedSymbolID {
   NormalizedSymbolID(IO &) {}
struct NormalizedSymbolFlag {
 
   Symbol::SymbolFlag denormalize(IO &) {
     return static_cast<Symbol::SymbolFlag>(Flag);
   }
 
   uint8_t Flag = 0;
 };
 
-struct NormalizedSymbolOrigin {
-  NormalizedSymbolOrigin(IO &) {}
-  NormalizedSymbolOrigin(IO &, SymbolOrigin O) {
-    Origin = static_cast<uint8_t>(O);
-  }
-
-  SymbolOrigin denormalize(IO &) { return static_cast<SymbolOrigin>(Origin); }
-
-  uint8_t Origin = 0;
-};
-
 template <> struct MappingTraits<YPosition> {
   static void mapping(IO &IO, YPosition &Value) {
     IO.mapRequired("Line", Value.Line);
     IO.mapRequired("Column", Value.Column);
   }
 };
 
 struct NormalizedPosition {
struct MappingTraits<clang::clangd::Symbol::IncludeHeaderWithReferences> {
   }
 };
 
 template <> struct MappingTraits<Symbol> {
   static void mapping(IO &IO, Symbol &Sym) {
     MappingNormalization<NormalizedSymbolID, SymbolID> NSymbolID(IO, Sym.ID);
     MappingNormalization<NormalizedSymbolFlag, Symbol::SymbolFlag> NSymbolFlag(
         IO, Sym.Flags);
-    MappingNormalization<NormalizedSymbolOrigin, SymbolOrigin> NSymbolOrigin(
-        IO, Sym.Origin);
     IO.mapRequired("ID", NSymbolID->HexString);
     IO.mapRequired("Name", Sym.Name);
     IO.mapRequired("Scope", Sym.Scope);
     IO.mapRequired("SymInfo", Sym.SymInfo);
     IO.mapOptional("CanonicalDeclaration", Sym.CanonicalDeclaration,
                    SymbolLocation());
     IO.mapOptional("Definition", Sym.Definition, SymbolLocation());
     IO.mapOptional("References", Sym.References, 0u);
-    IO.mapOptional("Origin", NSymbolOrigin->Origin);
     IO.mapOptional("Flags", NSymbolFlag->Flag);
     IO.mapOptional("Signature", Sym.Signature);
     IO.mapOptional("TemplateSpecializationArgs",
                    Sym.TemplateSpecializationArgs);
     IO.mapOptional("CompletionSnippetSuffix", Sym.CompletionSnippetSuffix);
     IO.mapOptional("Documentation", Sym.Documentation);
     IO.mapOptional("ReturnType", Sym.ReturnType);
     IO.mapOptional("Type", Sym.Type);
void writeYAML(const IndexFileOut &O, llvm::raw_ostream &OS) {
   }
   if (O.Cmd) {
     VariantEntry Entry;
     Entry.Cmd = *O.Cmd;
     Yout << Entry;
   }
 }
 
-llvm::Expected<IndexFileIn> readYAML(llvm::StringRef Data) {
+llvm::Expected<IndexFileIn> readYAML(llvm::StringRef Data,
+                                     SymbolOrigin Origin) {
   SymbolSlab::Builder Symbols;
   RefSlab::Builder Refs;
   RelationSlab::Builder Relations;
   llvm::BumpPtrAllocator
       Arena; // store the underlying data of Position::FileURI.
   llvm::UniqueStringSaver Strings(Arena);
   llvm::yaml::Input Yin(Data, &Strings);
   IncludeGraph Sources;
   llvm::Optional<tooling::CompileCommand> Cmd;
   while (Yin.setCurrentDocument()) {
     llvm::yaml::EmptyContext Ctx;
     VariantEntry Variant;
     yamlize(Yin, Variant, true, Ctx);
     if (Yin.error())
       return llvm::errorCodeToError(Yin.error());
 
-    if (Variant.Symbol)
+    if (Variant.Symbol) {
+      Variant.Symbol->Origin = Origin;
       Symbols.insert(*Variant.Symbol);
+    }
     if (Variant.Refs)
       for (const auto &Ref : Variant.Refs->second)
         Refs.insert(Variant.Refs->first, Ref);
     if (Variant.Relation)
       Relations.insert(*Variant.Relation);
     if (Variant.Source) {
       auto &IGN = Variant.Source.getValue();
       auto Entry = Sources.try_emplace(IGN.URI).first;
std::string Buf;
     llvm::raw_string_ostream OS(Buf);
     llvm::yaml::Output Yout(OS);
     Ref Reference = R; // copy: Yout<< requires mutability.
     Yout << Reference;
   }
   return Buf;
 }
 
-llvm::Expected<clangd::Symbol>
-symbolFromYAML(StringRef YAML, llvm::UniqueStringSaver *Strings) {
-  clangd::Symbol Deserialized;
-  llvm::yaml::Input YAMLInput(YAML, Strings);
-  if (YAMLInput.error())
-    return error("Unable to deserialize Symbol from YAML: {0}", YAML);
-  YAMLInput >> Deserialized;
-  return Deserialized;
-}
-
-llvm::Expected<clangd::Ref> refFromYAML(StringRef YAML,
-                                        llvm::UniqueStringSaver *Strings) {
-  clangd::Ref Deserialized;
-  llvm::yaml::Input YAMLInput(YAML, Strings);
-  if (YAMLInput.error())
-    return error("Unable to deserialize Symbol from YAML: {0}", YAML);
-  YAMLInput >> Deserialized;
-  return Deserialized;
-}
-
 } // namespace clangd
 } // namespace clang

clang-tools-extra/clangd/index/dex/dexp/Dexp.cpp

void run() override {
     // Read input file (as specified in global option)
     auto Buffer = llvm::MemoryBuffer::getFile(IndexLocation);
     if (!Buffer) {
       llvm::errs() << llvm::formatv("Can't open {0}", IndexLocation) << "\n";
       return;
     }
 
     // Auto-detects input format when parsing
-    auto IndexIn = clang::clangd::readIndexFile(Buffer->get()->getBuffer());
+    auto IndexIn = clang::clangd::readIndexFile(Buffer->get()->getBuffer(),
+                                                SymbolOrigin::Static);
     if (!IndexIn) {
       llvm::errs() << llvm::toString(IndexIn.takeError()) << "\n";
       return;
     }
 
     // Prepare output file
     std::error_code EC;
     llvm::raw_fd_ostream OutputStream(OutputFile, EC);
std::function<std::unique_ptr<Command>()> Implementation;
      std::make_unique<Relations>},
     {"export", "Export index", std::make_unique<Export>},
 };
 
 std::unique_ptr<SymbolIndex> openIndex(llvm::StringRef Index) {
   return Index.startswith("remote:")
              ? remote::getClient(Index.drop_front(strlen("remote:")),
                                  ProjectRoot)
-             : loadIndex(Index, /*UseDex=*/true);
+             : loadIndex(Index, SymbolOrigin::Static, /*UseDex=*/true);
 }
 
 bool runCommand(std::string Request, const SymbolIndex &Index) {
   // Split on spaces and add required null-termination.
   std::replace(Request.begin(), Request.end(), ' ', '\0');
   llvm::SmallVector<llvm::StringRef> Args;
   llvm::StringRef(Request).split(Args, '\0', /*MaxSplit=*/-1,
                                  /*KeepEmpty=*/false);

clang-tools-extra/clangd/index/remote/Index.proto

 message Symbol {
   optional string id = 1;
   optional SymbolInfo info = 2;
   optional string name = 3;
   optional SymbolLocation definition = 4;
   optional string scope = 5;
   optional SymbolLocation canonical_declaration = 6;
   optional int32 references = 7;
-  optional uint32 origin = 8;
+  reserved 8;
   optional string signature = 9;
   optional string template_specialization_args = 10;
   optional string completion_snippet_suffix = 11;
   optional string documentation = 12;
   optional string return_type = 13;
   optional string type = 14;
   repeated HeaderWithReferences headers = 15;
   optional uint32 flags = 16;

clang-tools-extra/clangd/index/remote/marshalling/Marshalling.cpp

if (*From.Definition.FileURI) {
     *Result.mutable_definition() = *Definition;
   }
   Result.set_scope(From.Scope.str());
   auto Declaration = toProtobuf(From.CanonicalDeclaration);
   if (!Declaration)
     return Declaration.takeError();
   *Result.mutable_canonical_declaration() = *Declaration;
   Result.set_references(From.References);
-  Result.set_origin(static_cast<uint32_t>(From.Origin));
   Result.set_signature(From.Signature.str());
   Result.set_template_specialization_args(
       From.TemplateSpecializationArgs.str());
   Result.set_completion_snippet_suffix(From.CompletionSnippetSuffix.str());
   Result.set_documentation(From.Documentation.str());
   Result.set_return_type(From.ReturnType.str());
   Result.set_type(From.Type.str());
   for (const auto &Header : From.IncludeHeaders) {

clang-tools-extra/clangd/index/remote/server/Server.cpp

void hotReload(clangd::SwapIndex &Index, llvm::StringRef IndexPath,
   if (!Status || (Status->getLastModificationTime() ==
                       LastStatus.getLastModificationTime() &&
                   Status->getSize() == LastStatus.getSize()))
     return;
   vlog("Found different index version: existing index was modified at "
        "{0}, new index was modified at {1}. Attempting to reload.",
        LastStatus.getLastModificationTime(), Status->getLastModificationTime());
   LastStatus = *Status;
-  std::unique_ptr<clang::clangd::SymbolIndex> NewIndex = loadIndex(IndexPath);
+  std::unique_ptr<clang::clangd::SymbolIndex> NewIndex =
+      loadIndex(IndexPath, SymbolOrigin::Static);
   if (!NewIndex) {
     elog("Failed to load new index. Old index will be served.");
     return;
   }
   Index.reset(std::move(NewIndex));
   Monitor.updateIndex(Status->getLastModificationTime());
   log("New index version loaded. Last modification time: {0}, size: {1} bytes.",
       Status->getLastModificationTime(), Status->getSize());

clang-tools-extra/clangd/tool/ClangdMain.cpp

log("Associating {0} with remote index at {1}.", External.MountPoint,
         External.Location);
     return remote::getClient(External.Location, External.MountPoint);
   case Config::ExternalIndexSpec::File:
     log("Associating {0} with monolithic index at {1}.", External.MountPoint,
         External.Location);
     auto NewIndex = std::make_unique<SwapIndex>(std::make_unique<MemIndex>());
     auto IndexLoadTask = [File = External.Location,
                           PlaceHolder = NewIndex.get()] {
-      if (auto Idx = loadIndex(File, /*UseDex=*/true))
+      if (auto Idx = loadIndex(File, SymbolOrigin::Static, /*UseDex=*/true))
         PlaceHolder->reset(std::move(Idx));
     };
     if (Tasks) {
       Tasks->runAsync("Load-index:" + External.Location,
                       std::move(IndexLoadTask));
     } else {
       IndexLoadTask();
     }

clang-tools-extra/clangd/unittests/BackgroundIndexTests.cpp

public:
   }
   std::unique_ptr<IndexFileIn>
   loadShard(llvm::StringRef ShardIdentifier) const override {
     std::lock_guard<std::mutex> Lock(StorageMu);
     AccessedPaths.insert(ShardIdentifier);
     if (Storage.find(ShardIdentifier) == Storage.end()) {
       return nullptr;
     }
-    auto IndexFile = readIndexFile(Storage[ShardIdentifier]);
+    auto IndexFile =
+        readIndexFile(Storage[ShardIdentifier], SymbolOrigin::Background);
     if (!IndexFile) {
       ADD_FAILURE() << "Error while reading " << ShardIdentifier << ':'
                     << IndexFile.takeError();
       return nullptr;
     }
     CacheHits++;
     return std::make_unique<IndexFileIn>(std::move(*IndexFile));
   }

clang-tools-extra/clangd/unittests/IndexTests.cpp

TEST(MergeTest, Merge) {
   L.Name = R.Name = "Foo";                           // same in both
   L.CanonicalDeclaration.FileURI = "file:///left.h"; // differs
   R.CanonicalDeclaration.FileURI = "file:///right.h";
   L.References = 1;
   R.References = 2;
   L.Signature = "()";                   // present in left only
   R.CompletionSnippetSuffix = "{$1:0}"; // present in right only
   R.Documentation = "--doc--";
-  L.Origin = SymbolOrigin::Dynamic;
+  L.Origin = SymbolOrigin::Preamble;
   R.Origin = SymbolOrigin::Static;
   R.Type = "expectedType";
 
   Symbol M = mergeSymbol(L, R);
   EXPECT_EQ(M.Name, "Foo");
   EXPECT_EQ(StringRef(M.CanonicalDeclaration.FileURI), "file:///left.h");
   EXPECT_EQ(M.References, 3u);
   EXPECT_EQ(M.Signature, "()");
   EXPECT_EQ(M.CompletionSnippetSuffix, "{$1:0}");
   EXPECT_EQ(M.Documentation, "--doc--");
   EXPECT_EQ(M.Type, "expectedType");
-  EXPECT_EQ(M.Origin,
-            SymbolOrigin::Dynamic | SymbolOrigin::Static | SymbolOrigin::Merge);
+  EXPECT_EQ(M.Origin, SymbolOrigin::Preamble | SymbolOrigin::Static |
+                          SymbolOrigin::Merge);
 }
 
 TEST(MergeTest, PreferSymbolWithDefn) {
   Symbol L, R;
 
   L.ID = R.ID = SymbolID("hello");
   L.CanonicalDeclaration.FileURI = "file:/left.h";
   R.CanonicalDeclaration.FileURI = "file:/right.h";

clang-tools-extra/clangd/unittests/SerializationTests.cpp

 CanonicalDeclaration:
   FileURI:        file:///path/foo.h
   Start:
     Line: 1
     Column: 0
   End:
     Line: 1
     Column: 1
-Origin:    128
 Flags:    129
 Documentation:    'Foo doc'
 ReturnType:    'int'
 IncludeHeaders:
   - Header:    'include1'
     References:    7
   - Header:    'include2'
     References:    3

 )";
 
 MATCHER_P(ID, I, "") { return arg.ID == cantFail(SymbolID::fromStr(I)); }
 MATCHER_P(QName, Name, "") { return (arg.Scope + arg.Name).str() == Name; }
 MATCHER_P2(IncludeHeaderWithRef, IncludeHeader, References, "") {
   return (arg.IncludeHeader == IncludeHeader) && (arg.References == References);
 }
 
+auto readIndexFile(llvm::StringRef Text) {
+  return readIndexFile(Text, SymbolOrigin::Static);
+}
+
 TEST(SerializationTest, NoCrashOnEmptyYAML) {
   EXPECT_TRUE(bool(readIndexFile("")));
 }
 
 TEST(SerializationTest, YAMLConversions) {
   auto ParsedYAML = readIndexFile(YAML);
   ASSERT_TRUE(bool(ParsedYAML)) << ParsedYAML.takeError();
   ASSERT_TRUE(bool(ParsedYAML->Symbols));
   EXPECT_THAT(
       *ParsedYAML->Symbols,
       UnorderedElementsAre(ID("057557CEBF6E6B2D"), ID("057557CEBF6E6B2E")));
 
   auto Sym1 = *ParsedYAML->Symbols->find(
       cantFail(SymbolID::fromStr("057557CEBF6E6B2D")));
   auto Sym2 = *ParsedYAML->Symbols->find(
       cantFail(SymbolID::fromStr("057557CEBF6E6B2E")));
 
   EXPECT_THAT(Sym1, QName("clang::Foo1"));
   EXPECT_EQ(Sym1.Signature, "");
   EXPECT_EQ(Sym1.Documentation, "Foo doc");
   EXPECT_EQ(Sym1.ReturnType, "int");
   EXPECT_EQ(StringRef(Sym1.CanonicalDeclaration.FileURI), "file:///path/foo.h");
-  EXPECT_EQ(Sym1.Origin, static_cast<SymbolOrigin>(1 << 7));
+  EXPECT_EQ(Sym1.Origin, SymbolOrigin::Static);
   EXPECT_EQ(static_cast<uint8_t>(Sym1.Flags), 129);
   EXPECT_TRUE(Sym1.Flags & Symbol::IndexedForCodeCompletion);
   EXPECT_FALSE(Sym1.Flags & Symbol::Deprecated);
   EXPECT_THAT(Sym1.IncludeHeaders,
               UnorderedElementsAre(IncludeHeaderWithRef("include1", 7u),
                                    IncludeHeaderWithRef("include2", 3u)));
 
   EXPECT_THAT(Sym2, QName("clang::Foo2"));