[clangd] Add index export to dexp

[mnauw]

Summary

Add a command to dexp that exports index data in chosen format (e.g. YAML).

[kadircet]

Thanks for the patch!

We already have one introspection tool called dexp have you give it a try? It can read both RIFF and YAML and allows you to run queries on the index.

If it is not enough for your use case, can you describe it a little more? Maybe it would be easier to extend dexp to accommodate your use case instead of introducing a new binary.

[mnauw]

Thanks for having a look at this patch. My use-case/goal would be to have a way to run/do a command that reads some index and dumps it (in e.g. YAML) to (e.g.) stdout. That way it can be inspected/viewed all at once by plain viewer. And not only symbols, refs, but also recorded paths (that form the include-graph). The latter in particular was/is useful when investigating some not-entirely-consistent symlink-resolution issues. The goal is not necessarily a separate binary, so if it's ok to extend dexp to achieve the former (e.g. by adding option(s)) then I can also update to patch that way accordingly.

[sammccall]

Yeah, I think adding a dump/export command to dexp would be nice.
We'd have to make dexp keep the parsed input file around, or just the path and reopen it when the command is issued, but that seems OK to me.

(One of the costs of adding a new binary is you have to at least build it to keep the code from rotting, and linking binaries is often the bottleneck for iterating with check-clangd)

[sammccall]

Thanks, this looks pretty good!

clang-tools-extra/clangd/index/YAMLSerialization.cpp

36

this is independent of adding the dump command - can we split this into a separate patch?
Partly because it will need to have some basic testing :-(

355

this + 2 helpers seems a bit verbose/indirect.

FileDigest D;
if (HexString.size() == D.size() && llvm::all_of(HexString, llvm:isHexDigit)) {
  memcpy(Digest.data(), llvm::fromHex(HexString).data(), Digest.size());
} else {
  I.setError("Bad hex file digest");
}
return D;

380

Unfortunately we don't own the CompileCommand type, so we shouldn't specialize traits for it (risk ODR violation if someone else does the same).
Wrap or inherit it in a trivial struct struct CompileCommandYAML : tooling::CompileCommand {} to make this safe.

clang-tools-extra/clangd/index/dex/dexp/Dexp.cpp

288

Writing it to stdout doesn't seem unreasonable but maybe not the right default.
What about requiring a filename arg, and creating a raw_fd_ostream - then you can pass - to get stdout.

322

I wonder if we should generalize this instead to running an arbitrary command non-interactively: dexp -c "dump"

No need to do that in this patch but maybe leave a TODO

[kbobyrev]

Typical use is to read RIFF data to dump to YAML for inspection of indexed data. The YAML (de)serialization has also been extended to aid in this regard.

Just curious: what is the typical usecase for YAML index inspection? Maybe if there is something missing in Dexp, it would be worth adding the corresponding inspection tooling (not suggesting it for this patch, this is basically for me to understand how to improve Dexp).

[sammccall]

clang-tools-extra/clangd/index/dex/dexp/Dexp.cpp

322

Done in 49268a678c2f0233f71363b0988d4b159496b036 - I think we can drop this and fold ExportImpl and ExportCmd together.

[mnauw]

As suggested, I will provide the YAMLSerialization changes in a separate patch. Regarding test for that, I was thinking of some running some "simple/mock YAML data" through some load and save cycle(s). I presume a separate YAML file (in e.g. test/Inputs) is best for that?

clang-tools-extra/clangd/index/YAMLSerialization.cpp

 #include "llvm/Support/StringSaver.h"
 #include "llvm/Support/YAMLTraits.h"
 #include "llvm/Support/raw_ostream.h"
 #include <cstdint>
 
 LLVM_YAML_IS_SEQUENCE_VECTOR(clang::clangd::Symbol::IncludeHeaderWithReferences)
 LLVM_YAML_IS_SEQUENCE_VECTOR(clang::clangd::Ref)
 
 namespace {
 using RefBundle =
     std::pair<clang::clangd::SymbolID, std::vector<clang::clangd::Ref>>;
 // This is a pale imitation of std::variant<Symbol, RefBundle, Relation>
 struct VariantEntry {
   llvm::Optional<clang::clangd::Symbol> Symbol;
   llvm::Optional<RefBundle> Refs;
   llvm::Optional<clang::clangd::Relation> Relation;
+  llvm::Optional<clang::clangd::IncludeGraphNode> Source;
+  llvm::Optional<clang::tooling::CompileCommand> Cmd;
 };
 // A class helps YAML to serialize the 32-bit encoded position (Line&Column),
 // as YAMLIO can't directly map bitfields.
 struct YPosition {
   uint32_t Line;
   uint32_t Column;
 };
 
 } // namespace
 namespace llvm {
 namespace yaml {
 
+using clang::clangd::FileDigest;
+using clang::clangd::IncludeGraph;
+using clang::clangd::IncludeGraphNode;
 using clang::clangd::Ref;
 using clang::clangd::RefKind;
 using clang::clangd::Relation;
 using clang::clangd::RelationKind;
 using clang::clangd::Symbol;
 using clang::clangd::SymbolID;
 using clang::clangd::SymbolLocation;
 using clang::clangd::SymbolOrigin;
 using clang::index::SymbolInfo;
 using clang::index::SymbolKind;
 using clang::index::SymbolLanguage;
 using clang::index::SymbolRole;
+using clang::tooling::CompileCommand;
 
 // Helper to (de)serialize the SymbolID. We serialize it as a hex string.
 struct NormalizedSymbolID {
   NormalizedSymbolID(IO &) {}
   NormalizedSymbolID(IO &, const SymbolID &ID) {
     llvm::raw_string_ostream OS(HexString);
     OS << ID;
   }
static void mapping(IO &IO, Relation &Relation) {
     MappingNormalization<NormalizedSymbolRole, RelationKind> NRole(
         IO, Relation.Predicate);
     IO.mapRequired("Subject", Relation.Subject);
     IO.mapRequired("Predicate", NRole->Kind);
     IO.mapRequired("Object", Relation.Object);
   }
 };
 
+struct NormalizedSourceFlag {
+  NormalizedSourceFlag(IO &) {}
+  NormalizedSourceFlag(IO &, IncludeGraphNode::SourceFlag O) {
+    Flag = static_cast<uint8_t>(O);
+  }
+
+  IncludeGraphNode::SourceFlag denormalize(IO &) {
+    return static_cast<IncludeGraphNode::SourceFlag>(Flag);
+  }
+
+  uint8_t Flag = 0;
+};
+
+struct NormalizedFileDigest {
+  NormalizedFileDigest(IO &) {}
+  NormalizedFileDigest(IO &, const FileDigest &Digest) {
+    HexString = llvm::toHex(Digest);
+  }
+
+  static FileDigest fromRaw(llvm::StringRef Raw) {
+    FileDigest Digest;
+    assert(Raw.size() == sizeof(Digest));
+    memcpy(Digest.data(), Raw.data(), Raw.size());
+    return Digest;
+  }
+
+  static llvm::Expected<FileDigest> fromStr(llvm::StringRef Str) {
+    const int RawSize = sizeof(FileDigest);
+    if (Str.size() != RawSize * 2)
+      return llvm::createStringError(llvm::inconvertibleErrorCode(),
+                                     "Bad ID length");
+    for (char C : Str)
+      if (!llvm::isHexDigit(C))
+        return llvm::createStringError(llvm::inconvertibleErrorCode(),
+                                       "Bad hex ID");
+    return fromRaw(llvm::fromHex(Str));
+  }
+
+  FileDigest denormalize(IO &I) {
+    auto Digest = fromStr(HexString);
+    if (!Digest) {
+      I.setError(llvm::toString(Digest.takeError()));
+      return FileDigest();
+    }
+    return *Digest;
+  }
+
+  std::string HexString;
+};
+
+template <> struct MappingTraits<IncludeGraphNode> {
+  static void mapping(IO &IO, IncludeGraphNode &Node) {
+    IO.mapRequired("URI", Node.URI);
+    MappingNormalization<NormalizedSourceFlag, IncludeGraphNode::SourceFlag>
+        NSourceFlag(IO, Node.Flags);
+    IO.mapRequired("Flags", NSourceFlag->Flag);
+    MappingNormalization<NormalizedFileDigest, FileDigest> NDigest(IO,
+                                                                   Node.Digest);
+    IO.mapRequired("Digest", NDigest->HexString);
+    IO.mapRequired("DirectIncludes", Node.DirectIncludes);
+  }
+};
+
+template <> struct MappingTraits<CompileCommand> {
+  static void mapping(IO &IO, CompileCommand &Cmd) {
+    IO.mapRequired("Directory", Cmd.Directory);
+    IO.mapRequired("CommandLine", Cmd.CommandLine);
+  }
+};
+
 template <> struct MappingTraits<VariantEntry> {
   static void mapping(IO &IO, VariantEntry &Variant) {
     if (IO.mapTag("!Symbol", Variant.Symbol.hasValue())) {
       if (!IO.outputting())
         Variant.Symbol.emplace();
       MappingTraits<Symbol>::mapping(IO, *Variant.Symbol);
     } else if (IO.mapTag("!Refs", Variant.Refs.hasValue())) {
       if (!IO.outputting())
         Variant.Refs.emplace();
       MappingTraits<RefBundle>::mapping(IO, *Variant.Refs);
     } else if (IO.mapTag("!Relations", Variant.Relation.hasValue())) {
       if (!IO.outputting())
         Variant.Relation.emplace();
       MappingTraits<Relation>::mapping(IO, *Variant.Relation);
+    } else if (IO.mapTag("!Source", Variant.Source.hasValue())) {
+      if (!IO.outputting())
+        Variant.Source.emplace();
+      MappingTraits<IncludeGraphNode>::mapping(IO, *Variant.Source);
+    } else if (IO.mapTag("!Cmd", Variant.Cmd.hasValue())) {
+      if (!IO.outputting())
+        Variant.Cmd.emplace();
+      MappingTraits<CompileCommand>::mapping(IO, *Variant.Cmd);
     }
   }
 };
 
 } // namespace yaml
 } // namespace llvm
 
 namespace clang {
for (auto &Sym : *O.Refs) {
       Yout << Entry;
     }
   if (O.Relations)
     for (auto &R : *O.Relations) {
       VariantEntry Entry;
       Entry.Relation = R;
       Yout << Entry;
     }
+  if (O.Sources) {
+    for (const auto &Source : *O.Sources) {
+      VariantEntry Entry;
+      Entry.Source = Source.getValue();
+      Yout << Entry;
+    }
+  }
+  if (O.Cmd) {
+    VariantEntry Entry;
+    Entry.Cmd = *O.Cmd;
+    Yout << Entry;
+  }
 }
 
 llvm::Expected<IndexFileIn> readYAML(llvm::StringRef Data) {
   SymbolSlab::Builder Symbols;
   RefSlab::Builder Refs;
   RelationSlab::Builder Relations;
   llvm::BumpPtrAllocator
       Arena; // store the underlying data of Position::FileURI.
   llvm::UniqueStringSaver Strings(Arena);
   llvm::yaml::Input Yin(Data, &Strings);
+  IncludeGraph Sources;
+  llvm::Optional<tooling::CompileCommand> Cmd;
   while (Yin.setCurrentDocument()) {
     llvm::yaml::EmptyContext Ctx;
     VariantEntry Variant;
     yamlize(Yin, Variant, true, Ctx);
     if (Yin.error())
       return llvm::errorCodeToError(Yin.error());
 
     if (Variant.Symbol)
       Symbols.insert(*Variant.Symbol);
     if (Variant.Refs)
       for (const auto &Ref : Variant.Refs->second)
         Refs.insert(Variant.Refs->first, Ref);
     if (Variant.Relation)
       Relations.insert(*Variant.Relation);
+    if (Variant.Source) {
+      auto &IGN = Variant.Source.getValue();
+      auto Entry = Sources.try_emplace(IGN.URI).first;
+      Entry->getValue() = std::move(IGN);
+      // Fixup refs to refer to map keys which will live on
+      Entry->getValue().URI = Entry->getKey();
+      for (auto &Include : Entry->getValue().DirectIncludes)
+        Include = Sources.try_emplace(Include).first->getKey();
+    }
+    if (Variant.Cmd)
+      Cmd = *Variant.Cmd;
     Yin.nextDocument();
   }
 
   IndexFileIn Result;
   Result.Symbols.emplace(std::move(Symbols).build());
   Result.Refs.emplace(std::move(Refs).build());
   Result.Relations.emplace(std::move(Relations).build());
+  if (Sources.size())
+    Result.Sources = std::move(Sources);
+  Result.Cmd = std::move(Cmd);
   return std::move(Result);
 }
 
 std::string toYAML(const Symbol &S) {
   std::string Buf;
   {
     llvm::raw_string_ostream OS(Buf);
     llvm::yaml::Output Yout(OS);

clang-tools-extra/clangd/index/dex/dexp/Dexp.cpp

 #include "llvm/LineEditor/LineEditor.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Signals.h"
 
 namespace clang {
 namespace clangd {
 namespace {
 
+static llvm::cl::opt<bool>
+    Export("export", llvm::cl::desc("Export index non-interactively"),
+           llvm::cl::init(false));
+
 llvm::cl::opt<std::string> IndexPath("index-path",
                                      llvm::cl::desc("Path to the index"),
                                      llvm::cl::Positional, llvm::cl::Required);
 
 static const std::string Overview = R"(
 This is an **experimental** interactive tool to process user-provided search
 queries over given symbol collection obtained via clangd-indexer. The
 tool can be used to evaluate search quality of existing index implementations
Index->refs(RefRequest, [&RegexFilter](const Ref &R) {
         return;
       }
       if (RegexFilter.match(U->body()))
         llvm::outs() << R << "\n";
     });
   }
 };
 
+class ExportImpl {
+  llvm::cl::opt<IndexFileFormat> Format{
+      "format", llvm::cl::desc("Format of index export"),
+      llvm::cl::values(
+          clEnumValN(IndexFileFormat::YAML, "yaml",
+                     "human-readable YAML format"),
+          clEnumValN(IndexFileFormat::RIFF, "binary", "binary RIFF format")),
+      llvm::cl::init(IndexFileFormat::YAML)};
+
+public:
+  int dump() {
+    using namespace clang::clangd;
+    // Read input file (as specified in global option)
+    auto Buffer = llvm::MemoryBuffer::getFile(IndexPath);
+    if (!Buffer) {
+      llvm::errs() << llvm::formatv("Can't open {0}", IndexPath) << "\n";
+      return 1;
+    }
+
+    // Auto-detects input format when parsing
+    auto IndexIn = clang::clangd::readIndexFile(Buffer->get()->getBuffer());
+    if (!IndexIn) {
+      llvm::errs() << llvm::toString(IndexIn.takeError()) << "\n";
+      return 1;
+    }
+
+    // Dump output file
+    clang::clangd::IndexFileOut IndexOut(IndexIn.get());
+    IndexOut.Format = Format;
+    llvm::outs() << IndexOut;
+    return 0;
+  }
+};
+
+class ExportCmd : public ExportImpl, public Command {
+  void run() override { dump(); }
+};
+
 struct {
   const char *Name;
   const char *Description;
   std::function<std::unique_ptr<Command>()> Implementation;
 } CommandInfo[] = {
     {"find", "Search for symbols with fuzzyFind", std::make_unique<FuzzyFind>},
     {"lookup", "Dump symbol details by ID or qualified name",
      std::make_unique<Lookup>},
     {"refs", "Find references by ID or qualified name",
      std::make_unique<Refs>},
+    {"export", "Export index", std::make_unique<ExportCmd>},
 };
 
 std::unique_ptr<SymbolIndex> openIndex(llvm::StringRef Index) {
   return loadIndex(Index, /*UseDex=*/true);
 }
 
 } // namespace
 } // namespace clangd
 } // namespace clang
 
 int main(int argc, const char *argv[]) {
   using namespace clang::clangd;
 
+  // Make Export command option(s) available on command line.
+  // That allows for convenient (piping/redirecting) a dump non-interactively
+  // without passing through REPL.
+  auto NonInteractiveExport = std::make_unique<ExportImpl>();
   llvm::cl::ParseCommandLineOptions(argc, argv, Overview);
-  llvm::cl::ResetCommandLineParser(); // We reuse it for REPL commands.
   llvm::sys::PrintStackTraceOnErrorSignal(argv[0]);
 
+  if (Export)
+    return NonInteractiveExport->dump();
+
+  // Back to normal
+  llvm::cl::ResetCommandLineParser(); // We reuse it for REPL commands.
+  NonInteractiveExport.reset();
+
   std::unique_ptr<SymbolIndex> Index;
   reportTime("Dex build", [&]() {
     Index = openIndex(IndexPath);
   });
 
   if (!Index) {
     llvm::outs() << "Failed to open the index.\n";
     return -1;