[clangd] Add index export to dexp

[mnauw]

Summary

Add a command to dexp that exports index data in chosen format (e.g. YAML).

[kadircet]

Thanks for the patch!

We already have one introspection tool called dexp have you give it a try? It can read both RIFF and YAML and allows you to run queries on the index.

If it is not enough for your use case, can you describe it a little more? Maybe it would be easier to extend dexp to accommodate your use case instead of introducing a new binary.

[mnauw]

Thanks for having a look at this patch. My use-case/goal would be to have a way to run/do a command that reads some index and dumps it (in e.g. YAML) to (e.g.) stdout. That way it can be inspected/viewed all at once by plain viewer. And not only symbols, refs, but also recorded paths (that form the include-graph). The latter in particular was/is useful when investigating some not-entirely-consistent symlink-resolution issues. The goal is not necessarily a separate binary, so if it's ok to extend dexp to achieve the former (e.g. by adding option(s)) then I can also update to patch that way accordingly.

[sammccall]

Yeah, I think adding a dump/export command to dexp would be nice.
We'd have to make dexp keep the parsed input file around, or just the path and reopen it when the command is issued, but that seems OK to me.

(One of the costs of adding a new binary is you have to at least build it to keep the code from rotting, and linking binaries is often the bottleneck for iterating with check-clangd)

[sammccall]

Thanks, this looks pretty good!

clang-tools-extra/clangd/index/YAMLSerialization.cpp

36 ↗

(On Diff #255019)

this is independent of adding the dump command - can we split this into a separate patch?
Partly because it will need to have some basic testing :-(

355 ↗

(On Diff #255019)

this + 2 helpers seems a bit verbose/indirect.

FileDigest D;
if (HexString.size() == D.size() && llvm::all_of(HexString, llvm:isHexDigit)) {
  memcpy(Digest.data(), llvm::fromHex(HexString).data(), Digest.size());
} else {
  I.setError("Bad hex file digest");
}
return D;

380 ↗

(On Diff #255019)

Unfortunately we don't own the CompileCommand type, so we shouldn't specialize traits for it (risk ODR violation if someone else does the same).
Wrap or inherit it in a trivial struct struct CompileCommandYAML : tooling::CompileCommand {} to make this safe.

clang-tools-extra/clangd/index/dex/dexp/Dexp.cpp

290

Writing it to stdout doesn't seem unreasonable but maybe not the right default.
What about requiring a filename arg, and creating a raw_fd_ostream - then you can pass - to get stdout.

364

I wonder if we should generalize this instead to running an arbitrary command non-interactively: dexp -c "dump"

No need to do that in this patch but maybe leave a TODO

[kbobyrev]

Typical use is to read RIFF data to dump to YAML for inspection of indexed data. The YAML (de)serialization has also been extended to aid in this regard.

Just curious: what is the typical usecase for YAML index inspection? Maybe if there is something missing in Dexp, it would be worth adding the corresponding inspection tooling (not suggesting it for this patch, this is basically for me to understand how to improve Dexp).

[sammccall]

clang-tools-extra/clangd/index/dex/dexp/Dexp.cpp

364

Done in 49268a678c2f0233f71363b0988d4b159496b036 - I think we can drop this and fold ExportImpl and ExportCmd together.

[mnauw]

As suggested, I will provide the YAMLSerialization changes in a separate patch. Regarding test for that, I was thinking of some running some "simple/mock YAML data" through some load and save cycle(s). I presume a separate YAML file (in e.g. test/Inputs) is best for that?

clang-tools-extra/clangd/index/dex/dexp/Dexp.cpp

Index->refs(RefRequest, [&RegexFilter](const Ref &R) {
         return;
       }
       if (RegexFilter.match(U->body()))
         llvm::outs() << R << "\n";
     });
   }
 };
 
+class Export : public Command {
+  llvm::cl::opt<IndexFileFormat> Format{
+      "format",
+      llvm::cl::desc("Format of index export"),
+      llvm::cl::values(
+          clEnumValN(IndexFileFormat::YAML, "yaml",
+                     "human-readable YAML format"),
+          clEnumValN(IndexFileFormat::RIFF, "binary", "binary RIFF format")),
+      llvm::cl::init(IndexFileFormat::YAML),
+  };
+  llvm::cl::opt<std::string> OutputFile{
+      "output-file",
+      llvm::cl::Positional,
+      llvm::cl::Required,
+      llvm::cl::desc("Output file for export"),
+  };
+
+public:
+  void run() {
+    using namespace clang::clangd;
+    // Read input file (as specified in global option)
+    auto Buffer = llvm::MemoryBuffer::getFile(IndexPath);
+    if (!Buffer) {
+      llvm::errs() << llvm::formatv("Can't open {0}", IndexPath) << "\n";
+      return;
+    }
+
+    // Auto-detects input format when parsing
+    auto IndexIn = clang::clangd::readIndexFile(Buffer->get()->getBuffer());
+    if (!IndexIn) {
+      llvm::errs() << llvm::toString(IndexIn.takeError()) << "\n";
+      return;
+    }
+
+    // Prepare output file
+    std::error_code EC;
+    llvm::raw_fd_ostream OutputStream(OutputFile, EC);
+    if (EC) {
+      llvm::errs() << llvm::formatv("Can't open {0} for writing", OutputFile)
+                   << "\n";
+      return;
+    }
+
+    // Export
+    clang::clangd::IndexFileOut IndexOut(IndexIn.get());
+    IndexOut.Format = Format;
+    OutputStream << IndexOut;
+  }
+};
+
 struct {
   const char *Name;
   const char *Description;
   std::function<std::unique_ptr<Command>()> Implementation;
 } CommandInfo[] = {
     {"find", "Search for symbols with fuzzyFind", std::make_unique<FuzzyFind>},
     {"lookup", "Dump symbol details by ID or qualified name",
      std::make_unique<Lookup>},
     {"refs", "Find references by ID or qualified name",
      std::make_unique<Refs>},
+    {"export", "Export index", std::make_unique<Export>},
 };
 
 std::unique_ptr<SymbolIndex> openIndex(llvm::StringRef Index) {
   return loadIndex(Index, /*UseDex=*/true);
 }
 
 bool runCommand(std::string Request, const SymbolIndex &Index) {
   // Split on spaces and add required null-termination.

 } // namespace
 } // namespace clangd
 } // namespace clang
 
 int main(int argc, const char *argv[]) {
   using namespace clang::clangd;
 
   llvm::cl::ParseCommandLineOptions(argc, argv, Overview);
   llvm::cl::ResetCommandLineParser(); // We reuse it for REPL commands.
   llvm::sys::PrintStackTraceOnErrorSignal(argv[0]);
 
   std::unique_ptr<SymbolIndex> Index;
   reportTime("Dex build", [&]() {
     Index = openIndex(IndexPath);
   });
 
   if (!Index) {