[clangd] Add a tool to build YAML-format global symbols.

[hokein]

Summary

The tools is used to generate global symbols for clangd (global code completion),
The format is YAML, which is only for experiment.

TEST: used the tool to generate global symbols for LLVM (~72MB).

[sammccall]

clangd/CMakeLists.txt

50

I think generally we run check-clang-tools before committing - I guess it's OK not to have tests for this experimental tool, but we should at least keep it building. Can you make check-clang-tools build it?

clangd/global-symbol-builder/GlobalSymbolBuilderMain.cpp

107

FYI D41506 changes this API, adding SymbolSlab::Builder. Shouldn't be a problem, but we'll need to merge.

113

This seems like you might end up overwriting good declarations with bad ones (e.g. forward decls). Picking one with a simple heuristic like "longest range" would get classes right.

176

I don't understand why we need to plumb callbacks through SymbolCollector.
Can't we just write the slab after this line?

clangd/global-symbol-builder/run-global-symbol-builder.py

1 ↗

(On Diff #127881)

So we now have three copies of this script...
I'm probably missing something, but what's here that's hard to do in a single binary?
I'd have to think it'd be faster if we weren't constantly spawning new processes and roundtripping all the data through YAML-on-disk.

(If this is just expedience because we have an existing template to copy, I'm happy to try to combine it into one binary myself)

[sammccall]

This looks like it does the right thing, ISTM it has a few too many moving parts.
I might be missing why they're necessary, or we might not want to bother polishing this - maybe you can give me more context offline.

[hokein]

Write all map-reduce logic into the global-symbol-builder binary, which makes
the patch much simpler.

[hokein]

Thanks for the comments.

clangd/CMakeLists.txt

50

Yeah, it builds.

clangd/global-symbol-builder/GlobalSymbolBuilderMain.cpp

107

Will do it after that patch is landed.

113

Add a FIXME here, will do it in a follow-up -- as the symbol structure would be changed.

176

ToolExecutor takes over the lifetime of FrontendActionFactory, there is no straightforward way to get the internal SymbolSlab from the indexAction.

Changed to use ClangTool, since it would simplify the code a lot, and no changes in SymbolCollector.

clangd/global-symbol-builder/run-global-symbol-builder.py

1 ↗

(On Diff #127881)

Yeah, the script is borrowed from other places for the fast prototype.

We can combine it into the global-symbol-builder binary. Done.

[sammccall]

Thanks for the simplification!
How long does it take to run over LLVM on your machine?

clangd/CMakeLists.txt

50

Can you add it to the deps of the testsuite, so we don't break it?

clangd/global-symbol-builder/GlobalSymbolBuilderMain.cpp

46

out of curiosity, what does this do?

58

you could avoid opt parsing, opening files and dealing with errors etc by just writing to llvm::outs() and running with > foo.yaml - up to you

91

if this isn't the typical use case, consider
llvm::errs() << "No compilation database found, processing individual files with flags from command-line\n"

[hokein]

Address remaining review comments.

[hokein]

In D41491#963103, @sammccall wrote:

Thanks for the simplification!
How long does it take to run over LLVM on your machine?

It took < 10 minutes (vs 20 minutes with python script) to run over LLVM.

clangd/global-symbol-builder/GlobalSymbolBuilderMain.cpp

46

No official document on this :(
IIUC, this is a filter option to control whether we get symbols from the system code (like std library). All means we index all of them.

58

Good idea. Done.

[sammccall]

clangd/global-symbol-builder/GlobalSymbolBuilderMain.cpp

46

Ah, that sounds fine. Later we may actually want to index stdlib separately.

57

(remove this and use GeneralCategory as default?)

[hokein]

Use GeneralCateogry.

clangd/CMakeLists.txt

add_clang_library(clangDaemon
   clangToolingRefactor
   ${LLVM_PTHREAD_LIB}
   )
 
 if( LLVM_LIB_FUZZING_ENGINE OR LLVM_USE_SANITIZE_COVERAGE )
   add_subdirectory(fuzzer)
 endif()
 add_subdirectory(tool)
+add_subdirectory(global-symbol-builder)

clangd/global-symbol-builder/CMakeLists.txt

+include_directories(${CMAKE_CURRENT_SOURCE_DIR}/../)
+
+set(LLVM_LINK_COMPONENTS
+    Support
+    )
+
+add_clang_executable(global-symbol-builder
+  GlobalSymbolBuilderMain.cpp
+  )
+
+target_link_libraries(global-symbol-builder
+  PRIVATE
+  clangAST
+  clangIndex
+  clangDaemon
+  clangBasic
+  clangFrontend
+  clangTooling
+)

clangd/global-symbol-builder/GlobalSymbolBuilderMain.cpp

+//===--- GlobalSymbolBuilderMain.cpp -----------------------------*- C++-*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// GlobalSymbolBuilder is a tool to generate YAML-format symbols across the
+// whole project. This tools is for **experimental** only. Don't use it in
+// production code.
+//
+//===---------------------------------------------------------------------===//
+
+#include "index/Index.h"
+#include "index/SymbolCollector.h"
+#include "index/SymbolYAML.h"
+#include "clang/Frontend/FrontendActions.h"
+#include "clang/Index/IndexingAction.h"
+#include "clang/Index/IndexDataConsumer.h"
+#include "clang/Tooling/CommonOptionsParser.h"
+#include "clang/Tooling/Tooling.h"
+#include "clang/Tooling/CommonOptionsParser.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/Signals.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/ThreadPool.h"
+
+using namespace llvm;
+using namespace clang::tooling;
+using clang::clangd::SymbolSlab;
+
+namespace clang {
+namespace clangd {
+
+class SymbolIndexActionFactory : public tooling::FrontendActionFactory {
+public:
+  SymbolIndexActionFactory() = default;
+
+  clang::FrontendAction *create() override {
+    index::IndexingOptions IndexOpts;
+    IndexOpts.SystemSymbolFilter =
+        index::IndexingOptions::SystemSymbolFilterKind::All;
+    IndexOpts.IndexFunctionLocals = false;
+    Collector = std::make_shared<SymbolCollector>();
+    return index::createIndexingAction(Collector, IndexOpts, nullptr).release();
+  }
+
+  std::shared_ptr<SymbolCollector> Collector;
+};
+
+} // namespace clangd
+} // namespace clang
+
+static cl::OptionCategory GlobalSymbolCategory("global-symbol-builder options");
+
+int main(int argc, const char **argv) {
+  llvm::sys::PrintStackTraceOnErrorSignal(argv[0]);
+
+  const char* Overview =
+      "This is an **experimental** tool to generate YAML-format "
+      "project-wide symbols for clangd (global code completion). It would be "
+      "changed and deprecated eventually. Don't use it in production code!";
+  CommonOptionsParser OptionsParser(argc, argv, GlobalSymbolCategory,
+                                    /*Overview=*/Overview);
+
+  // No compilation database found, fallback to single TU analysis, this is
+  // mainly for debugging purpose:
+  //   global-symbol-buidler /tmp/t.cc -- -std=c++11.
+  if (OptionsParser.getCompilations().getAllFiles().empty()) {
+    llvm::errs() << "No compilation database found, processing individual "
+                    "files with flags from command-line\n.";
+    ClangTool Tool(OptionsParser.getCompilations(),
+                   OptionsParser.getSourcePathList());
+    clang::clangd::SymbolIndexActionFactory IndexAction;
+    Tool.run(&IndexAction);
+    llvm::outs() << SymbolToYAML(IndexAction.Collector->takeSymbols());
+    return 0;
+  }
+
+  // Found compilation database, we iterate all TUs from database to get all
+  // symbols, and then merge them into a single SymbolSlab.
+  SymbolSlab GlobalSymbols;
+  std::mutex SymbolMutex;
+  auto AddSymbols = [&](const SymbolSlab& NewSymbols) {
+    // Synchronize set accesses.
+    std::unique_lock<std::mutex> LockGuard(SymbolMutex);
+    for (auto It : NewSymbols) {
+      // FIXME: Better handling the overlap symbols, currently we overwrite it
+      // with the latest one, but we always want to good declarations (class
+      // definitions, instead of forward declarations).
+      GlobalSymbols.insert(It.second);
+    }
+  };
+
+  {
+    llvm::ThreadPool Pool;
+    for (auto& file : OptionsParser.getCompilations().getAllFiles()) {
+      Pool.async([&OptionsParser, &AddSymbols](llvm::StringRef Path) {
+        ClangTool Tool(OptionsParser.getCompilations(), {Path});
+        clang::clangd::SymbolIndexActionFactory IndexAction;
+        Tool.run(&IndexAction);
+        AddSymbols(IndexAction.Collector->takeSymbols());
+      }, file);
+    }
+  }
+
+  llvm::outs() << SymbolToYAML(GlobalSymbols);
+  return 0;
+}

test/CMakeLists.txt

set(CLANG_TOOLS_TEST_DEPS
   clang-include-fixer
   clang-move
   clang-query
   clang-reorder-fields
   find-all-symbols
   modularize
   pp-trace
 
+  # These indiviual tools have no tests, add them here to make them compile
+  # together with chec-clang-tools, so that we won't break them in the future.
+  global-symbol-builder
   # Unit tests
   ExtraToolsUnitTests
   )
 
 if(CLANG_ENABLE_STATIC_ANALYZER)
   list(APPEND CLANG_TOOLS_TEST_DEPS
     # For the clang-tidy libclang integration test.
     c-index-test