[clangd] Add a tool to build YAML-format global symbols.

[hokein]

Summary

The tools is used to generate global symbols for clangd (global code completion),
The format is YAML, which is only for experiment.

TEST: used the tool to generate global symbols for LLVM (~72MB).

[sammccall]

clangd/CMakeLists.txt

50 ↗

(On Diff #127881)

I think generally we run check-clang-tools before committing - I guess it's OK not to have tests for this experimental tool, but we should at least keep it building. Can you make check-clang-tools build it?

clangd/global-symbol-builder/GlobalSymbolBuilderMain.cpp

106 ↗

(On Diff #127881)

FYI D41506 changes this API, adding SymbolSlab::Builder. Shouldn't be a problem, but we'll need to merge.

112 ↗

(On Diff #127881)

This seems like you might end up overwriting good declarations with bad ones (e.g. forward decls). Picking one with a simple heuristic like "longest range" would get classes right.

175 ↗

(On Diff #127881)

I don't understand why we need to plumb callbacks through SymbolCollector.
Can't we just write the slab after this line?

clangd/global-symbol-builder/run-global-symbol-builder.py

1 ↗

(On Diff #127881)

So we now have three copies of this script...
I'm probably missing something, but what's here that's hard to do in a single binary?
I'd have to think it'd be faster if we weren't constantly spawning new processes and roundtripping all the data through YAML-on-disk.

(If this is just expedience because we have an existing template to copy, I'm happy to try to combine it into one binary myself)

[sammccall]

This looks like it does the right thing, ISTM it has a few too many moving parts.
I might be missing why they're necessary, or we might not want to bother polishing this - maybe you can give me more context offline.

[hokein]

Write all map-reduce logic into the global-symbol-builder binary, which makes
the patch much simpler.

[hokein]

Thanks for the comments.

clangd/CMakeLists.txt

50 ↗

(On Diff #127881)

Yeah, it builds.

clangd/global-symbol-builder/GlobalSymbolBuilderMain.cpp

106 ↗

(On Diff #127881)

Will do it after that patch is landed.

112 ↗

(On Diff #127881)

Add a FIXME here, will do it in a follow-up -- as the symbol structure would be changed.

175 ↗

(On Diff #127881)

ToolExecutor takes over the lifetime of FrontendActionFactory, there is no straightforward way to get the internal SymbolSlab from the indexAction.

Changed to use ClangTool, since it would simplify the code a lot, and no changes in SymbolCollector.

clangd/global-symbol-builder/run-global-symbol-builder.py

1 ↗

(On Diff #127881)

Yeah, the script is borrowed from other places for the fast prototype.

We can combine it into the global-symbol-builder binary. Done.

[sammccall]

Thanks for the simplification!
How long does it take to run over LLVM on your machine?

clangd/CMakeLists.txt

50 ↗

(On Diff #127881)

Can you add it to the deps of the testsuite, so we don't break it?

clangd/global-symbol-builder/GlobalSymbolBuilderMain.cpp

45 ↗

(On Diff #128010)

out of curiosity, what does this do?

57 ↗

(On Diff #128010)

you could avoid opt parsing, opening files and dealing with errors etc by just writing to llvm::outs() and running with > foo.yaml - up to you

90 ↗

(On Diff #128010)

if this isn't the typical use case, consider
llvm::errs() << "No compilation database found, processing individual files with flags from command-line\n"

[hokein]

Address remaining review comments.

[hokein]

In D41491#963103, @sammccall wrote:

Thanks for the simplification!
How long does it take to run over LLVM on your machine?

It took < 10 minutes (vs 20 minutes with python script) to run over LLVM.

clangd/global-symbol-builder/GlobalSymbolBuilderMain.cpp

45 ↗

(On Diff #128010)

No official document on this :(
IIUC, this is a filter option to control whether we get symbols from the system code (like std library). All means we index all of them.

57 ↗

(On Diff #128010)

Good idea. Done.

[sammccall]

clangd/global-symbol-builder/GlobalSymbolBuilderMain.cpp

57 ↗

(On Diff #128020)

(remove this and use GeneralCategory as default?)

45 ↗

(On Diff #128010)

Ah, that sounds fine. Later we may actually want to index stdlib separately.

[hokein]

Use GeneralCateogry.

clang-tools-extra/trunk/clangd/CMakeLists.txt

add_clang_library(clangDaemon
   clangToolingRefactor
   ${LLVM_PTHREAD_LIB}
   )
 
 if( LLVM_LIB_FUZZING_ENGINE OR LLVM_USE_SANITIZE_COVERAGE )
   add_subdirectory(fuzzer)
 endif()
 add_subdirectory(tool)
+add_subdirectory(global-symbol-builder)

clang-tools-extra/trunk/clangd/global-symbol-builder/CMakeLists.txt

+include_directories(${CMAKE_CURRENT_SOURCE_DIR}/../)
+
+set(LLVM_LINK_COMPONENTS
+    Support
+    )
+
+add_clang_executable(global-symbol-builder
+  GlobalSymbolBuilderMain.cpp
+  )
+
+target_link_libraries(global-symbol-builder
+  PRIVATE
+  clangAST
+  clangIndex
+  clangDaemon
+  clangBasic
+  clangFrontend
+  clangTooling
+)

clang-tools-extra/trunk/clangd/global-symbol-builder/GlobalSymbolBuilderMain.cpp

+//===--- GlobalSymbolBuilderMain.cpp -----------------------------*- C++-*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// GlobalSymbolBuilder is a tool to generate YAML-format symbols across the
+// whole project. This tools is for **experimental** only. Don't use it in
+// production code.
+//
+//===---------------------------------------------------------------------===//
+
+#include "index/Index.h"
+#include "index/SymbolCollector.h"
+#include "index/SymbolYAML.h"
+#include "clang/Frontend/FrontendActions.h"
+#include "clang/Index/IndexingAction.h"
+#include "clang/Index/IndexDataConsumer.h"
+#include "clang/Tooling/CommonOptionsParser.h"
+#include "clang/Tooling/Tooling.h"
+#include "clang/Tooling/CommonOptionsParser.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/Signals.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/ThreadPool.h"
+
+using namespace llvm;
+using namespace clang::tooling;
+using clang::clangd::SymbolSlab;
+
+namespace clang {
+namespace clangd {
+
+class SymbolIndexActionFactory : public tooling::FrontendActionFactory {
+public:
+  SymbolIndexActionFactory() = default;
+
+  clang::FrontendAction *create() override {
+    index::IndexingOptions IndexOpts;
+    IndexOpts.SystemSymbolFilter =
+        index::IndexingOptions::SystemSymbolFilterKind::All;
+    IndexOpts.IndexFunctionLocals = false;
+    Collector = std::make_shared<SymbolCollector>();
+    return index::createIndexingAction(Collector, IndexOpts, nullptr).release();
+  }
+
+  std::shared_ptr<SymbolCollector> Collector;
+};
+
+} // namespace clangd
+} // namespace clang
+
+int main(int argc, const char **argv) {
+  llvm::sys::PrintStackTraceOnErrorSignal(argv[0]);
+
+  const char* Overview =
+      "This is an **experimental** tool to generate YAML-format "
+      "project-wide symbols for clangd (global code completion). It would be "
+      "changed and deprecated eventually. Don't use it in production code!";
+  CommonOptionsParser OptionsParser(argc, argv, cl::GeneralCategory,
+                                    /*Overview=*/Overview);
+
+  // No compilation database found, fallback to single TU analysis, this is
+  // mainly for debugging purpose:
+  //   global-symbol-buidler /tmp/t.cc -- -std=c++11.
+  if (OptionsParser.getCompilations().getAllFiles().empty()) {
+    llvm::errs() << "No compilation database found, processing individual "
+                    "files with flags from command-line\n.";
+    ClangTool Tool(OptionsParser.getCompilations(),
+                   OptionsParser.getSourcePathList());
+    clang::clangd::SymbolIndexActionFactory IndexAction;
+    Tool.run(&IndexAction);
+    llvm::outs() << SymbolToYAML(IndexAction.Collector->takeSymbols());
+    return 0;
+  }
+
+  // Found compilation database, we iterate all TUs from database to get all
+  // symbols, and then merge them into a single SymbolSlab.
+  SymbolSlab GlobalSymbols;
+  std::mutex SymbolMutex;
+  auto AddSymbols = [&](const SymbolSlab& NewSymbols) {
+    // Synchronize set accesses.
+    std::unique_lock<std::mutex> LockGuard(SymbolMutex);
+    for (auto It : NewSymbols) {
+      // FIXME: Better handling the overlap symbols, currently we overwrite it
+      // with the latest one, but we always want to good declarations (class
+      // definitions, instead of forward declarations).
+      GlobalSymbols.insert(It.second);
+    }
+  };
+
+  {
+    llvm::ThreadPool Pool;
+    for (auto& file : OptionsParser.getCompilations().getAllFiles()) {
+      Pool.async([&OptionsParser, &AddSymbols](llvm::StringRef Path) {
+        ClangTool Tool(OptionsParser.getCompilations(), {Path});
+        clang::clangd::SymbolIndexActionFactory IndexAction;
+        Tool.run(&IndexAction);
+        AddSymbols(IndexAction.Collector->takeSymbols());
+      }, file);
+    }
+  }
+
+  llvm::outs() << SymbolToYAML(GlobalSymbols);
+  return 0;
+}

clang-tools-extra/trunk/test/CMakeLists.txt

set(CLANG_TOOLS_TEST_DEPS
   clang-include-fixer
   clang-move
   clang-query
   clang-reorder-fields
   find-all-symbols
   modularize
   pp-trace
 
+  # These individual tools have no tests, add them here to make them compile
+  # together with check-clang-tools, so that we won't break them in the future.
+  global-symbol-builder
+
   # Unit tests
   ExtraToolsUnitTests
   )
 
 if(CLANG_ENABLE_STATIC_ANALYZER)
   list(APPEND CLANG_TOOLS_TEST_DEPS
     # For the clang-tidy libclang integration test.
     c-index-test