[clangd] (take 2) Try harder to find a plausible `clang` as argv0, particularly on Mac.

[sammccall]

Summary

This was originally committed in 88bccded8fa169481fa367debf5ec615640635a1,
and reverted in 93f77617abba512d2861e2fc50ce385883f587b6.

This version is now much more testable: the "detect toolchain properties" part
is still not tested but also not active in tests.
All the command manipulation based on the detected properties is
directly tested, and also not active in other tests.

Fixes https://github.com/clangd/clangd/issues/211
Fixes https://github.com/clangd/clangd/issues/178

[sammccall]

@kbobyrev would you mind verifying this version of the patch also works?
I don't expect changes vs the last one: it's been restructured a lot but the logic should be the same.

[sammccall]

Revert unneeded changes to tests.

[merge_guards_bot]

Build result: fail - 60479 tests passed, 1 failed and 726 were skipped.

failed: libc++.std/thread/thread_mutex/thread_mutex_requirements/thread_mutex_requirements_mutex/thread_mutex_recursive/lock.pass.cpp

Log files: console-log.txt, CMakeCache.txt

[merge_guards_bot]

Build result: pass - 60480 tests passed, 0 failed and 726 were skipped.

Log files: console-log.txt, CMakeCache.txt

[jkorous]

The use of xcrun looks sound to me.

clang-tools-extra/clangd/CompileCommands.cpp

54

The trim() is probably safe - just wondering if that could cause issues in some weird case.

66

Just wondering why you decided to not somehow propagate the error - are there scenarios where the path is correct yet not resolvable on real FS?

92

Maybe this warrants a mention in the header file?

[sammccall]

Address review comments

[sammccall]

Thanks for looking at this Jan, I should have put you on the reviewers in the first place...

clang-tools-extra/clangd/CompileCommands.cpp

54

Only if the path ends with a space, AFAICS. I don't think it's worth worrying about.

66

By definition probably not *correct* cases, but this will be hit if /usr/bin/clang is a dangling symlink, or xcrun lies to us, etc.

There's no error *propagation* because error recovery is here: even if we can't work out whether clang is a symlink, the best thing to do is create a mangler that assumes it isn't. (Or maybe better to create a mangler that ignores this path entirely, but I'm not convinced).
There's no value in knowing about this failure at the site where the mangler is created, or where the compile command is mangled, or where the file is opened - we're always going to want to recover and forge ahead, and we have the best information to recover here.

[sammccall]

clang-tools-extra/clangd/CompileCommands.cpp

66

(Forgot to mention: I added a log here, since this shouldn't happen.)

[merge_guards_bot]

Build result: pass - 60480 tests passed, 0 failed and 726 were skipped.

Log files: console-log.txt, CMakeCache.txt

[kbobyrev]

In D71029#1769416, @sammccall wrote:

@kbobyrev would you mind verifying this version of the patch also works?
I don't expect changes vs the last one: it's been restructured a lot but the logic should be the same.

I can confirm that the latest version of the patch works.

[sammccall]

Thanks. Can someone approve the patch and I'll land it again :-)

[ilya-biryukov]

LGTM

[thakis]

Breaks Win: http://45.33.8.238/win/3549/step_7.txt

clang-tools-extra/clangd/CMakeLists.txt

 
 add_clang_library(clangDaemon
   AST.cpp
   Cancellation.cpp
   ClangdLSPServer.cpp
   ClangdServer.cpp
   CodeComplete.cpp
   CodeCompletionStrings.cpp
+  CompileCommands.cpp
   Compiler.cpp
   Context.cpp
   Diagnostics.cpp
   DraftStore.cpp
   ExpectedTypes.cpp
   FindSymbols.cpp
   FindTarget.cpp
   FileDistance.cpp

clang-tools-extra/clangd/ClangdLSPServer.cpp

if (const auto &Dir = Params.initializationOptions.compilationDatabasePath)
     CompileCommandsDir = Dir;
   if (UseDirBasedCDB) {
     BaseCDB = std::make_unique<DirectoryBasedGlobalCompilationDatabase>(
         CompileCommandsDir);
     BaseCDB = getQueryDriverDatabase(
         llvm::makeArrayRef(ClangdServerOpts.QueryDriverGlobs),
         std::move(BaseCDB));
   }
+  auto Mangler = CommandMangler::detect();
+  if (ClangdServerOpts.ResourceDir)
+    Mangler.ResourceDir = *ClangdServerOpts.ResourceDir;
   CDB.emplace(BaseCDB.get(), Params.initializationOptions.fallbackFlags,
-              ClangdServerOpts.ResourceDir);
+              tooling::ArgumentsAdjuster(Mangler));
   {
     // Switch caller's context with LSPServer's background context. Since we
     // rather want to propagate information from LSPServer's context into the
     // Server, CDB, etc.
     WithContext MainContext(BackgroundContext.clone());
     llvm::Optional<WithContextValue> WithOffsetEncoding;
     if (NegotiatedOffsetEncoding)
       WithOffsetEncoding.emplace(kCurrentOffsetEncoding,

clang-tools-extra/clangd/CompileCommands.h

+//===--- CompileCommands.h - Manipulation of compile flags -------*- C++-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Tooling/ArgumentsAdjusters.h"
+#include "clang/Tooling/CompilationDatabase.h"
+#include <string>
+#include <vector>
+
+namespace clang {
+namespace clangd {
+
+// CommandMangler transforms compile commands from some external source
+// for use in clangd. This means:
+//  - running the frontend only, stripping args regarding output files etc
+//  - forcing the use of clangd's builtin headers rather than clang's
+//  - resolving argv0 as cc1 expects
+//  - injecting -isysroot flags on mac as the system clang does
+struct CommandMangler {
+  // Absolute path to clang.
+  llvm::Optional<std::string> ClangPath;
+  // Directory containing builtin headers.
+  llvm::Optional<std::string> ResourceDir;
+  // Root for searching for standard library (passed to -isysroot).
+  llvm::Optional<std::string> Sysroot;
+
+  // A command-mangler that doesn't know anything about the system.
+  // This is hermetic for unit-tests, but won't work well in production.
+  static CommandMangler forTests();
+  // Probe the system and build a command-mangler that knows the toolchain.
+  //  - try to find clang on $PATH, otherwise fake a path near clangd
+  //  - find the resource directory installed near clangd
+  //  - on mac, find clang and isysroot by querying the `xcrun` launcher
+  static CommandMangler detect();
+
+  void adjust(std::vector<std::string> &Cmd) const;
+  explicit operator clang::tooling::ArgumentsAdjuster();
+
+private:
+  CommandMangler() = default;
+};
+
+} // namespace clangd
+} // namespace clang

clang-tools-extra/clangd/CompileCommands.cpp

+//===--- CompileCommands.cpp ----------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "CompileCommands.h"
+#include "Logger.h"
+#include "clang/Frontend/CompilerInvocation.h"
+#include "clang/Tooling/ArgumentsAdjusters.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/FileUtilities.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/Program.h"
+
+namespace clang {
+namespace clangd {
+namespace {
+
+// Query apple's `xcrun` launcher, which is the source of truth for "how should"
+// clang be invoked on this system.
+llvm::Optional<std::string> queryXcrun(llvm::ArrayRef<llvm::StringRef> Argv) {
+  auto Xcrun = llvm::sys::findProgramByName("xcrun");
+  if (!Xcrun) {
+    log("Couldn't find xcrun. Hopefully you have a non-apple toolchain...");
+    return llvm::None;
+  }
+  llvm::SmallString<64> OutFile;
+  llvm::sys::fs::createTemporaryFile("clangd-xcrun", "", OutFile);
+  llvm::FileRemover OutRemover(OutFile);
+  llvm::Optional<llvm::StringRef> Redirects[3] = {
+      /*stdin=*/{""}, /*stdout=*/{OutFile}, /*stderr=*/{""}};
+  vlog("Invoking {0} to find clang installation", *Xcrun);
+  int Ret = llvm::sys::ExecuteAndWait(*Xcrun, Argv,
+                                      /*Env=*/llvm::None, Redirects,
+                                      /*SecondsToWait=*/10);
+  if (Ret != 0) {
+    log("xcrun exists but failed with code {0}. "
+        "If you have a non-apple toolchain, this is OK. "
+        "Otherwise, try xcode-select --install.",
+        Ret);
+    return llvm::None;
+  }
+
+  auto Buf = llvm::MemoryBuffer::getFile(OutFile);
+  if (!Buf) {
+    log("Can't read xcrun output: {0}", Buf.getError().message());
+    return llvm::None;
+  }
+  StringRef Path = Buf->get()->getBuffer().trim();
+  if (Path.empty()) {
+    log("xcrun produced no output");
+    return llvm::None;
+  }
+  return Path.str();
+}
+
+// Resolve symlinks if possible.
+std::string resolve(std::string Path) {
+  llvm::SmallString<128> Resolved;
+  if (llvm::sys::fs::real_path(Path, Resolved)) {
+    log("Failed to resolve possible symlink {0}", Path);
+    return Path;
+  }
+  return Resolved.str();
+}
+
+// Get a plausible full `clang` path.
+// This is used in the fallback compile command, or when the CDB returns a
+// generic driver with no path.
+std::string detectClangPath() {
+  // The driver and/or cc1 sometimes depend on the binary name to compute
+  // useful things like the standard library location.
+  // We need to emulate what clang on this system is likely to see.
+  // cc1 in particular looks at the "real path" of the running process, and
+  // so if /usr/bin/clang is a symlink, it sees the resolved path.
+  // clangd doesn't have that luxury, so we resolve symlinks ourselves.
+
+  // On Mac, `which clang` is /usr/bin/clang. It runs `xcrun clang`, which knows
+  // where the real clang is kept. We need to do the same thing,
+  // because cc1 (not the driver!) will find libc++ relative to argv[0].
+#ifdef __APPLE__
+  if (auto MacClang = queryXcrun({"xcrun", "--find", "clang"}))
+    return resolve(std::move(*MacClang));
+#endif
+  // On other platforms, just look for compilers on the PATH.
+  for (const char *Name : {"clang", "gcc", "cc"})
+    if (auto PathCC = llvm::sys::findProgramByName(Name))
+      return resolve(std::move(*PathCC));
+  // Fallback: a nonexistent 'clang' binary next to clangd.
+  static int Dummy;
+  std::string ClangdExecutable =
+      llvm::sys::fs::getMainExecutable("clangd", (void *)&Dummy);
+  SmallString<128> ClangPath;
+  ClangPath = llvm::sys::path::parent_path(ClangdExecutable);
+  llvm::sys::path::append(ClangPath, "clang");
+  return ClangPath.str();
+}
+
+// On mac, /usr/bin/clang sets SDKROOT and then invokes the real clang.
+// The effect of this is to set -isysroot correctly. We do the same.
+const llvm::Optional<std::string> detectSysroot() {
+#ifndef __APPLE__
+  return llvm::None;
+#endif
+
+  // SDKROOT overridden in environment, respect it. Driver will set isysroot.
+  if (::getenv("SDKROOT"))
+    return llvm::None;
+  return queryXcrun({"xcrun", "--show-sdk-path"});
+  return llvm::None;
+}
+
+std::string detectStandardResourceDir() {
+  static int Dummy; // Just an address in this process.
+  return CompilerInvocation::GetResourcesPath("clangd", (void *)&Dummy);
+}
+
+} // namespace
+
+CommandMangler CommandMangler::detect() {
+  CommandMangler Result;
+  Result.ClangPath = detectClangPath();
+  Result.ResourceDir = detectStandardResourceDir();
+  Result.Sysroot = detectSysroot();
+  return Result;
+}
+
+CommandMangler CommandMangler::forTests() {
+  return CommandMangler();
+}
+
+void CommandMangler::adjust(std::vector<std::string> &Cmd) const {
+  // Check whether the flag exists, either as -flag or -flag=*
+  auto Has = [&](llvm::StringRef Flag) {
+    for (llvm::StringRef Arg : Cmd) {
+      if (Arg.consume_front(Flag) && (Arg.empty() || Arg[0] == '='))
+        return true;
+    }
+    return false;
+  };
+
+  // clangd should not write files to disk, including dependency files
+  // requested on the command line.
+  Cmd = tooling::getClangStripDependencyFileAdjuster()(Cmd, "");
+  // Strip plugin related command line arguments. Clangd does
+  // not support plugins currently. Therefore it breaks if
+  // compiler tries to load plugins.
+  Cmd = tooling::getStripPluginsAdjuster()(Cmd, "");
+  Cmd = tooling::getClangSyntaxOnlyAdjuster()(Cmd, "");
+
+  if (ResourceDir && !Has("-resource-dir"))
+    Cmd.push_back(("-resource-dir=" + *ResourceDir));
+
+  if (Sysroot && !Has("-isysroot")) {
+    Cmd.push_back("-isysroot");
+    Cmd.push_back(*Sysroot);
+  }
+
+  // If the driver is a generic name like "g++" with no path, add a clang path.
+  // This makes it easier for us to find the standard libraries on mac.
+  if (ClangPath && llvm::sys::path::is_absolute(*ClangPath) && !Cmd.empty()) {
+    std::string &Driver = Cmd.front();
+    if (Driver == "clang" || Driver == "clang++" || Driver == "gcc" ||
+        Driver == "g++" || Driver == "cc" || Driver == "c++") {
+      llvm::SmallString<128> QualifiedDriver =
+          llvm::sys::path::parent_path(*ClangPath);
+      llvm::sys::path::append(QualifiedDriver, Driver);
+      Driver = QualifiedDriver.str();
+    }
+  }
+}
+
+CommandMangler::operator clang::tooling::ArgumentsAdjuster() {
+  return [Mangler{*this}](const std::vector<std::string> &Args,
+                          llvm::StringRef File) {
+    auto Result = Args;
+    Mangler.adjust(Result);
+    return Result;
+  };
+}
+
+} // namespace clangd
+} // namespace clang

clang-tools-extra/clangd/GlobalCompilationDatabase.h

 //===--- GlobalCompilationDatabase.h -----------------------------*- C++-*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_GLOBALCOMPILATIONDATABASE_H
 #define LLVM_CLANG_TOOLS_EXTRA_CLANGD_GLOBALCOMPILATIONDATABASE_H
 
+#include "CompileCommands.h"
 #include "Function.h"
 #include "Path.h"
+#include "clang/Tooling/ArgumentsAdjusters.h"
 #include "clang/Tooling/CompilationDatabase.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/StringMap.h"
 #include <memory>
 #include <mutex>
 #include <vector>
 
 namespace clang {
-
 namespace clangd {
 
 class Logger;
 
 struct ProjectInfo {
   // The directory in which the compilation database was discovered.
   // Empty if directory-based compilation database discovery was not used.
   std::string SourceRoot;

 
 /// Extracts system include search path from drivers matching QueryDriverGlobs
 /// and adds them to the compile flags. Base may not be nullptr.
 /// Returns Base when \p QueryDriverGlobs is empty.
 std::unique_ptr<GlobalCompilationDatabase>
 getQueryDriverDatabase(llvm::ArrayRef<std::string> QueryDriverGlobs,
                        std::unique_ptr<GlobalCompilationDatabase> Base);
 
+
 /// Wraps another compilation database, and supports overriding the commands
 /// using an in-memory mapping.
 class OverlayCDB : public GlobalCompilationDatabase {
 public:
   // Base may be null, in which case no entries are inherited.
   // FallbackFlags are added to the fallback compile command.
+  // Adjuster is applied to all commands, fallback or not.
   OverlayCDB(const GlobalCompilationDatabase *Base,
              std::vector<std::string> FallbackFlags = {},
-             llvm::Optional<std::string> ResourceDir = llvm::None);
+             tooling::ArgumentsAdjuster Adjuster = nullptr);
 
   llvm::Optional<tooling::CompileCommand>
   getCompileCommand(PathRef File) const override;
   tooling::CompileCommand getFallbackCommand(PathRef File) const override;
   llvm::Optional<ProjectInfo> getProjectInfo(PathRef File) const override;
 
   /// Sets or clears the compilation command for a particular file.
   void
   setCompileCommand(PathRef File,
                     llvm::Optional<tooling::CompileCommand> CompilationCommand);
 
 private:
   mutable std::mutex Mutex;
   llvm::StringMap<tooling::CompileCommand> Commands; /* GUARDED_BY(Mut) */
   const GlobalCompilationDatabase *Base;
-  std::string ResourceDir;
+  tooling::ArgumentsAdjuster ArgsAdjuster;
   std::vector<std::string> FallbackFlags;
   CommandChanged::Subscription BaseChanged;
 };
 
 } // namespace clangd
 } // namespace clang
 
 #endif // LLVM_CLANG_TOOLS_EXTRA_CLANGD_GLOBALCOMPILATIONDATABASE_H

clang-tools-extra/clangd/GlobalCompilationDatabase.cpp

 #include "clang/Frontend/CompilerInvocation.h"
 #include "clang/Tooling/ArgumentsAdjusters.h"
 #include "clang/Tooling/CompilationDatabase.h"
 #include "llvm/ADT/None.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/Support/FileSystem.h"
+#include "llvm/Support/FileUtilities.h"
 #include "llvm/Support/Path.h"
+#include "llvm/Support/Program.h"
 #include <string>
 #include <tuple>
 #include <vector>
 
 namespace clang {
 namespace clangd {
 namespace {
 
-void adjustArguments(tooling::CompileCommand &Cmd,
-                     llvm::StringRef ResourceDir) {
-  tooling::ArgumentsAdjuster ArgsAdjuster = tooling::combineAdjusters(
-      // clangd should not write files to disk, including dependency files
-      // requested on the command line.
-      tooling::getClangStripDependencyFileAdjuster(),
-      // Strip plugin related command line arguments. Clangd does
-      // not support plugins currently. Therefore it breaks if
-      // compiler tries to load plugins.
-      tooling::combineAdjusters(tooling::getStripPluginsAdjuster(),
-                                tooling::getClangSyntaxOnlyAdjuster()));
-
-  Cmd.CommandLine = ArgsAdjuster(Cmd.CommandLine, Cmd.Filename);
-  // Inject the resource dir.
-  // FIXME: Don't overwrite it if it's already there.
-  if (!ResourceDir.empty())
-    Cmd.CommandLine.push_back(("-resource-dir=" + ResourceDir).str());
-}
-
-std::string getStandardResourceDir() {
-  static int Dummy; // Just an address in this process.
-  return CompilerInvocation::GetResourcesPath("clangd", (void *)&Dummy);
-}
-
 // Runs the given action on all parent directories of filename, starting from
 // deepest directory and going up to root. Stops whenever action succeeds.
 void actOnAllParentDirectories(PathRef FileName,
                                llvm::function_ref<bool(PathRef)> Action) {
   for (auto Path = llvm::sys::path::parent_path(FileName);
        !Path.empty() && !Action(Path);
        Path = llvm::sys::path::parent_path(Path))
     ;
 }
 
 } // namespace
 
-static std::string getFallbackClangPath() {
-  static int Dummy;
-  std::string ClangdExecutable =
-      llvm::sys::fs::getMainExecutable("clangd", (void *)&Dummy);
-  SmallString<128> ClangPath;
-  ClangPath = llvm::sys::path::parent_path(ClangdExecutable);
-  llvm::sys::path::append(ClangPath, "clang");
-  return ClangPath.str();
-}
-
 tooling::CompileCommand
 GlobalCompilationDatabase::getFallbackCommand(PathRef File) const {
-  std::vector<std::string> Argv = {getFallbackClangPath()};
+  std::vector<std::string> Argv = {"clang"};
   // Clang treats .h files as C by default and files without extension as linker
   // input, resulting in unhelpful diagnostics.
   // Parsing as Objective C++ is friendly to more cases.
   auto FileExtension = llvm::sys::path::extension(File);
   if (FileExtension.empty() || FileExtension == ".h")
     Argv.push_back("-xobjective-c++-header");
   Argv.push_back(File);
   tooling::CompileCommand Cmd(llvm::sys::path::parent_path(File),
DirectoryBasedGlobalCompilationDatabase::getProjectInfo(PathRef File) const {
   auto Res = lookupCDB(Req);
   if (!Res)
     return llvm::None;
   return Res->PI;
 }
 
 OverlayCDB::OverlayCDB(const GlobalCompilationDatabase *Base,
                        std::vector<std::string> FallbackFlags,
-                       llvm::Optional<std::string> ResourceDir)
-    : Base(Base), ResourceDir(ResourceDir ? std::move(*ResourceDir)
-                                          : getStandardResourceDir()),
+                       tooling::ArgumentsAdjuster Adjuster)
+    : Base(Base), ArgsAdjuster(std::move(Adjuster)),
       FallbackFlags(std::move(FallbackFlags)) {
   if (Base)
     BaseChanged = Base->watch([this](const std::vector<std::string> Changes) {
       OnCommandChanged.broadcast(Changes);
     });
 }
 
 llvm::Optional<tooling::CompileCommand>
 OverlayCDB::getCompileCommand(PathRef File) const {
   llvm::Optional<tooling::CompileCommand> Cmd;
   {
     std::lock_guard<std::mutex> Lock(Mutex);
     auto It = Commands.find(removeDots(File));
     if (It != Commands.end())
       Cmd = It->second;
   }
   if (!Cmd && Base)
     Cmd = Base->getCompileCommand(File);
   if (!Cmd)
     return llvm::None;
-  adjustArguments(*Cmd, ResourceDir);
+  if (ArgsAdjuster)
+    Cmd->CommandLine = ArgsAdjuster(Cmd->CommandLine, Cmd->Filename);
   return Cmd;
 }
 
 tooling::CompileCommand OverlayCDB::getFallbackCommand(PathRef File) const {
   auto Cmd = Base ? Base->getFallbackCommand(File)
                   : GlobalCompilationDatabase::getFallbackCommand(File);
   std::lock_guard<std::mutex> Lock(Mutex);
   Cmd.CommandLine.insert(Cmd.CommandLine.end(), FallbackFlags.begin(),
                          FallbackFlags.end());
-  adjustArguments(Cmd, ResourceDir);
+  if (ArgsAdjuster)
+    Cmd.CommandLine = ArgsAdjuster(Cmd.CommandLine, Cmd.Filename);
   return Cmd;
 }
 
 void OverlayCDB::setCompileCommand(
     PathRef File, llvm::Optional<tooling::CompileCommand> Cmd) {
   // We store a canonical version internally to prevent mismatches between set
   // and get compile commands. Also it assures clients listening to broadcasts
   // doesn't receive different names for the same file.

clang-tools-extra/clangd/unittests/BackgroundIndexTests.cpp

TEST_F(BackgroundIndexTest, UncompilableFiles) {
   }
 }
 
 TEST_F(BackgroundIndexTest, CmdLineHash) {
   MockFSProvider FS;
   llvm::StringMap<std::string> Storage;
   size_t CacheHits = 0;
   MemoryShardStorage MSS(Storage, CacheHits);
-  OverlayCDB CDB(/*Base=*/nullptr, /*FallbackFlags=*/{},
-                 /*ResourceDir=*/std::string(""));
+  OverlayCDB CDB(/*Base=*/nullptr);
   BackgroundIndex Idx(Context::empty(), FS, CDB,
                       [&](llvm::StringRef) { return &MSS; });
 
   tooling::CompileCommand Cmd;
   FS.Files[testPath("A.cc")] = "#include \"A.h\"";
   FS.Files[testPath("A.h")] = "";
   Cmd.Filename = "../A.cc";
   Cmd.Directory = testPath("build");

clang-tools-extra/clangd/unittests/CMakeLists.txt

add_unittest(ClangdUnitTests ClangdTests
   ASTTests.cpp
   BackgroundIndexTests.cpp
   CancellationTests.cpp
   CanonicalIncludesTests.cpp
   ClangdTests.cpp
   CodeCompleteTests.cpp
   CodeCompletionStringsTests.cpp
   CollectMacrosTests.cpp
+  CompileCommandsTests.cpp
   ContextTests.cpp
   DexTests.cpp
   DiagnosticsTests.cpp
   DraftStoreTests.cpp
   ExpectedTypeTest.cpp
   FileDistanceTests.cpp
   FileIndexTests.cpp
   FindSymbolsTests.cpp

clang-tools-extra/clangd/unittests/ClangdTests.cpp

auto Completions = cantFail(runCodeComplete(Server, SourcePath, Code.point(),
                                               clangd::CodeCompleteOptions()))
                          .Completions;
   EXPECT_EQ(CountStats["foo.h"], Before);
   EXPECT_THAT(Completions,
               ElementsAre(Field(&CodeCompletion::Name, "TestSym")));
 }
 #endif
 
-TEST_F(ClangdVFSTest, FlagsWithPlugins) {
-  MockFSProvider FS;
-  ErrorCheckingDiagConsumer DiagConsumer;
-  MockCompilationDatabase CDB;
-  CDB.ExtraClangFlags = {
-      "-Xclang",
-      "-add-plugin",
-      "-Xclang",
-      "random-plugin",
-  };
-  OverlayCDB OCDB(&CDB);
-  ClangdServer Server(OCDB, FS, DiagConsumer, ClangdServer::optsForTest());
-
-  auto FooCpp = testPath("foo.cpp");
-  const auto SourceContents = "int main() { return 0; }";
-  FS.Files[FooCpp] = FooCpp;
-  Server.addDocument(FooCpp, SourceContents);
-  auto Result = dumpASTWithoutMemoryLocs(Server, FooCpp);
-  EXPECT_TRUE(Server.blockUntilIdleForTest()) << "Waiting for diagnostics";
-  EXPECT_NE(Result, "<no-ast>");
-}
-
 TEST_F(ClangdVFSTest, FallbackWhenPreambleIsNotReady) {
   MockFSProvider FS;
   ErrorCheckingDiagConsumer DiagConsumer;
   MockCompilationDatabase CDB;
   ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
 
   auto FooCpp = testPath("foo.cpp");
   Annotations Code(R"cpp(

clang-tools-extra/clangd/unittests/CompileCommandsTests.cpp

+//===-- CompileCommandsTests.cpp ------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "CompileCommands.h"
+
+#include "llvm/ADT/StringExtras.h"
+
+#include "gmock/gmock.h"
+#include "gtest/gtest.h"
+
+namespace clang {
+namespace clangd {
+namespace {
+
+using ::testing::Contains;
+using ::testing::ElementsAre;
+using ::testing::HasSubstr;
+using ::testing::Not;
+
+// Sadly, CommandMangler::detect(), which contains much of the logic, is
+// a bunch of untested integration glue. We test the string manipulation here
+// assuming its results are correct.
+
+// Make use of all features and assert the exact command we get out.
+// Other tests just verify presence/absence of certain args.
+TEST(CommandMangler, Everything) {
+  auto Mangler = CommandMangler::forTests();
+  Mangler.ClangPath = "/fake/bin/clang";
+  Mangler.ResourceDir = "/fake/resources";
+  Mangler.Sysroot = "/fake/sysroot";
+  std::vector<std::string> Cmd = {"clang++", "-Xclang", "-load", "-Xclang",
+                                  "plugin",  "-MF",     "dep",   "foo.cc"};
+  Mangler.adjust(Cmd);
+  EXPECT_THAT(Cmd, ElementsAre("/fake/bin/clang++", "foo.cc", "-fsyntax-only",
+                               "-resource-dir=/fake/resources", "-isysroot",
+                               "/fake/sysroot"));
+}
+
+TEST(CommandMangler, ResourceDir) {
+  auto Mangler = CommandMangler::forTests();
+  Mangler.ResourceDir = "/fake/resources";
+  std::vector<std::string> Cmd = {"clang++", "foo.cc"};
+  Mangler.adjust(Cmd);
+  EXPECT_THAT(Cmd, Contains("-resource-dir=/fake/resources"));
+}
+
+TEST(CommandMangler, Sysroot) {
+  auto Mangler = CommandMangler::forTests();
+  Mangler.Sysroot = "/fake/sysroot";
+
+  std::vector<std::string> Cmd = {"clang++", "foo.cc"};
+  Mangler.adjust(Cmd);
+  EXPECT_THAT(llvm::join(Cmd, " "), HasSubstr("-isysroot /fake/sysroot"));
+}
+
+TEST(CommandMangler, StripPlugins) {
+  auto Mangler = CommandMangler::forTests();
+  std::vector<std::string> Cmd = {"clang++", "-Xclang", "-load",
+                                  "-Xclang", "plugin",  "foo.cc"};
+  Mangler.adjust(Cmd);
+  for (const char* Stripped : {"-Xclang", "-load", "plugin"})
+    EXPECT_THAT(Cmd, Not(Contains(Stripped)));
+}
+
+TEST(CommandMangler, StripOutput) {
+  auto Mangler = CommandMangler::forTests();
+  std::vector<std::string> Cmd = {"clang++", "-MF", "dependency", "-c",
+                                  "foo.cc"};
+  Mangler.adjust(Cmd);
+  for (const char* Stripped : {"-MF", "dependency"})
+    EXPECT_THAT(Cmd, Not(Contains(Stripped)));
+}
+
+TEST(CommandMangler, ClangPath) {
+  auto Mangler = CommandMangler::forTests();
+  Mangler.ClangPath = "/fake/clang";
+
+  std::vector<std::string> Cmd = {"clang++", "foo.cc"};
+  Mangler.adjust(Cmd);
+  EXPECT_EQ("/fake/clang++", Cmd.front());
+
+  Cmd = {"unknown-binary", "foo.cc"};
+  Mangler.adjust(Cmd);
+  EXPECT_EQ("unknown-binary", Cmd.front());
+
+  Cmd = {"/path/clang++", "foo.cc"};
+  Mangler.adjust(Cmd);
+  EXPECT_EQ("/path/clang++", Cmd.front());
+}
+
+} // namespace
+} // namespace clangd
+} // namespace clang
+

clang-tools-extra/clangd/unittests/GlobalCompilationDatabaseTests.cpp

 using ::testing::Not;
 using ::testing::StartsWith;
 using ::testing::UnorderedElementsAre;
 
 TEST(GlobalCompilationDatabaseTest, FallbackCommand) {
   DirectoryBasedGlobalCompilationDatabase DB(None);
   auto Cmd = DB.getFallbackCommand(testPath("foo/bar.cc"));
   EXPECT_EQ(Cmd.Directory, testPath("foo"));
-  EXPECT_THAT(Cmd.CommandLine,
-              ElementsAre(EndsWith("clang"), testPath("foo/bar.cc")));
+  EXPECT_THAT(Cmd.CommandLine, ElementsAre("clang", testPath("foo/bar.cc")));
   EXPECT_EQ(Cmd.Output, "");
 
   // .h files have unknown language, so they are parsed liberally as obj-c++.
   Cmd = DB.getFallbackCommand(testPath("foo/bar.h"));
-  EXPECT_THAT(Cmd.CommandLine,
-              ElementsAre(EndsWith("clang"), "-xobjective-c++-header",
+  EXPECT_THAT(Cmd.CommandLine, ElementsAre("clang", "-xobjective-c++-header",
                                            testPath("foo/bar.h")));
   Cmd = DB.getFallbackCommand(testPath("foo/bar"));
-  EXPECT_THAT(Cmd.CommandLine,
-              ElementsAre(EndsWith("clang"), "-xobjective-c++-header",
+  EXPECT_THAT(Cmd.CommandLine, ElementsAre("clang", "-xobjective-c++-header",
                                            testPath("foo/bar")));
 }
 
 static tooling::CompileCommand cmd(llvm::StringRef File, llvm::StringRef Arg) {
   return tooling::CompileCommand(testRoot(), File, {"clang", Arg, File}, "");
 }
 
 class OverlayCDBTest : public ::testing::Test {
   class BaseCDB : public GlobalCompilationDatabase {
class OverlayCDBTest : public ::testing::Test {
   };
 
 protected:
   OverlayCDBTest() : Base(std::make_unique<BaseCDB>()) {}
   std::unique_ptr<GlobalCompilationDatabase> Base;
 };
 
 TEST_F(OverlayCDBTest, GetCompileCommand) {
-  OverlayCDB CDB(Base.get(), {}, std::string(""));
+  OverlayCDB CDB(Base.get());
   EXPECT_THAT(CDB.getCompileCommand(testPath("foo.cc"))->CommandLine,
               AllOf(Contains(testPath("foo.cc")), Contains("-DA=1")));
   EXPECT_EQ(CDB.getCompileCommand(testPath("missing.cc")), llvm::None);
 
   auto Override = cmd(testPath("foo.cc"), "-DA=3");
   CDB.setCompileCommand(testPath("foo.cc"), Override);
   EXPECT_THAT(CDB.getCompileCommand(testPath("foo.cc"))->CommandLine,
               Contains("-DA=3"));
   EXPECT_EQ(CDB.getCompileCommand(testPath("missing.cc")), llvm::None);
   CDB.setCompileCommand(testPath("missing.cc"), Override);
   EXPECT_THAT(CDB.getCompileCommand(testPath("missing.cc"))->CommandLine,
               Contains("-DA=3"));
 }
 
 TEST_F(OverlayCDBTest, GetFallbackCommand) {
   OverlayCDB CDB(Base.get(), {"-DA=4"});
   EXPECT_THAT(CDB.getFallbackCommand(testPath("bar.cc")).CommandLine,
-              ElementsAre("clang", "-DA=2", testPath("bar.cc"), "-DA=4",
-                          "-fsyntax-only", StartsWith("-resource-dir")));
+              ElementsAre("clang", "-DA=2", testPath("bar.cc"), "-DA=4"));
 }
 
 TEST_F(OverlayCDBTest, NoBase) {
-  OverlayCDB CDB(nullptr, {"-DA=6"}, std::string(""));
+  OverlayCDB CDB(nullptr, {"-DA=6"});
   EXPECT_EQ(CDB.getCompileCommand(testPath("bar.cc")), None);
   auto Override = cmd(testPath("bar.cc"), "-DA=5");
   CDB.setCompileCommand(testPath("bar.cc"), Override);
   EXPECT_THAT(CDB.getCompileCommand(testPath("bar.cc"))->CommandLine,
               Contains("-DA=5"));
 
   EXPECT_THAT(CDB.getFallbackCommand(testPath("foo.cc")).CommandLine,
-              ElementsAre(EndsWith("clang"), testPath("foo.cc"), "-DA=6",
-                          "-fsyntax-only"));
+              ElementsAre("clang", testPath("foo.cc"), "-DA=6"));
 }
 
 TEST_F(OverlayCDBTest, Watch) {
   OverlayCDB Inner(nullptr);
   OverlayCDB Outer(&Inner);
 
   std::vector<std::vector<std::string>> Changes;
   auto Sub = Outer.watch([&](const std::vector<std::string> &ChangedFiles) {
     Changes.push_back(ChangedFiles);
   });
 
   Inner.setCompileCommand("A.cpp", tooling::CompileCommand());
   Outer.setCompileCommand("B.cpp", tooling::CompileCommand());
   Inner.setCompileCommand("A.cpp", llvm::None);
   Outer.setCompileCommand("C.cpp", llvm::None);
   EXPECT_THAT(Changes, ElementsAre(ElementsAre("A.cpp"), ElementsAre("B.cpp"),
                                    ElementsAre("A.cpp"), ElementsAre("C.cpp")));
 }
 
 TEST_F(OverlayCDBTest, Adjustments) {
-  OverlayCDB CDB(Base.get(), {}, std::string(""));
+  OverlayCDB CDB(Base.get(), {"-DFallback"},
+                 [](const std::vector<std::string> &Cmd, llvm::StringRef File) {
+                   auto Ret = Cmd;
+                   Ret.push_back(
+                       ("-DAdjust_" + llvm::sys::path::filename(File)).str());
+                   return Ret;
+                 });
+  // Command from underlying gets adjusted.
   auto Cmd = CDB.getCompileCommand(testPath("foo.cc")).getValue();
-  // Delete the file name.
-  Cmd.CommandLine.pop_back();
-
-  // Check dependency file commands are dropped.
-  Cmd.CommandLine.push_back("-MF");
-  Cmd.CommandLine.push_back("random-dependency");
-
-  // Check plugin-related commands are dropped.
-  Cmd.CommandLine.push_back("-Xclang");
-  Cmd.CommandLine.push_back("-load");
-  Cmd.CommandLine.push_back("-Xclang");
-  Cmd.CommandLine.push_back("random-plugin");
-
-  Cmd.CommandLine.push_back("-DA=5");
-  Cmd.CommandLine.push_back(Cmd.Filename);
+  EXPECT_THAT(Cmd.CommandLine, ElementsAre("clang", "-DA=1", testPath("foo.cc"),
+                                           "-DAdjust_foo.cc"));
 
-  CDB.setCompileCommand(testPath("foo.cc"), Cmd);
-
-  EXPECT_THAT(CDB.getCompileCommand(testPath("foo.cc"))->CommandLine,
-              AllOf(Contains("-fsyntax-only"), Contains("-DA=5"),
-                    Contains(testPath("foo.cc")), Not(Contains("-MF")),
-                    Not(Contains("random-dependency")),
-                    Not(Contains("-Xclang")), Not(Contains("-load")),
-                    Not(Contains("random-plugin"))));
+  // Command from overlay gets adjusted.
+  tooling::CompileCommand BarCommand;
+  BarCommand.Filename = testPath("bar.cc");
+  BarCommand.CommandLine = {"clang++", "-DB=1", testPath("bar.cc")};
+  CDB.setCompileCommand(testPath("bar.cc"), BarCommand);
+  Cmd = CDB.getCompileCommand(testPath("bar.cc")).getValue();
+  EXPECT_THAT(
+      Cmd.CommandLine,
+      ElementsAre("clang++", "-DB=1", testPath("bar.cc"), "-DAdjust_bar.cc"));
+
+  // Fallback gets adjusted.
+  Cmd = CDB.getFallbackCommand("baz.cc");
+  EXPECT_THAT(Cmd.CommandLine, ElementsAre("clang", "-DA=2", "baz.cc",
+                                           "-DFallback", "-DAdjust_baz.cc"));
 }
 
 TEST(GlobalCompilationDatabaseTest, DiscoveryWithNestedCDBs) {
   const char *const CDBOuter =
       R"cdb(
       [
         {
           "file": "a.cc",