Add a utility function to add target information to a command line

[zarko]

Summary

This diff adds addTargetAndModeForProgramName, a utility function that will add appropriate -target foo and --driver-mode=g++ tokens to a command line for driver invocations of the form a/b/foo-g++. It is intended to support tooling: for example, should a compilation database record some invocation of foo-g++ without these implicit flags, a Clang tool may use this function to add them back.

Note that a compilation database won't necessarily store these invocations without their corresponding implicit flags. Some may use addTargetAndModeForProgramName when the database is first written out so that later readers don't need to repeat the operation.

[echristo]

This seems a little odd, could you explain in a bit more detail? Me not
understanding I imagine. :)

-eric

[klimek]

LG

include/clang/Tooling/Tooling.h

437

Add that for tools, CommandLine[0] will need to be a tool path relatively to which the builtin headers can be found.

[klimek]

In D13318#257091, @echristo wrote:

This seems a little odd, could you explain in a bit more detail? Me not
understanding I imagine. :)

Seems to be explained well in the comments?
If the compilation database contains:
i686-linux-android-g++ <flags> <file>
Then <file> might only be correctly parseable with the target implied; usually the driver figures that out from argv[0], but for tools, argv[0] is the tool name.

[echristo]

BTW as Manuel is happy it's fine with me. Still curious though.

[zarko]

Address comment comments.

[zarko]

Thanks; would someone with commit access land this?

echristo: for a more concrete example, imagine a source code indexer that collects up dependencies. This is a Clang tool that gets called by the build system for each translation unit and is provided with the path to the compiler it would use to build it along with the arguments to that compiler. The indexer is unlikely to be on the same path as the compiler, but the indexer's path needs to be preserved because of the way Clang tools search for resources.

The only important part of the path to the compiler is the compiler's name, because sometimes the name contains information about the target or driver mode that affects how the indexer should interpret the translation unit. This function applies the logic that Clang's driver uses to deduce that information; it then adds arguments that cause the tooling infrastructure to correctly interpret the code.

[klimek]

Did you test this with cmake? I get undef reference functions when linking ToolingTest...

[zarko]

Copy a dependency from a test Makefile to its CMakeLists.

[zarko]

I'd tested with the Makefile; it looks like CMakeLists for the unit tests isn't linking ${LLVM_TARGETS_TO_BUILD}. I've updated the diff.

[klimek]

Submitted as r249391.

include/clang/Tooling/Tooling.h

 /// the current directory and \c File.
 ///
 /// The difference to llvm::sys::fs::make_absolute is the canonicalization this
 /// does by removing "./" and computing native paths.
 ///
 /// \param File Either an absolute or relative path.
 std::string getAbsolutePath(StringRef File);
 
+/// \brief Changes CommandLine to contain implicit flags that would have been
+/// defined had the compiler driver been invoked through the path InvokedAs.
+///
+/// For example, when called with \c InvokedAs set to `i686-linux-android-g++`,
+/// the arguments '-target', 'i686-linux-android`, `--driver-mode=g++` will
+/// be inserted after the first argument in \c CommandLine.
+///
+/// This function will not add new `-target` or `--driver-mode` flags if they
+/// are already present in `CommandLine` (even if they have different settings
+/// than would have been inserted).
+///
+/// \pre `llvm::InitializeAllTargets()` has been called.
+///
+/// \param CommandLine the command line used to invoke the compiler driver or
+/// Clang tool, including the path to the executable as \c CommandLine[0].
+/// \param InvokedAs the path to the driver used to infer implicit flags.
+///
+/// \note This will not set \c CommandLine[0] to \c InvokedAs. The tooling
+/// infrastructure expects that CommandLine[0] is a tool path relative to which
+/// the builtin headers can be found.
+void addTargetAndModeForProgramName(std::vector<std::string> &CommandLine,
+                                    StringRef InvokedAs);
+
 /// \brief Creates a \c CompilerInvocation.
 clang::CompilerInvocation *newInvocation(
     clang::DiagnosticsEngine *Diagnostics,
     const llvm::opt::ArgStringList &CC1Args);
 
 } // end namespace tooling
 } // end namespace clang
 
 #endif // LLVM_CLANG_TOOLING_TOOLING_H

lib/Tooling/Tooling.cpp

 //
 //===----------------------------------------------------------------------===//
 
 #include "clang/Tooling/Tooling.h"
 #include "clang/AST/ASTConsumer.h"
 #include "clang/Driver/Compilation.h"
 #include "clang/Driver/Driver.h"
 #include "clang/Driver/Tool.h"
+#include "clang/Driver/ToolChain.h"
 #include "clang/Frontend/ASTUnit.h"
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/FrontendDiagnostic.h"
 #include "clang/Frontend/TextDiagnosticPrinter.h"
 #include "clang/Tooling/ArgumentsAdjusters.h"
 #include "clang/Tooling/CompilationDatabase.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/Config/llvm-config.h"
std::string getAbsolutePath(StringRef File) {
   SmallString<1024> AbsolutePath = RelativePath;
   std::error_code EC = llvm::sys::fs::make_absolute(AbsolutePath);
   assert(!EC);
   (void)EC;
   llvm::sys::path::native(AbsolutePath);
   return AbsolutePath.str();
 }
 
+void addTargetAndModeForProgramName(std::vector<std::string> &CommandLine,
+                                    StringRef InvokedAs) {
+  if (!CommandLine.empty() && !InvokedAs.empty()) {
+    bool AlreadyHasTarget = false;
+    bool AlreadyHasMode = false;
+    // Skip CommandLine[0].
+    for (auto Token = ++CommandLine.begin(); Token != CommandLine.end();
+         ++Token) {
+      StringRef TokenRef(*Token);
+      AlreadyHasTarget |=
+          (TokenRef == "-target" || TokenRef.startswith("-target="));
+      AlreadyHasMode |= (TokenRef == "--driver-mode" ||
+                         TokenRef.startswith("--driver-mode="));
+    }
+    auto TargetMode =
+        clang::driver::ToolChain::getTargetAndModeFromProgramName(InvokedAs);
+    if (!AlreadyHasMode && !TargetMode.second.empty()) {
+      CommandLine.insert(++CommandLine.begin(), TargetMode.second);
+    }
+    if (!AlreadyHasTarget && !TargetMode.first.empty()) {
+      CommandLine.insert(++CommandLine.begin(), {"-target", TargetMode.first});
+    }
+  }
+}
+
 namespace {
 
 class SingleFrontendActionFactory : public FrontendActionFactory {
   FrontendAction *Action;
 
 public:
   SingleFrontendActionFactory(FrontendAction *Action) : Action(Action) {}
 

unittests/Tooling/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
+  ${LLVM_TARGETS_TO_BUILD}
   Support
   )
 
 add_clang_unittest(ToolingTests
   CommentHandlerTest.cpp
   CompilationDatabaseTest.cpp
   ToolingTest.cpp
   RecursiveASTVisitorTest.cpp

unittests/Tooling/ToolingTest.cpp

 #include "clang/Frontend/ASTUnit.h"
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/FrontendAction.h"
 #include "clang/Frontend/FrontendActions.h"
 #include "clang/Tooling/CompilationDatabase.h"
 #include "clang/Tooling/Tooling.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/Config/llvm-config.h"
+#include "llvm/Support/TargetSelect.h"
+#include "llvm/Support/TargetRegistry.h"
 #include "gtest/gtest.h"
 #include <algorithm>
 #include <string>
 
 namespace clang {
 namespace tooling {
 
 namespace {
TEST(ClangToolTest, ArgumentAdjusters) {
   Tool.clearArgumentsAdjusters();
   Tool.appendArgumentsAdjuster(CheckSyntaxOnlyAdjuster);
   Tool.appendArgumentsAdjuster(getClangSyntaxOnlyAdjuster());
   Tool.run(Action.get());
   EXPECT_TRUE(Ran);
   EXPECT_FALSE(Found);
 }
 
+namespace {
+/// Find a target name such that looking for it in TargetRegistry by that name
+/// returns the same target. We expect that there is at least one target
+/// configured with this property.
+std::string getAnyTarget() {
+  llvm::InitializeAllTargets();
+  for (const auto &Target : llvm::TargetRegistry::targets()) {
+    std::string Error;
+    if (llvm::TargetRegistry::lookupTarget(Target.getName(), Error) ==
+        &Target) {
+      return Target.getName();
+    }
+  }
+  return "";
+}
+}
+
+TEST(addTargetAndModeForProgramName, AddsTargetAndMode) {
+  std::string Target = getAnyTarget();
+  ASSERT_FALSE(Target.empty());
+
+  std::vector<std::string> Args = {"clang", "-foo"};
+  addTargetAndModeForProgramName(Args, "");
+  EXPECT_EQ((std::vector<std::string>{"clang", "-foo"}), Args);
+  addTargetAndModeForProgramName(Args, Target + "-g++");
+  EXPECT_EQ((std::vector<std::string>{"clang", "-target", Target,
+                                      "--driver-mode=g++", "-foo"}),
+            Args);
+}
+
+TEST(addTargetAndModeForProgramName, PathIgnored) {
+  std::string Target = getAnyTarget();
+  ASSERT_FALSE(Target.empty());
+
+  SmallString<32> ToolPath;
+  llvm::sys::path::append(ToolPath, "foo", "bar", Target + "-g++");
+
+  std::vector<std::string> Args = {"clang", "-foo"};
+  addTargetAndModeForProgramName(Args, ToolPath);
+  EXPECT_EQ((std::vector<std::string>{"clang", "-target", Target,
+                                      "--driver-mode=g++", "-foo"}),
+            Args);
+}
+
+TEST(addTargetAndModeForProgramName, IgnoresExistingTarget) {
+  std::string Target = getAnyTarget();
+  ASSERT_FALSE(Target.empty());
+
+  std::vector<std::string> Args = {"clang", "-foo", "-target", "something"};
+  addTargetAndModeForProgramName(Args, Target + "-g++");
+  EXPECT_EQ((std::vector<std::string>{"clang", "--driver-mode=g++", "-foo",
+                                      "-target", "something"}),
+            Args);
+
+  std::vector<std::string> ArgsAlt = {"clang", "-foo", "-target=something"};
+  addTargetAndModeForProgramName(ArgsAlt, Target + "-g++");
+  EXPECT_EQ((std::vector<std::string>{"clang", "--driver-mode=g++", "-foo",
+                                      "-target=something"}),
+            ArgsAlt);
+}
+
+TEST(addTargetAndModeForProgramName, IgnoresExistingMode) {
+  std::string Target = getAnyTarget();
+  ASSERT_FALSE(Target.empty());
+
+  std::vector<std::string> Args = {"clang", "-foo", "--driver-mode=abc"};
+  addTargetAndModeForProgramName(Args, Target + "-g++");
+  EXPECT_EQ((std::vector<std::string>{"clang", "-target", Target, "-foo",
+                                      "--driver-mode=abc"}),
+            Args);
+
+  std::vector<std::string> ArgsAlt = {"clang", "-foo", "--driver-mode", "abc"};
+  addTargetAndModeForProgramName(ArgsAlt, Target + "-g++");
+  EXPECT_EQ((std::vector<std::string>{"clang", "-target", Target, "-foo",
+                                      "--driver-mode", "abc"}),
+            ArgsAlt);
+}
+
 #ifndef LLVM_ON_WIN32
 TEST(ClangToolTest, BuildASTs) {
   FixedCompilationDatabase Compilations("/", std::vector<std::string>());
 
   std::vector<std::string> Sources;
   Sources.push_back("/a.cc");
   Sources.push_back("/b.cc");
   ClangTool Tool(Compilations, Sources);