Introduce clang-query tool.

[pcc]

Summary

This tool is for interactive exploration of the Clang AST using AST matchers.
It currently allows the user to enter a matcher at an interactive prompt
and view the resulting bindings as diagnostics, AST pretty prints or AST
dumps. Example session:

$ cat foo.c
void foo(void) {}
$ clang-query foo.c --
clang-query> match functionDecl()

Match #1:

foo.c:1:1: note: "root" binds here
void foo(void) {}
^~~~~~~~~~~~~~~~~
1 match.

[klimek]

clang-query/QueryEngine.cpp

34 ↗

(On Diff #5337)

This seems like a strong indication that we want virtual dispatch here :) (I thought that when reading the header, but this kinda drives it home)

clang-query/QueryEngine.h

28 ↗

(On Diff #5337)

I'm a very strong proponent of "public-first". The reason is that a reader of the class usually starts at the top, and a user of a class should not care about the privates, and thus be able to skip them - this is considerably easier if they're at the bottom of the class definition.

37 ↗

(On Diff #5337)

Some doxygen comments would be highly appreciated ;)

clang-query/tool/ClangQuery.cpp

51–58

Can we add a -c "..." or -f <command-file> batch mode? This would also allow us to add a FileCheck integration test...

[sbenza]

clang-query/QueryParser.cpp

23

isWhitespace() from clang/Basic/CharInfo.h

clang-query/QueryParser.h

19

StringRef instead of const char*

[pcc]

• Address reviewer comments

clang-query/QueryEngine.cpp

34 ↗

(On Diff #5337)

I wanted to keep Query a pure data structure in order to simplify the parser tests. But of course there are other considerations. Done.

clang-query/QueryEngine.h

28 ↗

(On Diff #5337)

I believe this comment is moot following the design change.

37 ↗

(On Diff #5337)

Added.

clang-query/QueryParser.cpp

23

Done.

clang-query/QueryParser.h

19

I'd prefer not to. The implementation of the parser is simpler when the string is known to be null terminated.

clang-query/tool/ClangQuery.cpp

51–58

I think both would be useful; done.

[klimek]

lg with the nits fixed

clang-query/Query.cpp

56–66

Yea, we really need addDynamicMatcher ...

clang-query/Query.h

47

I kinda dislike the introduction of RTTI just for testing ... Since this is all in clang-query I think it's ok, as I cannot come up with a significantly better way right now.

clang-query/QueryParser.h

23

Here also StringRef...

[sbenza]

clang-query/Query.cpp

56–66

If I remember correctly, I added one, then we decide to remove it again.
This tool would be the first use case that would require it.

59

You can bind at the DynTypedMatcher level using DynTypedMatcher::tryBind(). No need to do it for every type.

60

What is the purpose of forEachDescendant() here?
Finder will recursively try to match the provided matchers already.

Doing it this way will match the topmost node if something in it matches, instead of returning the actual node that matched.

clang-query/QueryParser.h

21

There should be a comment somewhere specifying the valid commands and their syntax.
It could in the parser, or somewhere in ClangQuery.cpp (possibly printing it to the user with a --help option).

[silvas]

clang-query/Query.h

61–62

These return true classof's should be here. See http://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html. In particular, see the second rule of thumb http://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html#rules-of-thumb.

[pcc]

• Address reviewer comments

[pcc]

clang-query/Query.cpp

56–66

I've resurrected it in D2114.

59

Done.

60

MatchFinder::match will not match recursively, so we need to do it ourselves with forEachDescendant.

However, upon further reflection, I've decided that it would be better to have a MatchFinder::matchAST that does the same thing for ASTs as the ASTConsumer returned by MatchFinder::getConsumer. See D2115.

clang-query/Query.h

61–62

I assume you meant "should not be here". Done.

clang-query/QueryParser.h

21

Done.

23

OK, done.

[pcc]

Closed by commit rL194227 (authored by @pcc).

[kwk]

Can you please explain a usage of the "set bind-root" command? Why would I not want to match the loaded AST against a given matcher?

clang-tools-extra/trunk/clang-query/Query.cpp

38 ↗

(On Diff #5404)

What does this mean? And why would I not want to match the loaded AST against a given matcher? Can you please give a useful small example of how to use the "set bind-root" command?

[klimek]

clang-tools-extra/trunk/clang-query/Query.cpp

38 ↗

(On Diff #5404)

I think you're misreading this; the doc for bind-root is below ("Set whether to bind the root matcher to "root".)

CMakeLists.txt

+check_library_exists(edit el_init "" HAVE_LIBEDIT)
+
 add_subdirectory(clang-apply-replacements)
 add_subdirectory(clang-modernize)
+add_subdirectory(clang-query)
 add_subdirectory(clang-tidy)
 add_subdirectory(modularize)
 add_subdirectory(pp-trace)
 add_subdirectory(remove-cstr-calls)
 add_subdirectory(tool-template)
 
 # Add the common testsuite after all the tools.
 add_subdirectory(test)
 add_subdirectory(unittests)

Makefile

 ##===- tools/extra/Makefile --------------------------------*- Makefile -*-===##
 #
 #                     The LLVM Compiler Infrastructure
 #
 # This file is distributed under the University of Illinois Open Source
 # License. See LICENSE.TXT for details.
 #
 ##===----------------------------------------------------------------------===##
 
 CLANG_LEVEL := ../..
 
 include $(CLANG_LEVEL)/../../Makefile.config
 
 PARALLEL_DIRS := remove-cstr-calls tool-template modularize pp-trace
-DIRS := clang-apply-replacements clang-modernize clang-tidy unittests
+DIRS := clang-apply-replacements clang-modernize clang-tidy clang-query \
+  unittests
 
 include $(CLANG_LEVEL)/Makefile
 
 ###
 # Handle the nested test suite.
 
 ifneq ($(PROJ_SRC_ROOT),$(PROJ_OBJ_ROOT))
 $(RecursiveTargets)::

clang-query/CMakeLists.txt

+add_clang_library(clangQuery
+  Query.cpp
+  QueryParser.cpp
+  )
+target_link_libraries(clangQuery
+  clangAST
+  clangASTMatchers
+  clangBasic
+  clangDynamicASTMatchers
+  clangFrontend
+  )
+
+add_subdirectory(tool)

clang-query/Makefile

-##===- tools/extra/test/Unit/Makefile ----------------------*- Makefile -*-===##
+##===- tools/extra/clang-query/Makefile --------------------*- Makefile -*-===##
 #
 #                     The LLVM Compiler Infrastructure
 #
 # This file is distributed under the University of Illinois Open Source
 # License. See LICENSE.TXT for details.
 #
 ##===----------------------------------------------------------------------===##
 
 CLANG_LEVEL := ../../..
+LIBRARYNAME := clangQuery
 include $(CLANG_LEVEL)/../../Makefile.config
 
-PARALLEL_DIRS := clang-apply-replacements clang-modernize clang-tidy
-
 include $(CLANG_LEVEL)/Makefile

clang-query/Query.h

+//===--- Query.h - clang-query ----------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_TOOLS_EXTRA_CLANG_QUERY_QUERY_H
+#define LLVM_CLANG_TOOLS_EXTRA_CLANG_QUERY_QUERY_H
+
+#include <string>
+#include "clang/ASTMatchers/Dynamic/VariantValue.h"
+#include "llvm/ADT/IntrusiveRefCntPtr.h"
+#include "llvm/ADT/Optional.h"
+
+namespace clang {
+namespace query {
+
+enum OutputKind {
+  OK_Diag,
+  OK_Print,
+  OK_Dump
+};
+
+enum QueryKind {
+  QK_Invalid,
+  QK_NoOp,
+  QK_Help,
+  QK_Match,
+  QK_SetBool,
+  QK_SetOutputKind
+};
+
+class QuerySession;
+
+struct Query : llvm::RefCountedBase<Query> {
+  Query(QueryKind Kind) : Kind(Kind) {}
+  virtual ~Query();
+
+  /// Perform the query on \p QS and print output to \p OS.
+  ///
+  /// \return false if an error occurs, otherwise return true.
+  virtual bool run(llvm::raw_ostream &OS, QuerySession &QS) const = 0;
+
+  const QueryKind Kind;
+};
+
+typedef llvm::IntrusiveRefCntPtr<Query> QueryRef;
+
+/// Any query which resulted in a parse error.  The error message is in ErrStr.
+struct InvalidQuery : Query {
+  InvalidQuery(const Twine &ErrStr) : Query(QK_Invalid), ErrStr(ErrStr.str()) {}
+  bool run(llvm::raw_ostream &OS, QuerySession &QS) const LLVM_OVERRIDE;
+
+  std::string ErrStr;
+
+  static bool classof(const Query *Q) { return Q->Kind == QK_Invalid; }
+};
+
+/// No-op query (i.e. a blank line).
+struct NoOpQuery : Query {
+  NoOpQuery() : Query(QK_NoOp) {}
+  bool run(llvm::raw_ostream &OS, QuerySession &QS) const LLVM_OVERRIDE;
+
+  static bool classof(const Query *Q) { return Q->Kind == QK_NoOp; }
+};
+
+/// Query for "help".
+struct HelpQuery : Query {
+  HelpQuery() : Query(QK_Help) {}
+  bool run(llvm::raw_ostream &OS, QuerySession &QS) const LLVM_OVERRIDE;
+
+  static bool classof(const Query *Q) { return Q->Kind == QK_Help; }
+};
+
+/// Query for "match MATCHER".
+struct MatchQuery : Query {
+  MatchQuery(const ast_matchers::dynamic::DynTypedMatcher &Matcher)
+      : Query(QK_Match), Matcher(Matcher) {}
+  bool run(llvm::raw_ostream &OS, QuerySession &QS) const LLVM_OVERRIDE;
+
+  ast_matchers::dynamic::DynTypedMatcher Matcher;
+
+  static bool classof(const Query *Q) { return Q->Kind == QK_Match; }
+};
+
+template <typename T> struct SetQueryKind {};
+
+template <> struct SetQueryKind<bool> {
+  static const QueryKind value = QK_SetBool;
+};
+
+template <> struct SetQueryKind<OutputKind> {
+  static const QueryKind value = QK_SetOutputKind;
+};
+
+/// Query for "set VAR VALUE".
+template <typename T> struct SetQuery : Query {
+  SetQuery(T QuerySession::*Var, T Value)
+      : Query(SetQueryKind<T>::value), Var(Var), Value(Value) {}
+  bool run(llvm::raw_ostream &OS, QuerySession &QS) const LLVM_OVERRIDE {
+    QS.*Var = Value;
+    return true;
+  }
+
+  static bool classof(const Query *Q) {
+    return Q->Kind == SetQueryKind<T>::value;
+  }
+
+  T QuerySession::*Var;
+  T Value;
+};
+
+} // namespace query
+} // namespace clang
+
+#endif

clang-query/Query.cpp

+//===---- Query.cpp - clang-query query -----------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "Query.h"
+#include "QuerySession.h"
+#include "clang/ASTMatchers/ASTMatchFinder.h"
+#include "clang/Frontend/ASTUnit.h"
+#include "clang/Frontend/TextDiagnostic.h"
+#include "llvm/Support/raw_ostream.h"
+
+using namespace clang::ast_matchers;
+using namespace clang::ast_matchers::dynamic;
+
+namespace clang {
+namespace query {
+
+Query::~Query() {}
+
+bool InvalidQuery::run(llvm::raw_ostream &OS, QuerySession &QS) const {
+  OS << ErrStr << "\n";
+  return false;
+}
+
+bool NoOpQuery::run(llvm::raw_ostream &OS, QuerySession &QS) const {
+  return true;
+}
+
+bool HelpQuery::run(llvm::raw_ostream &OS, QuerySession &QS) const {
+  OS << "Available commands:\n\n"
+        "  match MATCHER, m MATCHER      "
+        "Match the loaded ASTs against the given matcher.\n"
+        "  set bind-root (true|false)    "
+        "Set whether to bind the root matcher to \"root\".\n"
+        "  set output (diag|print|dump)  "
+        "Set whether to print bindings as diagnostics,\n"
+        "                                "
+        "AST pretty prints or AST dumps.\n\n";
+  return true;
+}
+
+namespace {
+
+struct CollectBoundNodes : MatchFinder::MatchCallback {
+  std::vector<BoundNodes> &Bindings;
+  CollectBoundNodes(std::vector<BoundNodes> &Bindings) : Bindings(Bindings) {}
+  void run(const MatchFinder::MatchResult &Result) {
+    Bindings.push_back(Result.Nodes);
+  }
+};
+
+}
+
+bool MatchQuery::run(llvm::raw_ostream &OS, QuerySession &QS) const {
+  unsigned MatchCount = 0;
+
+  for (llvm::ArrayRef<ASTUnit *>::iterator I = QS.ASTs.begin(),
+                                           E = QS.ASTs.end();
+       I != E; ++I) {
+    ASTUnit *AST = *I;
+    MatchFinder Finder;
+    std::vector<BoundNodes> Matches;
+    DynTypedMatcher MaybeBoundMatcher = Matcher;
+    if (QS.BindRoot) {
+      llvm::Optional<DynTypedMatcher> M = Matcher.tryBind("root");
+      if (M)
+        MaybeBoundMatcher = *M;
+    }
+    CollectBoundNodes Collect(Matches);
+    if (!Finder.addDynamicMatcher(MaybeBoundMatcher, &Collect)) {
+      OS << "Not a valid top-level matcher.\n";
+      return false;
+    }
+    Finder.matchAST(AST->getASTContext());
+
+    for (std::vector<BoundNodes>::iterator MI = Matches.begin(),
+                                           ME = Matches.end();
+         MI != ME; ++MI) {
+      OS << "\nMatch #" << ++MatchCount << ":\n\n";
+
+      for (BoundNodes::IDToNodeMap::const_iterator BI = MI->getMap().begin(),
+                                                   BE = MI->getMap().end();
+           BI != BE; ++BI) {
+        switch (QS.OutKind) {
+        case OK_Diag: {
+          clang::SourceRange R = BI->second.getSourceRange();
+          if (R.isValid()) {
+            TextDiagnostic TD(OS, AST->getASTContext().getLangOpts(),
+                              &AST->getDiagnostics().getDiagnosticOptions());
+            TD.emitDiagnostic(
+                R.getBegin(), DiagnosticsEngine::Note,
+                "\"" + BI->first + "\" binds here",
+                ArrayRef<CharSourceRange>(CharSourceRange::getTokenRange(R)),
+                ArrayRef<FixItHint>(), &AST->getSourceManager());
+          }
+          break;
+        }
+        case OK_Print: {
+          OS << "Binding for \"" << BI->first << "\":\n";
+          BI->second.print(OS, AST->getASTContext().getPrintingPolicy());
+          OS << "\n";
+          break;
+        }
+        case OK_Dump: {
+          OS << "Binding for \"" << BI->first << "\":\n";
+          BI->second.dump(OS, AST->getSourceManager());
+          OS << "\n";
+          break;
+        }
+        }
+      }
+
+      if (MI->getMap().empty())
+        OS << "No bindings.\n";
+    }
+  }
+
+  OS << MatchCount << (MatchCount == 1 ? " match.\n" : " matches.\n");
+  return true;
+}
+
+const QueryKind SetQueryKind<bool>::value;
+const QueryKind SetQueryKind<OutputKind>::value;
+
+} // namespace query
+} // namespace clang

clang-query/QueryParser.h

+//===--- QueryParser.h - clang-query ----------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_TOOLS_EXTRA_CLANG_QUERY_QUERY_PARSER_H
+#define LLVM_CLANG_TOOLS_EXTRA_CLANG_QUERY_QUERY_PARSER_H
+
+#include "Query.h"
+
+namespace clang {
+namespace query {
+
+/// \brief Parse \p Line.
+///
+/// \return A reference to the parsed query object, which may be an
+/// \c InvalidQuery if a parse error occurs.
+QueryRef ParseQuery(StringRef Line);
+
+} // namespace query
+} // namespace clang
+
+#endif

clang-query/QueryParser.cpp

+//===---- QueryParser.cpp - clang-query command parser --------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "QueryParser.h"
+#include "Query.h"
+#include "QuerySession.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/ADT/StringSwitch.h"
+#include "clang/ASTMatchers/Dynamic/Parser.h"
+#include "clang/Basic/CharInfo.h"
+
+using namespace llvm;
+using namespace clang::ast_matchers::dynamic;
+
+namespace clang {
+namespace query {
+
+// Lex any amount of whitespace followed by a "word" (any sequence of
+// non-whitespace characters) from the start of region [Begin,End).  If no word
+// is found before End, return StringRef().  Begin is adjusted to exclude the
+// lexed region.
+static StringRef LexWord(const char *&Begin, const char *End) {
+  while (true) {
+    if (Begin == End)
+      return StringRef();
+
+    if (!isWhitespace(*Begin))
+      break;
+
+    ++Begin;
+  }
+
+  const char *WordBegin = Begin;
+
+  while (true) {
+    ++Begin;
+
+    if (Begin == End || isWhitespace(*Begin))
+      return StringRef(WordBegin, Begin - WordBegin);
+  }
+}
+
+static QueryRef ParseSetBool(bool QuerySession::*Var, StringRef ValStr) {
+  unsigned Value = StringSwitch<unsigned>(ValStr)
+                      .Case("false", 0)
+                      .Case("true", 1)
+                      .Default(~0u);
+  if (Value == ~0u) {
+    return new InvalidQuery("expected 'true' or 'false', got '" + ValStr + "'");
+  }
+  return new SetQuery<bool>(Var, Value);
+}
+
+static QueryRef ParseSetOutputKind(StringRef ValStr) {
+  unsigned OutKind = StringSwitch<unsigned>(ValStr)
+                         .Case("diag", OK_Diag)
+                         .Case("print", OK_Print)
+                         .Case("dump", OK_Dump)
+                         .Default(~0u);
+  if (OutKind == ~0u) {
+    return new InvalidQuery("expected 'diag', 'print' or 'dump', got '" +
+                            ValStr + "'");
+  }
+  return new SetQuery<OutputKind>(&QuerySession::OutKind, OutputKind(OutKind));
+}
+
+static QueryRef EndQuery(const char *Begin, const char *End, QueryRef Q) {
+  const char *Extra = Begin;
+  if (!LexWord(Begin, End).empty())
+    return new InvalidQuery("unexpected extra input: '" +
+                            StringRef(Extra, End - Extra) + "'");
+  return Q;
+}
+
+enum ParsedQueryKind {
+  PQK_Invalid,
+  PQK_NoOp,
+  PQK_Help,
+  PQK_Match,
+  PQK_Set
+};
+
+enum ParsedQueryVariable {
+  PQV_Invalid,
+  PQV_Output,
+  PQV_BindRoot
+};
+
+QueryRef ParseQuery(StringRef Line) {
+  const char *Begin = Line.data();
+  const char *End = Line.data() + Line.size();
+
+  StringRef CommandStr = LexWord(Begin, End);
+  ParsedQueryKind QKind = StringSwitch<ParsedQueryKind>(CommandStr)
+                              .Case("", PQK_NoOp)
+                              .Case("help", PQK_Help)
+                              .Case("m", PQK_Match)
+                              .Case("match", PQK_Match)
+                              .Case("set", PQK_Set)
+                              .Default(PQK_Invalid);
+
+  switch (QKind) {
+  case PQK_NoOp:
+    return new NoOpQuery;
+
+  case PQK_Help:
+    return EndQuery(Begin, End, new HelpQuery);
+
+  case PQK_Match: {
+    Diagnostics Diag;
+    Optional<DynTypedMatcher> Matcher =
+        Parser::parseMatcherExpression(StringRef(Begin, End - Begin), &Diag);
+    if (!Matcher) {
+      std::string ErrStr;
+      llvm::raw_string_ostream OS(ErrStr);
+      Diag.printToStreamFull(OS);
+      return new InvalidQuery(OS.str());
+    }
+    return new MatchQuery(*Matcher);
+  }
+
+  case PQK_Set: {
+    StringRef VarStr = LexWord(Begin, End);
+    if (VarStr.empty())
+      return new InvalidQuery("expected variable name");
+
+    ParsedQueryVariable Var = StringSwitch<ParsedQueryVariable>(VarStr)
+                .Case("output", PQV_Output)
+                .Case("bind-root", PQV_BindRoot)
+                .Default(PQV_Invalid);
+    if (Var == PQV_Invalid)
+      return new InvalidQuery("unknown variable: '" + VarStr + "'");
+
+    StringRef ValStr = LexWord(Begin, End);
+    if (ValStr.empty())
+      return new InvalidQuery("expected variable value");
+
+    QueryRef Q;
+    switch (Var) {
+    case PQV_Output:
+      Q = ParseSetOutputKind(ValStr);
+      break;
+    case PQV_BindRoot:
+      Q = ParseSetBool(&QuerySession::BindRoot, ValStr);
+      break;
+    case PQV_Invalid:
+      llvm_unreachable("Invalid query kind");
+    }
+
+    return EndQuery(Begin, End, Q);
+  }
+
+  case PQK_Invalid:
+    return new InvalidQuery("unknown command: " + CommandStr);
+  }
+}
+
+} // namespace query
+} // namespace clang

clang-query/QuerySession.h

+//===--- QuerySession.h - clang-query ---------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_TOOLS_EXTRA_CLANG_QUERY_QUERY_SESSION_H
+#define LLVM_CLANG_TOOLS_EXTRA_CLANG_QUERY_QUERY_SESSION_H
+
+#include "llvm/ADT/ArrayRef.h"
+#include "Query.h"
+
+namespace clang {
+
+class ASTUnit;
+
+namespace query {
+
+/// Represents the state for a particular clang-query session.
+class QuerySession {
+public:
+  QuerySession(llvm::ArrayRef<ASTUnit *> ASTs)
+      : ASTs(ASTs), OutKind(OK_Diag), BindRoot(true) {}
+
+  llvm::ArrayRef<ASTUnit *> ASTs;
+  OutputKind OutKind;
+  bool BindRoot;
+};
+
+} // namespace query
+} // namespace clang
+
+#endif

clang-query/tool/CMakeLists.txt

+if(HAVE_LIBEDIT)
+  include_directories(${CMAKE_CURRENT_SOURCE_DIR}/..)
+
+  add_clang_executable(clang-query ClangQuery.cpp)
+  target_link_libraries(clang-query
+    ${LIBEDIT}
+    clangFrontend
+    clangQuery
+    clangTooling
+    )
+endif()

clang-query/tool/ClangQuery.cpp

+//===---- ClangQuery.cpp - clang-query tool -------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This tool is for interactive exploration of the Clang AST using AST matchers.
+// It currently allows the user to enter a matcher at an interactive prompt and
+// view the resulting bindings as diagnostics, AST pretty prints or AST dumps.
+// Example session:
+//
+// $ cat foo.c
+// void foo(void) {}
+// $ clang-query foo.c --
+// clang-query> match functionDecl()
+//
+// Match #1:
+//
+// foo.c:1:1: note: "root" binds here
+// void foo(void) {}
+// ^~~~~~~~~~~~~~~~~
+// 1 match.
+//
+//===----------------------------------------------------------------------===//
+
+#include "Query.h"
+#include "QuerySession.h"
+#include "QueryParser.h"
+
+#include "clang/Frontend/ASTUnit.h"
+#include "clang/Tooling/CompilationDatabase.h"
+#include "clang/Tooling/Tooling.h"
+#include "llvm/ADT/OwningPtr.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/Signals.h"
+#include <fstream>
+#include <string>
+
+#include <histedit.h>
+
+using namespace clang;
+using namespace clang::ast_matchers;
+using namespace clang::ast_matchers::dynamic;
+using namespace clang::query;
+using namespace clang::tooling;
+using namespace llvm;
+
+static cl::opt<std::string> BuildPath("b", cl::desc("Specify build path"),
+                                      cl::value_desc("<path>"));
+
+static cl::list<std::string> Commands("c", cl::desc("Specify command to run"),
+                                      cl::value_desc("<command>"));
+
+static cl::list<std::string> CommandFiles("f",
+                                          cl::desc("Read commands from file"),
+                                          cl::value_desc("<file>"));
+
+static cl::list<std::string> SourcePaths(cl::Positional,
+                                         cl::desc("<source0> [... <sourceN>]"),
+                                         cl::OneOrMore);
+
+static char *ReturnPrompt(EditLine *EL) {
+  static char Prompt[] = "clang-query> ";
+  return Prompt;
+}
+
+int main(int argc, const char **argv) {
+  llvm::sys::PrintStackTraceOnErrorSignal();
+  cl::ParseCommandLineOptions(argc, argv);
+
+  if (!Commands.empty() && !CommandFiles.empty()) {
+    llvm::errs() << argv[0] << ": cannot specify both -c and -f\n";
+    return 1;
+  }
+
+  llvm::OwningPtr<CompilationDatabase> Compilations(
+        FixedCompilationDatabase::loadFromCommandLine(argc, argv));
+  if (!Compilations) {  // Couldn't find a compilation DB from the command line
+    std::string ErrorMessage;
+    Compilations.reset(
+      !BuildPath.empty() ?
+        CompilationDatabase::autoDetectFromDirectory(BuildPath, ErrorMessage) :
+        CompilationDatabase::autoDetectFromSource(SourcePaths[0], ErrorMessage)
+      );
+
+    // Still no compilation DB? - bail.
+    if (!Compilations)
+      llvm::report_fatal_error(ErrorMessage);
+  }
+
+  ClangTool Tool(*Compilations, SourcePaths);
+  std::vector<ASTUnit *> ASTs;
+  if (Tool.buildASTs(ASTs) != 0)
+    return 1;
+
+  QuerySession QS(ASTs);
+
+  if (!Commands.empty()) {
+    for (cl::list<std::string>::iterator I = Commands.begin(),
+                                         E = Commands.end();
+         I != E; ++I) {
+      QueryRef Q = ParseQuery(I->c_str());
+      if (!Q->run(llvm::outs(), QS))
+        return 1;
+    }
+  } else if (!CommandFiles.empty()) {
+    for (cl::list<std::string>::iterator I = CommandFiles.begin(),
+                                         E = CommandFiles.end();
+         I != E; ++I) {
+      std::ifstream Input(I->c_str());
+      if (!Input.is_open()) {
+        llvm::errs() << argv[0] << ": cannot open " << *I << "\n";
+        return 1;
+      }
+      while (Input.good()) {
+        std::string Line;
+        std::getline(Input, Line);
+
+        QueryRef Q = ParseQuery(Line.c_str());
+        if (!Q->run(llvm::outs(), QS))
+          return 1;
+      }
+    }
+  } else {
+    History *Hist = history_init();
+    HistEvent Event;
+    history(Hist, &Event, H_SETSIZE, 100);
+
+    EditLine *EL = el_init("clang-query", stdin, stdout, stderr);
+    el_set(EL, EL_PROMPT, ReturnPrompt);
+    el_set(EL, EL_EDITOR, "emacs");
+    el_set(EL, EL_HIST, history, Hist);
+
+    int Count;
+    while (const char *Line = el_gets(EL, &Count)) {
+      if (Count == 0)
+        break;
+
+      history(Hist, &Event, H_ENTER, Line);
+
+      QueryRef Q = ParseQuery(Line);
+      Q->run(llvm::outs(), QS);
+    }
+
+    history_end(Hist);
+    el_end(EL);
+
+    llvm::outs() << "\n";
+  }
+
+  llvm::DeleteContainerPointers(ASTs);
+
+  return 0;
+}

test/CMakeLists.txt

set(CLANG_TOOLS_TEST_DEPS
   modularize
   pp-trace
   remove-cstr-calls
 
   # Unit tests
   ExtraToolsUnitTests
   )
 
+if(HAVE_LIBEDIT)
+  list(APPEND CLANG_TOOLS_TEST_DEPS clang-query)
+endif()
+
 add_lit_testsuite(check-clang-tools "Running the Clang extra tools' regression tests"
   ${CMAKE_CURRENT_BINARY_DIR}
   DEPENDS ${CLANG_TOOLS_TEST_DEPS}
   ARGS ${CLANG_TOOLS_TEST_EXTRA_ARGS}
   )
 set_target_properties(check-clang-tools PROPERTIES FOLDER "Clang extra tools' tests")
 

test/clang-query/Inputs/foo.script

+foo
+bar

test/clang-query/errors.c

+// RUN: not clang-query -c foo -c bar %s -- | FileCheck %s
+// RUN: not clang-query -f %S/Inputs/foo.script %s -- | FileCheck %s
+// RUN: not clang-query -f %S/Inputs/nonexistent.script %s -- 2>&1 | FileCheck --check-prefix=CHECK-NONEXISTENT %s
+// RUN: not clang-query -c foo -f foo %s -- 2>&1 | FileCheck --check-prefix=CHECK-BOTH %s
+// REQUIRES: libedit
+
+// CHECK: unknown command: foo
+// CHECK-NOT: unknown command: bar
+
+// CHECK-NONEXISTENT: cannot open {{.*}}nonexistent.script
+// CHECK-BOTH: cannot specify both -c and -f

test/clang-query/function-decl.c

+// RUN: clang-query -c "match functionDecl()" %s -- | FileCheck %s
+// REQUIRES: libedit
+
+// CHECK: function-decl.c:5:1: note: "root" binds here
+void foo(void) {}

test/lit.cfg

 
 # Shell execution
 if platform.system() not in ['Windows'] or lit_config.getBashPath() != '':
     config.available_features.add('shell')
 
 # ANSI escape sequences in non-dumb terminal
 if platform.system() not in ['Windows']:
     config.available_features.add('ansi-escape-sequences')
+
+if config.have_libedit == "1":
+  config.available_features.add('libedit')

test/lit.site.cfg.in

 ## Autogenerated by LLVM/Clang configuration.
 # Do not edit!
 config.llvm_src_root = "@LLVM_SOURCE_DIR@"
 config.llvm_obj_root = "@LLVM_BINARY_DIR@"
 config.llvm_tools_dir = "@LLVM_TOOLS_DIR@"
 config.llvm_libs_dir = "@LLVM_LIBS_DIR@"
 config.lit_tools_dir = "@LLVM_LIT_TOOLS_DIR@"
 config.clang_tools_binary_dir = "@CLANG_TOOLS_BINARY_DIR@"
 config.target_triple = "@TARGET_TRIPLE@"
+config.have_libedit = "@HAVE_LIBEDIT@"
 
 # Support substitution of the tools and libs dirs with user parameters. This is
 # used when we can't determine the tool dir at configuration time.
 try:
     config.llvm_tools_dir = config.llvm_tools_dir % lit_config.params
     config.llvm_libs_dir = config.llvm_libs_dir % lit_config.params
 except KeyError,e:
     key, = e.args
     lit_config.fatal("unable to find %r parameter, use '--param=%s=VALUE'" % (key,key))
 
 # Let the main config do the real work.
 lit_config.load_config(config, "@CLANG_TOOLS_SOURCE_DIR@/test/lit.cfg")

unittests/CMakeLists.txt

 add_custom_target(ExtraToolsUnitTests)
 set_target_properties(ExtraToolsUnitTests PROPERTIES FOLDER "Extra Tools Unit Tests")
 
 function(add_extra_unittest test_dirname)
   add_unittest(ExtraToolsUnitTests ${test_dirname} ${ARGN})
 endfunction()
 
 add_subdirectory(clang-apply-replacements)
 add_subdirectory(clang-modernize)
+add_subdirectory(clang-query)
 add_subdirectory(clang-tidy)

unittests/Makefile

 ##===- tools/extra/test/Unit/Makefile ----------------------*- Makefile -*-===##
 #
 #                     The LLVM Compiler Infrastructure
 #
 # This file is distributed under the University of Illinois Open Source
 # License. See LICENSE.TXT for details.
 #
 ##===----------------------------------------------------------------------===##
 
 CLANG_LEVEL := ../../..
 include $(CLANG_LEVEL)/../../Makefile.config
 
-PARALLEL_DIRS := clang-apply-replacements clang-modernize clang-tidy
+PARALLEL_DIRS := clang-apply-replacements clang-modernize clang-query clang-tidy
 
 include $(CLANG_LEVEL)/Makefile

unittests/clang-query/CMakeLists.txt

+set(LLVM_LINK_COMPONENTS
+  support
+  )
+
+include_directories(
+  ${CMAKE_CURRENT_SOURCE_DIR}/../../clang-query 
+  )
+
+add_extra_unittest(ClangQueryTests
+  QueryEngineTest.cpp
+  QueryParserTest.cpp
+  )
+
+target_link_libraries(ClangQueryTests
+  clangASTMatchers
+  clangQuery
+  clangTooling
+  )

unittests/clang-query/Makefile

+##===- unittests/clang-query/Makefile ----------------------*- Makefile -*-===##
+#
+#                     The LLVM Compiler Infrastructure
+#
+# This file is distributed under the University of Illinois Open Source
+# License. See LICENSE.TXT for details.
+#
+##===----------------------------------------------------------------------===##
+
+CLANG_LEVEL = ../../../..
+include $(CLANG_LEVEL)/../../Makefile.config
+
+TESTNAME = ClangQuery
+LINK_COMPONENTS := asmparser bitreader support MC MCParser option \
+     TransformUtils
+USEDLIBS = clangQuery.a clangTooling.a clangFrontend.a clangSerialization.a \
+     clangDriver.a clangParse.a clangSema.a clangEdit.a clangAnalysis.a \
+     clangAST.a clangASTMatchers.a clangDynamicASTMatchers.a clangLex.a \
+     clangBasic.a
+
+include $(CLANG_LEVEL)/Makefile
+MAKEFILE_UNITTEST_NO_INCLUDE_COMMON := 1
+CPP.Flags += -I$(PROJ_SRC_DIR)/../../clang-query
+include $(LLVM_SRC_ROOT)/unittests/Makefile.unittest

unittests/clang-query/QueryEngineTest.cpp

+//===---- QueryTest.cpp - clang-query test --------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "Query.h"
+#include "QuerySession.h"
+#include "clang/ASTMatchers/ASTMatchers.h"
+#include "clang/ASTMatchers/Dynamic/VariantValue.h"
+#include "clang/Frontend/ASTUnit.h"
+#include "clang/Tooling/Tooling.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/Support/raw_ostream.h"
+#include "gtest/gtest.h"
+#include <string>
+
+using namespace clang;
+using namespace clang::ast_matchers;
+using namespace clang::ast_matchers::dynamic;
+using namespace clang::query;
+using namespace clang::tooling;
+
+TEST(Query, Basic) {
+  OwningPtr<ASTUnit> FooAST(
+      buildASTFromCode("void foo1(void) {}\nvoid foo2(void) {}", "foo.cc"));
+  ASSERT_TRUE(FooAST.get());
+  OwningPtr<ASTUnit> BarAST(
+      buildASTFromCode("void bar1(void) {}\nvoid bar2(void) {}", "bar.cc"));
+  ASSERT_TRUE(BarAST.get());
+
+  ASTUnit *ASTs[] = { FooAST.get(), BarAST.get() };
+
+  std::string Str;
+  llvm::raw_string_ostream OS(Str);
+  QuerySession S(ASTs);
+
+  DynTypedMatcher FnMatcher = functionDecl();
+  DynTypedMatcher FooMatcher = functionDecl(hasName("foo1"));
+
+  EXPECT_TRUE(NoOpQuery().run(OS, S));
+
+  EXPECT_EQ("", OS.str());
+
+  Str.clear();
+
+  EXPECT_FALSE(InvalidQuery("Parse error").run(OS, S));
+
+  EXPECT_EQ("Parse error\n", OS.str());
+
+  Str.clear();
+
+  EXPECT_TRUE(HelpQuery().run(OS, S));
+
+  EXPECT_TRUE(OS.str().find("Available commands:") != std::string::npos);
+
+  Str.clear();
+
+  EXPECT_TRUE(MatchQuery(FnMatcher).run(OS, S));
+
+  EXPECT_TRUE(OS.str().find("foo.cc:1:1: note: \"root\" binds here") !=
+              std::string::npos);
+  EXPECT_TRUE(OS.str().find("foo.cc:2:1: note: \"root\" binds here") !=
+              std::string::npos);
+  EXPECT_TRUE(OS.str().find("bar.cc:1:1: note: \"root\" binds here") !=
+              std::string::npos);
+  EXPECT_TRUE(OS.str().find("bar.cc:2:1: note: \"root\" binds here") !=
+              std::string::npos);
+  EXPECT_TRUE(OS.str().find("4 matches.") != std::string::npos);
+
+  Str.clear();
+
+  EXPECT_TRUE(MatchQuery(FooMatcher).run(OS, S));
+
+  EXPECT_TRUE(OS.str().find("foo.cc:1:1: note: \"root\" binds here") !=
+              std::string::npos);
+  EXPECT_TRUE(OS.str().find("1 match.") != std::string::npos);
+
+  Str.clear();
+
+  EXPECT_TRUE(
+      SetQuery<OutputKind>(&QuerySession::OutKind, OK_Print).run(OS, S));
+  EXPECT_TRUE(MatchQuery(FooMatcher).run(OS, S));
+
+  EXPECT_TRUE(OS.str().find("Binding for \"root\":\nvoid foo1()") !=
+              std::string::npos);
+
+  Str.clear();
+
+  EXPECT_TRUE(SetQuery<OutputKind>(&QuerySession::OutKind, OK_Dump).run(OS, S));
+  EXPECT_TRUE(MatchQuery(FooMatcher).run(OS, S));
+
+  EXPECT_TRUE(OS.str().find("FunctionDecl") != std::string::npos);
+
+  Str.clear();
+
+  EXPECT_TRUE(SetQuery<bool>(&QuerySession::BindRoot, false).run(OS, S));
+  EXPECT_TRUE(MatchQuery(FooMatcher).run(OS, S));
+
+  EXPECT_TRUE(OS.str().find("No bindings.") != std::string::npos);
+
+  Str.clear();
+
+  EXPECT_FALSE(MatchQuery(isArrow()).run(OS, S));
+
+  EXPECT_EQ("Not a valid top-level matcher.\n", OS.str());
+}

unittests/clang-query/QueryParserTest.cpp

+//===---- QueryParserTest.cpp - clang-query test --------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "QueryParser.h"
+#include "Query.h"
+#include "QuerySession.h"
+#include "gtest/gtest.h"
+
+using namespace clang;
+using namespace clang::query;
+
+TEST(QueryParser, NoOp) {
+  QueryRef Q = ParseQuery("");
+  EXPECT_TRUE(isa<NoOpQuery>(Q));
+
+  Q = ParseQuery("\n");
+  EXPECT_TRUE(isa<NoOpQuery>(Q));
+}
+
+TEST(QueryParser, Invalid) {
+  QueryRef Q = ParseQuery("foo");
+  ASSERT_TRUE(isa<InvalidQuery>(Q));
+  EXPECT_EQ("unknown command: foo", cast<InvalidQuery>(Q)->ErrStr);
+}
+
+TEST(QueryParser, Help) {
+  QueryRef Q = ParseQuery("help");
+  ASSERT_TRUE(isa<HelpQuery>(Q));
+
+  Q = ParseQuery("help me");
+  ASSERT_TRUE(isa<InvalidQuery>(Q));
+  EXPECT_EQ("unexpected extra input: ' me'", cast<InvalidQuery>(Q)->ErrStr);
+}
+
+TEST(QueryParser, Set) {
+  QueryRef Q = ParseQuery("set");
+  ASSERT_TRUE(isa<InvalidQuery>(Q));
+  EXPECT_EQ("expected variable name", cast<InvalidQuery>(Q)->ErrStr);
+
+  Q = ParseQuery("set foo bar");
+  ASSERT_TRUE(isa<InvalidQuery>(Q));
+  EXPECT_EQ("unknown variable: 'foo'", cast<InvalidQuery>(Q)->ErrStr);
+
+  Q = ParseQuery("set output");
+  ASSERT_TRUE(isa<InvalidQuery>(Q));
+  EXPECT_EQ("expected variable value", cast<InvalidQuery>(Q)->ErrStr);
+
+  Q = ParseQuery("set bind-root true foo");
+  ASSERT_TRUE(isa<InvalidQuery>(Q));
+  EXPECT_EQ("unexpected extra input: ' foo'", cast<InvalidQuery>(Q)->ErrStr);
+
+  Q = ParseQuery("set output foo");
+  ASSERT_TRUE(isa<InvalidQuery>(Q));
+  EXPECT_EQ("expected 'diag', 'print' or 'dump', got 'foo'",
+            cast<InvalidQuery>(Q)->ErrStr);
+
+  Q = ParseQuery("set output dump");
+  ASSERT_TRUE(isa<SetQuery<OutputKind> >(Q));
+  EXPECT_EQ(&QuerySession::OutKind, cast<SetQuery<OutputKind> >(Q)->Var);
+  EXPECT_EQ(OK_Dump, cast<SetQuery<OutputKind> >(Q)->Value);
+
+  Q = ParseQuery("set bind-root foo");
+  ASSERT_TRUE(isa<InvalidQuery>(Q));
+  EXPECT_EQ("expected 'true' or 'false', got 'foo'",
+            cast<InvalidQuery>(Q)->ErrStr);
+
+  Q = ParseQuery("set bind-root true");
+  ASSERT_TRUE(isa<SetQuery<bool> >(Q));
+  EXPECT_EQ(&QuerySession::BindRoot, cast<SetQuery<bool> >(Q)->Var);
+  EXPECT_EQ(true, cast<SetQuery<bool> >(Q)->Value);
+}
+
+TEST(QueryParser, Match) {
+  QueryRef Q = ParseQuery("match decl()");
+  ASSERT_TRUE(isa<MatchQuery>(Q));
+  EXPECT_TRUE(cast<MatchQuery>(Q)->Matcher.canConvertTo<Decl>());
+
+  Q = ParseQuery("m stmt()");
+  ASSERT_TRUE(isa<MatchQuery>(Q));
+  EXPECT_TRUE(cast<MatchQuery>(Q)->Matcher.canConvertTo<Stmt>());
+}