Initial version of libInterpreter.

[v.g.vassilev]

Summary

Hey, as we discussed in private I started working towards moving some code from cling to clang.

This is the very minimal patch (barely functional) which I intend to propose publicly.

I'd be happy to address your comments first and then send this to cfe-commits.

[v.g.vassilev]

clang-format

[spyffe]

Very exciting, thank you for this patch!
I have a couple of top-level comments first, since I'm still getting familiar with this code.
The primary concern I have is the exit; secondary concerns are composability with other tools.

lib/Interpreter/IncrementalParser.cpp

87

If the incremental parser is no longer usable, that should be communicated out to the client rather than exiting.

143

It looks like we should be sharing this between the incremental interpreter and clang-import-test.

lib/Interpreter/IncrementalParser.h

45

Could this be an accessor, given that Consumer can always be derived from CI or Act?

54

It's not clear to me whether your code is going to rely on Consumer to do anything important, or whether it's just a no-op with the intent that the consumer of Parse's return value actually does the work. Can you clarify this relationship?

If Consumer is a no-op, then it's fine being opaque.
If it is going to be doing work, then at least from LLDB's perspective I would like to be able to provide my own.

57

Does getCI() need to be public? I know it's const, but it does seem to leak the abstraction that IncrementalParser is providing.

[rsmith]

lib/Interpreter/IncrementalParser.cpp

289–311

Can you factor this out into perhaps a method on Parser, along with the corresponding code in clang::ParseAST?

353

This is wrong: LLVM_ON_WIN32 is about the build environment where Clang is compiled, not about the target. The relevant consideration is likely the DelayedTemplateParsing LangOption.

353–362

This sounds like a bug. What tokens specifically are being left behind here?

366

It seems dangerous to lex an extra token in asserts mode. Maybe lex that final token unconditionally? (Given that we expect to have already lexed an EOF token here, lexing another one should be very cheap, so the #ifndef NDEBUG seems unnecessary.)

[rsmith]

lib/Interpreter/IncrementalParser.cpp

103

Creating this should be the responsibility of the cling driver binary; it's not appropriate for this library to choose to write textual diagnostics to stderr. (Consider the needs of a REPL in an IDE for instance.)

112

Likewise, this piece should be handled by the driver. This library should not assume any relationship between argv[0] and the compiler's resource directory.

119–122

Setting this up should be the responsibility of the cling driver binary, not of this library. This library should not be setting global state.

[v.g.vassilev]

lib/Interpreter/IncrementalParser.cpp

87

This is copy from the driver. We can fix this by introducing some sort of callback layer.

143

Absolutely. This code is very similar to what is done in runToolOnCodeWithArgs from tooling library (see one of the FIXMEs). Maybe we could reuse it even more there.

lib/Interpreter/IncrementalParser.h

54

Cling caches the produced top level decls before piping them to codegen. We do that for cases like int i = 42; error_here; int j; We do not want to produce code for int i = 42 if the input overall had errors.

This consumer is our special error recovery handler (keeping track of the state changes in the compiler). I can imagine that something similar could be used by lldb to make some parts of the AST visible when switching between frames.

I intentionally kept it as simple as possible. I can add accessors to it on demand.

57

You are right, giving access to the CompilerInstance though is our current point of end user customization.

Based on this leaking abstraction I will start playing with clang-import-test to figure out how much code could be reused.

[v.g.vassilev]

Version of this work landed as https://reviews.llvm.org/D96033

include/clang/Frontend/FrontendAction.h

public:
   /// several objects which would normally be owned by the
   /// CompilerInstance. When processing AST input files, these objects should
   /// generally not be initialized in the CompilerInstance -- they will
   /// automatically be shared with the AST file in between
   /// BeginSourceFile() and EndSourceFile().
   ///
   /// \return True on success; on failure the compilation of this file should
   /// be aborted and neither Execute() nor EndSourceFile() should be called.
-  bool BeginSourceFile(CompilerInstance &CI, const FrontendInputFile &Input);
+  virtual bool BeginSourceFile(CompilerInstance &CI,
+                               const FrontendInputFile &Input);
 
   /// \brief Set the source manager's main input file, and run the action.
   bool Execute();
 
   /// \brief Perform any per-file post processing, deallocate per-file
   /// objects, and run statistics and output file cleanup code.
-  void EndSourceFile();
+  virtual void EndSourceFile();
 
   /// @}
 };
 
 /// \brief Abstract base class to use for AST consumer-based frontend actions.
 class ASTFrontendAction : public FrontendAction {
 protected:
   /// \brief Implement the ExecuteAction interface by running Sema on

include/clang/FrontendTool/Utils.h

 //  This header contains miscellaneous utilities for various front-end actions
 //  which were split from Frontend to minimise Frontend's dependencies.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_FRONTENDTOOL_UTILS_H
 #define LLVM_CLANG_FRONTENDTOOL_UTILS_H
 
+#include <memory>
+
 namespace clang {
 
 class CompilerInstance;
+class FrontendAction;
 
 /// ExecuteCompilerInvocation - Execute the given actions described by the
 /// compiler invocation object in the given compiler instance.
 ///
 /// \return - True on success.
 bool ExecuteCompilerInvocation(CompilerInstance *Clang);
 
-}  // end namespace clang
+std::unique_ptr<FrontendAction> CreateFrontendAction(CompilerInstance &CI);
+
+} // end namespace clang
 
 #endif

lib/CMakeLists.txt

 add_subdirectory(Frontend)
 add_subdirectory(FrontendTool)
 add_subdirectory(Tooling)
 add_subdirectory(Index)
 if(CLANG_ENABLE_STATIC_ANALYZER)
   add_subdirectory(StaticAnalyzer)
 endif()
 add_subdirectory(Format)
+add_subdirectory(Interpreter)

lib/FrontendTool/ExecuteCompilerInvocation.cpp

#if !defined(CLANG_ENABLE_ARCMT) || !defined(CLANG_ENABLE_STATIC_ANALYZER) \
   || !defined(CLANG_ENABLE_OBJC_REWRITER)
   CI.getDiagnostics().Report(diag::err_fe_action_not_available) << Action;
   return 0;
 #else
   llvm_unreachable("Invalid program action!");
 #endif
 }
 
-static std::unique_ptr<FrontendAction>
-CreateFrontendAction(CompilerInstance &CI) {
+std::unique_ptr<FrontendAction>
+clang::CreateFrontendAction(CompilerInstance &CI) {
   // Create the underlying action.
   std::unique_ptr<FrontendAction> Act = CreateFrontendBaseAction(CI);
   if (!Act)
     return nullptr;
 
   const FrontendOptions &FEOpts = CI.getFrontendOpts();
 
   if (FEOpts.FixAndRecompile) {

lib/Interpreter/CMakeLists.txt

+set(LLVM_LINK_COMPONENTS
+   native
+   Target
+   core
+  )
+
+add_clang_library(clangInterpreter
+  IncrementalParser.cpp
+  Interpreter.cpp
+
+  LINK_LIBS
+  clangAST
+  clangAnalysis
+  clangBasic
+  clangEdit
+  clangLex
+  clangSema
+  clangCodeGen
+  clangFrontendTool
+  )

lib/Interpreter/IncrementalParser.h

+//===--- IncrementalParser.h - Incremental Compilation ----------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements the class which performs incremental code compilation.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INTERPRETER_INCREMENTALPARSER_H
+#define LLVM_CLANG_INTERPRETER_INCREMENTALPARSER_H
+
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/Error.h"
+
+#include <memory>
+#include <vector>
+
+namespace clang {
+class ASTConsumer;
+class CompilerInstance;
+class DeclGroupRef;
+class FrontendAction;
+class Parser;
+
+/// Provides support for incremental compilation. Keeps track of the state
+/// changes between the subsequent incremental input.
+///
+class IncrementalParser {
+  /// Long-lived, incremental parsing action.
+  std::unique_ptr<FrontendAction> Act;
+
+  /// Compiler instance performing the incremental compilation.
+  std::unique_ptr<CompilerInstance> CI;
+
+  /// Parser.
+  std::unique_ptr<Parser> P;
+
+  /// Consumer to process the produced top level decls. Owned by Act.
+  ASTConsumer *Consumer = nullptr;
+
+  /// Incremental input counter.
+  unsigned InputCount = 0;
+
+  /// List containing every parsed top level decl.
+  std::vector<DeclGroupRef> TopLevelDecls;
+
+public:
+  IncrementalParser(std::vector<const char *> &ClangArgv);
+  ~IncrementalParser();
+
+  const CompilerInstance *getCI() const { return CI.get(); }
+
+  llvm::Expected<llvm::ArrayRef<DeclGroupRef>> Parse(llvm::StringRef Input);
+
+private:
+  bool ParseOrWrapTopLevelDecl();
+};
+} // end namespace clang
+
+#endif // LLVM_CLANG_INTERPRETER_INCREMENTALPARSER_H

lib/Interpreter/IncrementalParser.cpp

+//===--------- IncrementalParser.cpp - Incremental Compilation  -----------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements the class which performs incremental code compilation.
+//
+//===----------------------------------------------------------------------===//
+
+#include "IncrementalParser.h"
+
+#include "clang/Basic/TargetInfo.h"
+#include "clang/CodeGen/BackendUtil.h"
+#include "clang/CodeGen/CodeGenAction.h"
+#include "clang/CodeGen/ModuleBuilder.h"
+#include "clang/CodeGen/ObjectFilePCHContainerOperations.h"
+#include "clang/Driver/Compilation.h"
+#include "clang/Driver/Driver.h"
+#include "clang/Driver/Job.h"
+#include "clang/Driver/Options.h"
+#include "clang/Driver/Tool.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Frontend/FrontendAction.h"
+#include "clang/Frontend/FrontendDiagnostic.h"
+#include "clang/Frontend/TextDiagnosticBuffer.h"
+#include "clang/Frontend/TextDiagnosticPrinter.h"
+#include "clang/Frontend/VerifyDiagnosticConsumer.h"
+#include "clang/FrontendTool/Utils.h"
+#include "clang/Lex/PreprocessorOptions.h"
+#include "clang/Parse/Parser.h"
+#include "clang/Sema/Sema.h"
+
+#include "llvm/Option/ArgList.h"
+#include "llvm/Support/CrashRecoveryContext.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/Signals.h"
+#include "llvm/Support/Timer.h"
+
+#include <sstream>
+
+using namespace clang;
+
+// FIXME: Figure out how to unify with namespace init_convenience from
+//        tools/clang-import-test/clang-import-test.cpp and
+//        examples/clang-interpreter/main.cpp
+namespace {
+/// \brief Retrieves the clang CC1 specific flags out of the compilation's
+/// jobs. Returns NULL on error.
+static const llvm::opt::ArgStringList *
+GetCC1Arguments(DiagnosticsEngine *Diagnostics,
+                driver::Compilation *Compilation) {
+  // We expect to get back exactly one Command job, if we didn't something
+  // failed. Extract that job from the Compilation.
+  const driver::JobList &Jobs = Compilation->getJobs();
+  if (!Jobs.size() || !isa<driver::Command>(*Jobs.begin())) {
+    // FIXME: diagnose this...
+    return nullptr;
+  }
+
+  // The one job we find should be to invoke clang again.
+  const driver::Command *Cmd = cast<driver::Command>(&(*Jobs.begin()));
+  if (llvm::StringRef(Cmd->getCreator().getName()) != "clang") {
+    // FIXME: diagnose this...
+    return nullptr;
+  }
+
+  return &Cmd->getArguments();
+}
+
+static void LLVMErrorHandler(void *UserData, const std::string &Message,
+                             bool GenCrashDiag) {
+  DiagnosticsEngine &Diags = *static_cast<DiagnosticsEngine *>(UserData);
+
+  Diags.Report(diag::err_fe_error_backend) << Message;
+
+  // Run the interrupt handlers to make sure any special cleanups get done, in
+  // particular that we remove files registered with RemoveFileOnSignal.
+  llvm::sys::RunInterruptHandlers();
+
+  // We cannot recover from llvm errors.  When reporting a fatal error, exit
+  // with status 70 to generate crash diagnostics.  For BSD systems this is
+  // defined as an internal software error.  Otherwise, exit with status 1.
+  exit(GenCrashDiag ? 70 : 1);
+}
+
+static std::unique_ptr<CompilerInstance>
+CreateCI(const driver::ArgStringList &Argv) {
+  std::unique_ptr<CompilerInstance> Clang(new CompilerInstance());
+  IntrusiveRefCntPtr<DiagnosticIDs> DiagID(new DiagnosticIDs());
+
+  // Register the support for object-file-wrapped Clang modules.
+  auto PCHOps = Clang->getPCHContainerOperations();
+  PCHOps->registerWriter(llvm::make_unique<ObjectFilePCHContainerWriter>());
+  PCHOps->registerReader(llvm::make_unique<ObjectFilePCHContainerReader>());
+
+  // Buffer diagnostics from argument parsing so that we can output them using
+  // a well formed diagnostic object.
+  IntrusiveRefCntPtr<DiagnosticOptions> DiagOpts = new DiagnosticOptions();
+  TextDiagnosticBuffer *DiagsBuffer = new TextDiagnosticBuffer;
+  DiagnosticsEngine Diags(DiagID, &*DiagOpts, DiagsBuffer);
+  bool Success = CompilerInvocation::CreateFromArgs(
+      Clang->getInvocation(), Argv.begin(), Argv.end(), Diags);
+
+  // Infer the builtin include path if unspecified.
+  if (Clang->getHeaderSearchOpts().UseBuiltinIncludes &&
+      Clang->getHeaderSearchOpts().ResourceDir.empty())
+    Clang->getHeaderSearchOpts().ResourceDir =
+        CompilerInvocation::GetResourcesPath(Argv[0], nullptr);
+
+  // Create the actual diagnostics engine.
+  Clang->createDiagnostics();
+  if (!Clang->hasDiagnostics())
+    return nullptr;
+
+  // Set an error handler, so that any LLVM backend diagnostics go through our
+  // error handler.
+  llvm::install_fatal_error_handler(
+      LLVMErrorHandler, static_cast<void *>(&Clang->getDiagnostics()));
+
+  DiagsBuffer->FlushDiagnostics(Clang->getDiagnostics());
+  if (!Success)
+    return nullptr;
+
+  // FIXME: Merge with CompilerInstance::ExecuteAction.
+  llvm::MemoryBuffer *MB = llvm::MemoryBuffer::getMemBuffer("").release();
+  Clang->getPreprocessorOpts().addRemappedFile("<<< inputs >>>", MB);
+
+  Clang->setTarget(TargetInfo::CreateTargetInfo(
+      Clang->getDiagnostics(), Clang->getInvocation().TargetOpts));
+  if (!Clang->hasTarget())
+    return 0;
+
+  Clang->getTarget().adjust(Clang->getLangOpts());
+
+  return Clang;
+}
+
+static std::unique_ptr<CompilerInstance>
+CreateCI(std::vector<const char *> &ClangArgv) {
+
+  // If we don't know ClangArgv0 or the address of main() at this point, try
+  // to guess it anyway (it's possible on some platforms).
+  std::string MainExecutableName =
+      llvm::sys::fs::getMainExecutable(nullptr, nullptr);
+
+  ClangArgv.insert(ClangArgv.begin(), MainExecutableName.c_str());
+
+  if (std::find(ClangArgv.begin(), ClangArgv.end(), " -x") == ClangArgv.end()) {
+    // We do C++ by default; append right after argv[0] if no "-x" given
+    ClangArgv.push_back("-x");
+    ClangArgv.push_back("c++");
+  }
+  // By adding -c, we force the driver to treat compilation as the last phase.
+  // It will then issue warnings via Diagnostics about un-used options that
+  // would have been used for linking. If the user provided a compiler name as
+  // the original argv[0], this will be treated as a linker input thanks to
+  // insertng a new argv[0] above. All un-used options get collected by
+  // UnusedInputdiagConsumer and get stripped out later.
+  ClangArgv.push_back("-c");
+
+  // Put a dummy C++ file on to ensure there's at least one compile job for the
+  // driver to construct. If the user specified some other argument that
+  // prevents compilation, e.g. -E or something like -version, we may still end
+  // up with no jobs but then this is the user's fault.
+  ClangArgv.push_back("<<< inputs >>>");
+
+  CompilerInvocation Invocation;
+  // Buffer diagnostics from argument parsing so that we can output them using a
+  // well formed diagnostic object.
+  IntrusiveRefCntPtr<DiagnosticIDs> DiagID(new DiagnosticIDs());
+  IntrusiveRefCntPtr<DiagnosticOptions> DiagOpts = new DiagnosticOptions();
+  TextDiagnosticBuffer *DiagsBuffer = new TextDiagnosticBuffer;
+  DiagnosticsEngine Diags(DiagID, &*DiagOpts, DiagsBuffer);
+  unsigned MissingArgIndex, MissingArgCount;
+  std::unique_ptr<llvm::opt::OptTable> Opts(driver::createDriverOptTable());
+  llvm::opt::InputArgList ParsedArgs =
+      Opts->ParseArgs(ArrayRef<const char *>(ClangArgv).slice(1),
+                      MissingArgIndex, MissingArgCount);
+  ParseDiagnosticArgs(*DiagOpts, ParsedArgs, &Diags);
+
+  driver::Driver Driver(/*MainBinaryName*/ ClangArgv[0],
+                        llvm::sys::getDefaultTargetTriple(), Diags);
+  Driver.setCheckInputsExist(false); // the input comes from mem buffers
+  llvm::ArrayRef<const char *> RF = llvm::makeArrayRef(ClangArgv);
+  std::unique_ptr<driver::Compilation> Compilation(Driver.BuildCompilation(RF));
+
+  if (Compilation->getArgs().hasArg(driver::options::OPT_v))
+    Compilation->getJobs().Print(llvm::errs(), "\n", /*Quote*/ false);
+
+  const driver::ArgStringList *CC1Args =
+      GetCC1Arguments(&Diags, Compilation.get());
+  if (!CC1Args)
+    return 0;
+
+  return CreateCI(*CC1Args);
+}
+
+} // anonymous namespace
+
+/// A custom action enabling the incremental processing functionality.
+///
+/// The usual \p FrontendAction expects one call to ExecuteAction and once it
+/// sees a call to \p EndSourceFile it deletes some of the important objects
+/// such as \p Preprocessor and \p Sema assuming no further input will come.
+///
+/// \p IncrementalAction ensures it keep its underlying action's objects alive
+/// as long as the \p IncrementalParser needs them.
+///
+class IncrementalAction : public WrapperFrontendAction {
+private:
+  bool IsTerminating = false;
+
+public:
+  IncrementalAction(CompilerInstance &CI)
+      : WrapperFrontendAction(CreateFrontendAction(CI)) {}
+
+  void ExecuteAction() override {
+    CompilerInstance &CI = getCompilerInstance();
+    assert(CI.hasPreprocessor() && "No PP!");
+
+    // FIXME: Move the truncation aspect of this into Sema, we delayed this till
+    // here so the source manager would be initialized.
+    if (hasCodeCompletionSupport() &&
+        !CI.getFrontendOpts().CodeCompletionAt.FileName.empty())
+      CI.createCodeCompletionConsumer();
+
+    // Use a code completion consumer?
+    CodeCompleteConsumer *CompletionConsumer = nullptr;
+    if (CI.hasCodeCompletionConsumer())
+      CompletionConsumer = &CI.getCodeCompletionConsumer();
+
+    if (!CI.hasSema())
+      CI.createSema(getTranslationUnitKind(), CompletionConsumer);
+
+    Preprocessor &PP = CI.getPreprocessor();
+    PP.enableIncrementalProcessing();
+    PP.EnterMainSourceFile();
+  }
+
+  void EndSourceFile() override {
+    if (IsTerminating) {
+      WrapperFrontendAction::EndSourceFile();
+    }
+  }
+
+  void FinalizeAction() {
+    assert(!IsTerminating && "Already finalized!");
+    IsTerminating = true;
+    EndSourceFile();
+  }
+};
+
+IncrementalParser::IncrementalParser(std::vector<const char *> &ClangArgv) {
+  // FIXME: Investigate if we could use runToolOnCodeWithArgs from tooling. It
+  // can replace the boilerplate code for creation of the compiler instance.
+  CI = CreateCI(ClangArgv);
+
+  Act = llvm::make_unique<IncrementalAction>(*CI);
+  CI->ExecuteAction(*Act);
+  Consumer = &CI->getASTConsumer();
+  P.reset(
+      new Parser(CI->getPreprocessor(), CI->getSema(), /*SkipBodies*/ false));
+  P->Initialize();
+}
+
+IncrementalParser::~IncrementalParser() {
+  Consumer->HandleTranslationUnit(getCI()->getASTContext());
+  ((IncrementalAction *)Act.get())->FinalizeAction();
+  // Our error handler depends on the Diagnostics object, which we're
+  // potentially about to delete. Uninstall the handler now so that any
+  // later errors use the default handling behavior instead.
+  llvm::remove_fatal_error_handler();
+}
+
+bool IncrementalParser::ParseOrWrapTopLevelDecl() {
+  // Recover resources if we crash before exiting this method.
+  Sema &S = CI->getSema();
+  llvm::CrashRecoveryContextCleanupRegistrar<Sema> CleanupSema(&S);
+  Sema::SavePendingInstantiationsAndVTableUsesRAII SavedPendingInstantiations(
+      S, /*Enabled*/ true);
+
+  if (!P->isTopLevelDecl()) {
+    // FIXME: Else, wrap in a function and try to parse it.
+  }
+  Parser::DeclGroupPtrTy ADecl;
+  if (P->getCurToken().isNot(tok::eof)) {
+    if (P->ParseFirstTopLevelDecl(ADecl))
+      return true;
+    if (ADecl) {
+      if (!Consumer->HandleTopLevelDecl(ADecl.get()))
+        return true;
+      TopLevelDecls.push_back(ADecl.get());
+    }
+  } else
+    // Skip previous eof due to last incremental input.
+    P->ConsumeToken();
+
+  for (bool AtEOF = false; !AtEOF; AtEOF = P->ParseTopLevelDecl(ADecl)) {
+    // If we got a null return and something *was* parsed, ignore it.  This
+    // is due to a top-level semicolon, an action override, or a parse error
+    // skipping something.
+    if (ADecl) {
+      if (!Consumer->HandleTopLevelDecl(ADecl.get()))
+        return true;
+      TopLevelDecls.push_back(ADecl.get());
+    }
+  }
+
+  return CI->getDiagnostics().hasErrorOccurred();
+}
+
+llvm::Expected<llvm::ArrayRef<DeclGroupRef>>
+IncrementalParser::Parse(llvm::StringRef input) {
+  Preprocessor &PP = CI->getPreprocessor();
+  assert(PP.isIncrementalProcessingEnabled() && "Not in incremental mode!?");
+
+  std::ostringstream SourceName;
+  SourceName << "input_line_" << InputCount++;
+
+  // Create an uninitialized memory buffer, copy code in and append "\n"
+  size_t InputSize = input.size(); // don't include trailing 0
+  // MemBuffer size should *not* include terminating zero
+  std::unique_ptr<llvm::MemoryBuffer> MB(
+      llvm::MemoryBuffer::getNewUninitMemBuffer(InputSize + 1,
+                                                SourceName.str()));
+  char *MBStart = const_cast<char *>(MB->getBufferStart());
+  memcpy(MBStart, input.data(), InputSize);
+  memcpy(MBStart + InputSize, "\n", 2);
+
+  SourceManager &SM = CI->getSourceManager();
+
+  // Create SourceLocation, which will allow clang to order the overload
+  // candidates for example
+  SourceLocation NewLoc = SM.getLocForStartOfFile(SM.getMainFileID())
+                              .getLocWithOffset(InputCount + 2);
+
+  // Create FileID for the current buffer.
+  FileID FID = SM.createFileID(std::move(MB), SrcMgr::C_User, /*LoadedID*/ 0,
+                               /*LoadedOffset*/ 0, NewLoc);
+
+  // NewLoc only used for diags.
+  PP.EnterSourceFile(FID, /*DirLookup*/ 0, NewLoc);
+
+  unsigned lastTransaction = TopLevelDecls.size();
+  if (ParseOrWrapTopLevelDecl())
+    return llvm::make_error<llvm::StringError>("Parsing failed.",
+                                               std::error_code());
+
+#ifdef LLVM_ON_WIN32
+  // Microsoft-specific:
+  // Late parsed templates can leave unswallowed "macro"-like tokens.
+  // They will seriously confuse the Parser when entering the next
+  // source file. So lex until we are EOF.
+  Token Tok;
+  do {
+    PP.Lex(Tok);
+  } while (Tok.isNot(tok::eof));
+#endif
+
+#ifndef NDEBUG
+  Token AssertTok;
+  PP.Lex(AssertTok);
+  assert(AssertTok.is(tok::eof) &&
+         "Lexer must be EOF when starting incremental parse!");
+#endif
+
+  return llvm::makeArrayRef(&TopLevelDecls[lastTransaction],
+                            TopLevelDecls.size() - lastTransaction);
+}

lib/Interpreter/Interpreter.cpp

+//===------ Interpreter.cpp - Incremental Compilation and Execution -------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements the component which performs incremental code
+// compilation and execution.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Interpreter/Interpreter.h"
+
+#include "IncrementalParser.h"
+
+using namespace clang;
+
+Interpreter::Interpreter() {
+  std::vector<const char *> v = {"-Xclang", "-emit-llvm-only"};
+  Initialize(v);
+}
+
+Interpreter::Interpreter(std::vector<const char *> &ClangArgs) {
+  Initialize(ClangArgs);
+}
+
+Interpreter::Interpreter(const llvm::cl::list<std::string> &ClangArgs) {
+  std::vector<const char *> ClangArgv(ClangArgs.size());
+  std::transform(ClangArgs.begin(), ClangArgs.end(), ClangArgv.begin(),
+                 [](const std::string &s) -> const char * { return s.data(); });
+
+  Initialize(ClangArgv);
+}
+
+Interpreter::~Interpreter() {}
+
+void Interpreter::Initialize(std::vector<const char *> &ClangArgs) {
+  IncrParser = llvm::make_unique<IncrementalParser>(ClangArgs);
+}
+
+const CompilerInstance *Interpreter::getCompilerInstance() const {
+  return IncrParser->getCI();
+}
+
+llvm::Expected<llvm::ArrayRef<DeclGroupRef>>
+Interpreter::Process(llvm::StringRef Code) {
+  return IncrParser->Parse(Code);
+}

test/Interpreter/sanity.c

+// RUN: cat %s | \
+// RUN:   cling -Xcc -fno-color-diagnostics -Xcc -Xclang -Xcc -ast-dump \
+// RUN:      -Xcc -Xclang -Xcc -ast-dump-filter -Xcc -Xclang -Xcc Test | \
+// RUN:         FileCheck %s
+
+int TestVar = 12;
+// CHECK: Dumping TestVar:
+// CHECK-NEXT: VarDecl [[var_ptr:0x[0-9a-f]+]] <{{.*}} TestVar 'int' cinit
+// CHECK-NEXT:   IntegerLiteral {{.*}} 'int' 12
+
+void TestFunc() { ++TestVar; }
+// CHECK: Dumping TestFunc:
+// CHECK-NEXT: FunctionDecl {{.*}} TestFunc 'void (void)'
+// CHECK-NEXT:   CompoundStmt{{.*}}
+// CHECK-NEXT:     UnaryOperator{{.*}} 'int' lvalue prefix '++'
+// CHECK-NEXT:       DeclRefExpr{{.*}} 'int' lvalue Var [[var_ptr]] 'TestVar' 'int'
+
+quit

tools/CMakeLists.txt

 create_subdirectory_options(CLANG TOOL)
 
 add_clang_subdirectory(diagtool)
 add_clang_subdirectory(driver)
 add_clang_subdirectory(clang-format)
 add_clang_subdirectory(clang-format-vs)
 add_clang_subdirectory(clang-fuzzer)
 add_clang_subdirectory(clang-import-test)
 add_clang_subdirectory(clang-offload-bundler)
+add_clang_subdirectory(cling)
 
 add_clang_subdirectory(c-index-test)
 
 if(CLANG_ENABLE_ARCMT)
   add_clang_subdirectory(arcmt-test)
   add_clang_subdirectory(c-arcmt-test)
 endif()
 

tools/cling/CMakeLists.txt

+set( LLVM_LINK_COMPONENTS
+#  ${LLVM_TARGETS_TO_BUILD}
+#  Option
+  Support
+  )
+
+add_clang_executable(cling
+  Cling.cpp
+  )
+
+target_link_libraries(cling
+  clangInterpreter
+  clangTooling
+  LLVMLineEditor
+  )
+
+install(TARGETS cling
+  RUNTIME DESTINATION bin)

tools/cling/Cling.cpp

+//===--- tools/cling/Cling.cpp - Cling - the Clang-based REPL -------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+//  This file implements a REPL tool on top of clang.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Interpreter/Interpreter.h"
+
+#include "llvm/LineEditor/LineEditor.h"
+#include "llvm/Support/ManagedStatic.h" // llvm_shutdown
+#include "llvm/Support/TargetSelect.h"  // llvm::Initialize*
+
+static llvm::cl::list<std::string>
+    ClangArgs("Xcc", llvm::cl::ZeroOrMore,
+              llvm::cl::desc("Argument to pass to the CompilerInvocation"),
+              llvm::cl::CommaSeparated);
+
+int main(int argc, const char **argv) {
+  llvm::cl::ParseCommandLineOptions(argc, argv);
+
+  // Initialize targets first, so that --version shows registered targets.
+  llvm::InitializeAllTargets();
+  llvm::InitializeAllTargetMCs();
+  llvm::InitializeAllAsmPrinters();
+  llvm::InitializeAllAsmParsers();
+
+  if (ClangArgs.empty()) {
+    ClangArgs.push_back("-Xclang");
+    ClangArgs.push_back("-emit-llvm-only");
+  }
+  clang::Interpreter Interp(ClangArgs);
+  llvm::LineEditor LE("cling");
+  // FIXME: Add LE.setListCompleter
+  while (llvm::Optional<std::string> Line = LE.readLine()) {
+    if (*Line == "quit")
+      break;
+    // Handle a potential parsing error.
+    // FIXME: We do not really need to do this when running in interactive mode.
+    // The parser prints proper onscreen diagnostics but we still need to make
+    // llvm::Expected<T> think we handled the error.
+    if (!Interp.Process(*Line))
+      continue;
+  }
+
+  llvm::llvm_shutdown();
+
+  return 0;
+}

unittests/CMakeLists.txt

 endif()
 add_subdirectory(ASTMatchers)
 add_subdirectory(AST)
 add_subdirectory(Tooling)
 add_subdirectory(Format)
 add_subdirectory(Rewrite)
 add_subdirectory(Sema)
 add_subdirectory(CodeGen)
+add_subdirectory(Interpreter)
 # FIXME: libclang unit tests are disabled on Windows due
 # to failures, mostly in libclang.VirtualFileOverlay_*.
 if(NOT WIN32 AND CLANG_TOOL_LIBCLANG_BUILD) 
   add_subdirectory(libclang)
 endif()

unittests/Interpreter/CMakeLists.txt

+set(LLVM_LINK_COMPONENTS
+  )
+
+add_clang_unittest(InterpreterTests
+  InterpreterTest.cpp
+  )
+target_link_libraries(InterpreterTests
+  clangInterpreter
+  clangFrontend
+  )

unittests/Interpreter/InterpreterTest.cpp

+//===- unittests/Interpreter/InterpreterTest.cpp --- Interpreter tests ----===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// Unit tests for our Interpreter library.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Interpreter/Interpreter.h"
+
+#include "clang/AST/Decl.h"
+#include "clang/AST/DeclGroup.h"
+
+#include "llvm/ADT/ArrayRef.h"
+
+#include "gtest/gtest.h"
+
+using namespace clang;
+
+namespace {
+
+TEST(InterpreterTest, Sanity) {
+  llvm::ExitOnError ExitOnErr;
+
+  Interpreter Interp;
+  if (auto DeclsOrErr = Interp.process("void g(); void g() {}")) {
+    llvm::ArrayRef<DeclGroupRef> R1 = *DeclsOrErr;
+    EXPECT_EQ(2U, R1.size());
+  }
+  if (auto DeclsOrErr = Interp.process("int i;")) {
+    llvm::ArrayRef<DeclGroupRef> R2 = *DeclsOrErr;
+    EXPECT_EQ(1U, R2.size());
+  }
+}
+
+static std::string DeclToString(DeclGroupRef DGR) {
+  return llvm::cast<NamedDecl>(DGR.getSingleDecl())->getQualifiedNameAsString();
+}
+
+TEST(InterpreterTest, IncrementalInputTopLevelDecls) {
+  Interpreter Interp;
+  auto R1OrErr = Interp.process("int var1 = 42; int f() { return var1; }");
+  // gtest doesn't expand into explicit bool conversions.
+  EXPECT_TRUE(!!R1OrErr);
+  auto R1 = R1OrErr.get();
+  EXPECT_EQ(2U, R1.size());
+  EXPECT_EQ("var1", DeclToString(R1[0]));
+  EXPECT_EQ("f", DeclToString(R1[1]));
+
+  auto R2OrErr = Interp.process("int var2 = f();");
+  EXPECT_TRUE(!!R2OrErr);
+  auto R2 = R2OrErr.get();
+  EXPECT_EQ(1U, R2.size());
+  EXPECT_EQ("var2", DeclToString(R2[0]));
+}
+
+// Here we test whether the user can mix declarations and statements. The
+// interpreter should be smart enough to recognize the declarations from the
+// statements and wrap the latter into a declaration, producing valid code.
+TEST(InterpreterTest, DeclsAndStatements) {
+  Interpreter Interp;
+  auto R1OrErr = Interp.process(
+      "int var1 = 42; extern \"C\" int printf(const char*, ...);");
+  // gtest doesn't expand into explicit bool conversions.
+  EXPECT_TRUE(!!R1OrErr);
+
+  auto R1 = R1OrErr.get();
+  EXPECT_EQ(2U, R1.size());
+
+  // FIXME: Add support for wrapping and running statements.
+  auto R2OrErr =
+      Interp.process("var1++; printf(\"var1 value is %d\\n\", var1);");
+  EXPECT_FALSE(!!R2OrErr);
+  auto Err = R2OrErr.takeError();
+  EXPECT_EQ("Parsing failed.", llvm::toString(std::move(Err)));
+}
+
+} // end anonymous namespace