[clang-repl] Implement partial translation units and error recovery.

[v.g.vassilev]

Summary

https://reviews.llvm.org/D96033 contained a discussion regarding efficient modeling of error recovery. @rjmccall has outlined the key ideas:

Conceptually, we can split the translation unit into a sequence of partial translation units (PTUs). Every declaration will be associated with a unique PTU that owns it.

The first key insight here is that the owning PTU isn't always the "active" (most recent) PTU, and it isn't always the PTU that the declaration "comes from". A new declaration (that isn't a redeclaration or specialization of anything) does belong to the active PTU. A template specialization, however, belongs to the most recent PTU of all the declarations in its signature - mostly that means that it can be pulled into a more recent PTU by its template arguments.

The second key insight is that processing a PTU might extend an earlier PTU. Rolling back the later PTU shouldn't throw that extension away. For example, if the second PTU defines a template, and the third PTU requires that template to be instantiated at float, that template specialization is still part of the second PTU. Similarly, if the fifth PTU uses an inline function belonging to the fourth, that definition still belongs to the fourth. When we go to emit code in a new PTU, we map each declaration we have to emit back to its owning PTU and emit it in a new module for just the extensions to that PTU. We keep track of all the modules we've emitted for a PTU so that we can unload them all if we decide to roll it back.

Most declarations/definitions will only refer to entities from the same or earlier PTUs. However, it is possible (primarily by defining a previously-declared entity, but also through templates or ADL) for an entity that belongs to one PTU to refer to something from a later PTU. We will have to keep track of this and prevent unwinding to later PTU when we recognize it.

Fortunately, this should be very rare; and crucially, we don't have to do the bookkeeping for this if we've only got one PTU, e.g. in normal compilation. Otherwise, PTUs after the first just need to record enough metadata to be able to revert any changes they've made to declarations belonging to earlier PTUs, e.g. to redeclaration chains or template specialization lists.

It should even eventually be possible for PTUs to provide their own slab allocators which can be thrown away as part of rolling back the PTU. We can maintain a notion of the active allocator and allocate things like Stmt/Expr nodes in it, temporarily changing it to the appropriate PTU whenever we go to do something like instantiate a function template. More care will be required when allocating declarations and types, though.

We would want the PTU to be efficiently recoverable from a Decl; I'm not sure how best to do that. An easy option that would cover most declarations would be to make multiple TranslationUnitDecls and parent the declarations appropriately, but I don't think that's good enough for things like member function templates, since an instantiation of that would still be parented by its original class. Maybe we can work this into the DC chain somehow, like how lexical DCs are.

This patch teaches clang-repl how to recover from errors by disconnecting the most recent PTU and update the primary PTU lookup tables. For instance:

./clang-repl
clang-repl> int i = 12; error;
In file included from <<< inputs >>>:1:
input_line_0:1:13: error: C++ requires a type specifier for all declarations
int i = 12; error;
                 ^
error: Parsing failed.
clang-repl> int i = 13; extern "C" int printf(const char*,...);
clang-repl> auto r1 = printf("i=%d\n", i);
i=13
clang-repl> quit

[rsmith]

I'm (happily) surprised this required so few changes!

clang/include/clang/Basic/LangOptions.h

702–703

I don't think this is clear enough about the difference between TU_Prefix and TU_Partial. I think the difference is:

• TU_Prefix is an incomplete prefix of a translation unit. Because it's not complete, we don't perform (most) finalization steps (eg, template instantiation) at the end.
• TU_Partial is a complete translation unit that we might nonetheless incrementally extend later. Because it is complete (and we might want to generate code for it), we do perform template instantiation, but because it might be extended later, we don't warn if it declares or uses undefined internal-linkage symbols.

I wonder if TU_Incremental would be a better name than TU_Partial.

[sgraenitz]

A few minor notes

clang/include/clang/Interpreter/Interpreter.h

60–61

ErrorOr<T> has different semantics and is still in use, so the name could be confusing. Idiomatic usage for Expected<T> would rather be like:

auto PTU = Parse(Code);
if (!PTU)
  return PTU.takeError();

The patch didn't introduce it, but it might be a good chance for improvement.

clang/lib/Interpreter/IncrementalParser.cpp

210

Where does it point, if the very first TU fails? Maybe worth noting in the assert and/or adding a test case?

262–264

auto PTU = ParseOrWrapTopLevelDecl();
if (!PTU)
  return PTU.takeError();

[v.g.vassilev]

Address review comments.

clang/include/clang/Basic/LangOptions.h

702–703

Good point. Fixed.

clang/include/clang/Interpreter/Interpreter.h

60–61

Ha, that made the code much nicer looking! Thanks!!

clang/lib/Interpreter/IncrementalParser.cpp

210

The very first TU contains the compiler builtins and it is created when the ASTContext is created. Not having a PreviousTU would mean that we did not initialize the CompilerInstance properly.

Added some description in the assert.

[teemperor]

@v.g.vassilev For LLDB you need to change the TypeSystemClang::SetExternalSource function to this (it just moves the setHasExternalLexicalStorage one line up. You can just add that change and the compilation fix to this commit.

void TypeSystemClang::SetExternalSource(
    llvm::IntrusiveRefCntPtr<ExternalASTSource> &ast_source_up) {
  ASTContext &ast = getASTContext();
  ast.getTranslationUnitDecl()->setHasExternalLexicalStorage(true);
  ast.setExternalSource(ast_source_up);
}

The problem is that a few ASTs in LLDB change the ExternalSource that is originally set, but the LazyGenerationalUpdatePtr remembers the initial ExternalSource and tries to access it to complete the redecl chain (but the original ExternalSource just got replaced and deleted). The better fix is to not even create that original ExternalSource, but I can fix that in a follow up as that seems out of scope for this patch.

[v.g.vassilev]

In D104918#2870582, @teemperor wrote:

@v.g.vassilev For LLDB you need to change the TypeSystemClang::SetExternalSource function to this (it just moves the setHasExternalLexicalStorage one line up. You can just add that change and the compilation fix to this commit.

void TypeSystemClang::SetExternalSource(
    llvm::IntrusiveRefCntPtr<ExternalASTSource> &ast_source_up) {
  ASTContext &ast = getASTContext();
  ast.getTranslationUnitDecl()->setHasExternalLexicalStorage(true);
  ast.setExternalSource(ast_source_up);
}

The problem is that a few ASTs in LLDB change the ExternalSource that is originally set, but the LazyGenerationalUpdatePtr remembers the initial ExternalSource and tries to access it to complete the redecl chain (but the original ExternalSource just got replaced and deleted). The better fix is to not even create that original ExternalSource, but I can fix that in a follow up as that seems out of scope for this patch.

Thanks @teemperor! I will add your suggestion and recommit.

clang/include/clang/AST/ASTContext.h

using TemplateOrSpecializationInfo =
       llvm::PointerUnion<VarTemplateDecl *, MemberSpecializationInfo *>;
 
 private:
   friend class ASTDeclReader;
   friend class ASTReader;
   friend class ASTWriter;
   template <class> friend class serialization::AbstractTypeReader;
   friend class CXXRecordDecl;
+  friend class IncrementalParser;
 
   /// A mapping to contain the template or declaration that
   /// a variable declaration describes or was instantiated from,
   /// respectively.
   ///
   /// For non-templates, this value will be NULL. For variable
   /// declarations that describe a variable template, this will be a
   /// pointer to a VarTemplateDecl. For static data members
private:
   /// Mapping that stores parameterIndex values for ParmVarDecls when
   /// that value exceeds the bitfield size of ParmVarDeclBits.ParameterIndex.
   using ParameterIndexTable = llvm::DenseMap<const VarDecl *, unsigned>;
   ParameterIndexTable ParamIndices;
 
   ImportDecl *FirstLocalImport = nullptr;
   ImportDecl *LastLocalImport = nullptr;
 
-  TranslationUnitDecl *TUDecl;
+  TranslationUnitDecl *TUDecl = nullptr;
   mutable ExternCContextDecl *ExternCContext = nullptr;
   mutable BuiltinTemplateDecl *MakeIntegerSeqDecl = nullptr;
   mutable BuiltinTemplateDecl *TypePackElementDecl = nullptr;
 
   /// The associated SourceManager object.
   SourceManager &SourceMgr;
 
   /// The language options used to create the AST associated with
private:
 
   /// Keeps track of the deallocated DeclListNodes for future reuse.
   DeclListNode *ListNodeFreeList = nullptr;
 
 public:
   IdentifierTable &Idents;
   SelectorTable &Selectors;
   Builtin::Context &BuiltinInfo;
+  const TranslationUnitKind TUKind;
   mutable DeclarationNameTable DeclarationNames;
   IntrusiveRefCntPtr<ExternalASTSource> ExternalSource;
   ASTMutationListener *Listener = nullptr;
 
   /// Returns the clang bytecode interpreter context.
   interp::Context &getInterpContext();
 
   /// Returns the dynamic AST node parent map context.
public:
   /// or an ImportDecl nominating another module that has initializers.
   void addModuleInitializer(Module *M, Decl *Init);
 
   void addLazyModuleInitializers(Module *M, ArrayRef<uint32_t> IDs);
 
   /// Get the initializations to perform when importing a module, if any.
   ArrayRef<Decl*> getModuleInitializers(Module *M);
 
-  TranslationUnitDecl *getTranslationUnitDecl() const { return TUDecl; }
+  TranslationUnitDecl *getTranslationUnitDecl() const {
+    return TUDecl->getMostRecentDecl();
+  }
+  void addTranslationUnitDecl() {
+    assert(!TUDecl || TUKind == TU_Incremental);
+    TranslationUnitDecl *NewTUDecl = TranslationUnitDecl::Create(*this);
+    if (TraversalScope.empty() || TraversalScope.back() == TUDecl)
+      TraversalScope = {NewTUDecl};
+    if (TUDecl)
+      NewTUDecl->setPreviousDecl(TUDecl);
+    TUDecl = NewTUDecl;
+  }
 
   ExternCContextDecl *getExternCContextDecl() const;
   BuiltinTemplateDecl *getMakeIntegerSeqDecl() const;
   BuiltinTemplateDecl *getTypePackElementDecl() const;
 
   // Builtin Types.
   CanQualType VoidTy;
   CanQualType BoolTy;
#include "clang/Basic/RISCVVTypes.def"
 
   // Implicitly-declared type 'struct _GUID'.
   mutable TagDecl *MSGuidTagDecl = nullptr;
 
   /// Keep track of CUDA/HIP device-side variables ODR-used by host code.
   llvm::DenseSet<const VarDecl *> CUDADeviceVarODRUsedByHost;
 
   ASTContext(LangOptions &LOpts, SourceManager &SM, IdentifierTable &idents,
-             SelectorTable &sels, Builtin::Context &builtins);
+             SelectorTable &sels, Builtin::Context &builtins,
+             TranslationUnitKind TUKind);
   ASTContext(const ASTContext &) = delete;
   ASTContext &operator=(const ASTContext &) = delete;
   ~ASTContext();
 
   /// Attach an external AST source to the AST context.
   ///
   /// The external AST source provides the ability to load parts of
   /// the abstract syntax tree as needed from some external storage,

clang/include/clang/AST/Decl.h

 class TemplateArgumentListInfo;
 class TemplateParameterList;
 class TypeAliasTemplateDecl;
 class TypeLoc;
 class UnresolvedSetImpl;
 class VarTemplateDecl;
 
 /// The top declaration context.
-class TranslationUnitDecl : public Decl, public DeclContext {
+class TranslationUnitDecl : public Decl,
+                            public DeclContext,
+                            public Redeclarable<TranslationUnitDecl> {
+  using redeclarable_base = Redeclarable<TranslationUnitDecl>;
+
+  TranslationUnitDecl *getNextRedeclarationImpl() override {
+    return getNextRedeclaration();
+  }
+
+  TranslationUnitDecl *getPreviousDeclImpl() override {
+    return getPreviousDecl();
+  }
+
+  TranslationUnitDecl *getMostRecentDeclImpl() override {
+    return getMostRecentDecl();
+  }
+
   ASTContext &Ctx;
 
   /// The (most recently entered) anonymous namespace for this
   /// translation unit, if one has been created.
   NamespaceDecl *AnonymousNamespace = nullptr;
 
   explicit TranslationUnitDecl(ASTContext &ctx);
 
   virtual void anchor();
 
 public:
+  using redecl_range = redeclarable_base::redecl_range;
+  using redecl_iterator = redeclarable_base::redecl_iterator;
+
+  using redeclarable_base::getMostRecentDecl;
+  using redeclarable_base::getPreviousDecl;
+  using redeclarable_base::isFirstDecl;
+  using redeclarable_base::redecls;
+  using redeclarable_base::redecls_begin;
+  using redeclarable_base::redecls_end;
+
   ASTContext &getASTContext() const { return Ctx; }
 
   NamespaceDecl *getAnonymousNamespace() const { return AnonymousNamespace; }
   void setAnonymousNamespace(NamespaceDecl *D) { AnonymousNamespace = D; }
 
   static TranslationUnitDecl *Create(ASTContext &C);
 
   // Implement isa/cast/dyncast/etc.

clang/include/clang/AST/Redeclarable.h

protected:
 
   decl_type *getNextRedeclaration() const {
     return RedeclLink.getPrevious(static_cast<const decl_type *>(this));
   }
 
 public:
   friend class ASTDeclReader;
   friend class ASTDeclWriter;
+  friend class IncrementalParser;
 
   Redeclarable(const ASTContext &Ctx)
       : RedeclLink(LatestDeclLink(Ctx)),
         First(static_cast<decl_type *>(this)) {}
 
   /// Return the previous declaration of this declaration or NULL if this
   /// is the first declaration.
   decl_type *getPreviousDecl() {

clang/include/clang/Basic/LangOptions.h

enum TranslationUnitKind {
   /// The translation unit is a complete translation unit.
   TU_Complete,
 
   /// The translation unit is a prefix to a translation unit, and is
   /// not complete.
   TU_Prefix,
 
   /// The translation unit is a module.
-  TU_Module
+  TU_Module,
+
+  /// The translation unit is a is a complete translation unit that we might
+  /// incrementally extend later.
+  TU_Incremental
 };
 
 } // namespace clang
 
 #endif // LLVM_CLANG_BASIC_LANGOPTIONS_H

clang/include/clang/Interpreter/Interpreter.h

 //===--- Interpreter.h - Incremental Compilation and Execution---*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file defines the component which performs incremental code
 // compilation and execution.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_INTERPRETER_INTERPRETER_H
 #define LLVM_CLANG_INTERPRETER_INTERPRETER_H
 
-#include "clang/Interpreter/Transaction.h"
+#include "clang/Interpreter/PartialTranslationUnit.h"
 
 #include "llvm/Support/Error.h"
 
 #include <memory>
 #include <vector>
 
 namespace llvm {
 namespace orc {
class Interpreter {
 
   Interpreter(std::unique_ptr<CompilerInstance> CI, llvm::Error &Err);
 
 public:
   ~Interpreter();
   static llvm::Expected<std::unique_ptr<Interpreter>>
   create(std::unique_ptr<CompilerInstance> CI);
   const CompilerInstance *getCompilerInstance() const;
-  llvm::Expected<Transaction &> Parse(llvm::StringRef Code);
-  llvm::Error Execute(Transaction &T);
+  llvm::Expected<PartialTranslationUnit &> Parse(llvm::StringRef Code);
+  llvm::Error Execute(PartialTranslationUnit &T);
   llvm::Error ParseAndExecute(llvm::StringRef Code) {
-    auto ErrOrTransaction = Parse(Code);
-    if (auto Err = ErrOrTransaction.takeError())
-      return Err;
-    if (ErrOrTransaction->TheModule)
-      return Execute(*ErrOrTransaction);
+    auto PTU = Parse(Code);
+    if (!PTU)
+      return PTU.takeError();
+    if (PTU->TheModule)
+      return Execute(*PTU);
     return llvm::Error::success();
   }
 };
 } // namespace clang
 
 #endif // LLVM_CLANG_INTERPRETER_INTERPRETER_H

clang/include/clang/Interpreter/PartialTranslationUnit.h

 //===--- Transaction.h - Incremental Compilation and Execution---*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file defines utilities tracking the incrementally processed pieces of
 // code.
 //
 //===----------------------------------------------------------------------===//
 
-#ifndef LLVM_CLANG_INTERPRETER_TRANSACTION_H
-#define LLVM_CLANG_INTERPRETER_TRANSACTION_H
+#ifndef LLVM_CLANG_INTERPRETER_PARTIALTRANSLATIONUNIT_H
+#define LLVM_CLANG_INTERPRETER_PARTIALTRANSLATIONUNIT_H
 
 #include <memory>
-#include <vector>
 
 namespace llvm {
 class Module;
 }
 
 namespace clang {
 
-class DeclGroupRef;
+class TranslationUnitDecl;
 
 /// The class keeps track of various objects created as part of processing
 /// incremental inputs.
-struct Transaction {
-  /// The decls created for the input.
-  std::vector<clang::DeclGroupRef> Decls;
+struct PartialTranslationUnit {
+  TranslationUnitDecl *TUPart = nullptr;
 
   /// The llvm IR produced for the input.
   std::unique_ptr<llvm::Module> TheModule;
 };
 } // namespace clang
 
-#endif // LLVM_CLANG_INTERPRETER_TRANSACTION_H
+#endif // LLVM_CLANG_INTERPRETER_PARTIALTRANSLATIONUNIT_H

clang/include/clang/Lex/Preprocessor.h

class Preprocessor {
 
   /// Empty line handler.
   EmptylineHandler *Emptyline = nullptr;
 
   /// True if we want to ignore EOF token and continue later on (thus
   /// avoid tearing the Lexer and etc. down).
   bool IncrementalProcessing = false;
 
+public:
   /// The kind of translation unit we are processing.
-  TranslationUnitKind TUKind;
+  const TranslationUnitKind TUKind;
 
+private:
   /// The code-completion handler.
   CodeCompletionHandler *CodeComplete = nullptr;
 
   /// The file that we're performing code-completion for, if any.
   const FileEntry *CodeCompletionFile = nullptr;
 
   /// The offset in file for the code-completion point.
   unsigned CodeCompletionOffset = 0;

clang/include/clang/Sema/Sema.h

public:
 
   /// The kind of translation unit we are processing.
   ///
   /// When we're processing a complete translation unit, Sema will perform
   /// end-of-translation-unit semantic tasks (such as creating
   /// initializers for tentative definitions in C) once parsing has
   /// completed. Modules and precompiled headers perform different kinds of
   /// checks.
-  TranslationUnitKind TUKind;
+  const TranslationUnitKind TUKind;
 
   llvm::BumpPtrAllocator BumpAlloc;
 
   /// The number of SFINAE diagnostics that have been trapped.
   unsigned NumSFINAEErrors;
 
   typedef llvm::DenseMap<ParmVarDecl *, llvm::TinyPtrVector<ParmVarDecl *>>
     UnparsedDefaultArgInstantiationsMap;

clang/lib/AST/ASTContext.cpp

static bool isAddrSpaceMapManglingEnabled(const TargetInfo &TI,
   case LangOptions::ASMM_Off:
     return false;
   }
   llvm_unreachable("getAddressSpaceMapMangling() doesn't cover anything.");
 }
 
 ASTContext::ASTContext(LangOptions &LOpts, SourceManager &SM,
                        IdentifierTable &idents, SelectorTable &sels,
-                       Builtin::Context &builtins)
+                       Builtin::Context &builtins, TranslationUnitKind TUKind)
     : ConstantArrayTypes(this_()), FunctionProtoTypes(this_()),
       TemplateSpecializationTypes(this_()),
       DependentTemplateSpecializationTypes(this_()), AutoTypes(this_()),
       SubstTemplateTemplateParmPacks(this_()),
       CanonTemplateTemplateParms(this_()), SourceMgr(SM), LangOpts(LOpts),
       NoSanitizeL(new NoSanitizeList(LangOpts.NoSanitizeFiles, SM)),
       XRayFilter(new XRayFunctionFilter(LangOpts.XRayAlwaysInstrumentFiles,
                                         LangOpts.XRayNeverInstrumentFiles,
                                         LangOpts.XRayAttrListFiles, SM)),
       ProfList(new ProfileList(LangOpts.ProfileListFiles, SM)),
       PrintingPolicy(LOpts), Idents(idents), Selectors(sels),
-      BuiltinInfo(builtins), DeclarationNames(*this), Comments(SM),
-      CommentCommandTraits(BumpAlloc, LOpts.CommentOpts),
+      BuiltinInfo(builtins), TUKind(TUKind), DeclarationNames(*this),
+      Comments(SM), CommentCommandTraits(BumpAlloc, LOpts.CommentOpts),
       CompCategories(this_()), LastSDM(nullptr, 0) {
-  TUDecl = TranslationUnitDecl::Create(*this);
-  TraversalScope = {TUDecl};
+  addTranslationUnitDecl();
 }
 
 ASTContext::~ASTContext() {
   // Release the DenseMaps associated with DeclContext objects.
   // FIXME: Is this the ideal solution?
   ReleaseDeclContextMaps();
 
   // Call all of the deallocation functions on all of their targets.
if (!ExternCContext)
     ExternCContext = ExternCContextDecl::Create(*this, getTranslationUnitDecl());
 
   return ExternCContext;
 }
 
 BuiltinTemplateDecl *
 ASTContext::buildBuiltinTemplateDecl(BuiltinTemplateKind BTK,
                                      const IdentifierInfo *II) const {
-  auto *BuiltinTemplate = BuiltinTemplateDecl::Create(*this, TUDecl, II, BTK);
+  auto *BuiltinTemplate =
+      BuiltinTemplateDecl::Create(*this, getTranslationUnitDecl(), II, BTK);
   BuiltinTemplate->setImplicit();
-  TUDecl->addDecl(BuiltinTemplate);
+  getTranslationUnitDecl()->addDecl(BuiltinTemplate);
 
   return BuiltinTemplate;
 }
 
 BuiltinTemplateDecl *
 ASTContext::getMakeIntegerSeqDecl() const {
   if (!MakeIntegerSeqDecl)
     MakeIntegerSeqDecl = buildBuiltinTemplateDecl(BTK__make_integer_seq,
#include "clang/Basic/RISCVVTypes.def"
   InitBuiltinType(BFloat16Ty, BuiltinType::BFloat16);
 
   // Builtin type used to help define __builtin_va_list.
   VaListTagDecl = nullptr;
 
   // MSVC predeclares struct _GUID, and we need it to create MSGuidDecls.
   if (LangOpts.MicrosoftExt || LangOpts.Borland) {
     MSGuidTagDecl = buildImplicitRecord("_GUID");
-    TUDecl->addDecl(MSGuidTagDecl);
+    getTranslationUnitDecl()->addDecl(MSGuidTagDecl);
   }
 }
 
 DiagnosticsEngine &ASTContext::getDiagnostics() const {
   return SourceMgr.getDiagnostics();
 }
 
 AttrVec& ASTContext::getDeclAttrs(const Decl *D) {

 // getCFConstantStringType - Return the type used for constant CFStrings.
 QualType ASTContext::getCFConstantStringType() const {
   return getTypedefType(getCFConstantStringDecl());
 }
 
 QualType ASTContext::getObjCSuperType() const {
   if (ObjCSuperType.isNull()) {
     RecordDecl *ObjCSuperTypeDecl = buildImplicitRecord("objc_super");
-    TUDecl->addDecl(ObjCSuperTypeDecl);
+    getTranslationUnitDecl()->addDecl(ObjCSuperTypeDecl);
     ObjCSuperType = getTagDeclType(ObjCSuperTypeDecl);
   }
   return ObjCSuperType;
 }
 
 void ASTContext::setCFConstantStringType(QualType T) {
   const auto *TD = T->castAs<TypedefType>();
   CFConstantStringTypeDecl = cast<TypedefDecl>(TD->getDecl());

clang/lib/AST/Decl.cpp

 
 // Defined here so that it can be inlined into its direct callers.
 bool Decl::isOutOfLine() const {
   return !getLexicalDeclContext()->Equals(getDeclContext());
 }
 
 TranslationUnitDecl::TranslationUnitDecl(ASTContext &ctx)
     : Decl(TranslationUnit, nullptr, SourceLocation()),
-      DeclContext(TranslationUnit), Ctx(ctx) {}
+      DeclContext(TranslationUnit), redeclarable_base(ctx), Ctx(ctx) {}
 
 //===----------------------------------------------------------------------===//
 // NamedDecl Implementation
 //===----------------------------------------------------------------------===//
 
 // Visibility rules aren't rigorously externally specified, but here
 // are the basic principles behind what we implement:
 //

clang/lib/AST/DeclBase.cpp

bool DeclContext::Encloses(const DeclContext *DC) const {
   for (; DC; DC = DC->getParent())
     if (DC->getPrimaryContext() == this)
       return true;
   return false;
 }
 
 DeclContext *DeclContext::getPrimaryContext() {
   switch (getDeclKind()) {
-  case Decl::TranslationUnit:
   case Decl::ExternCContext:
   case Decl::LinkageSpec:
   case Decl::Export:
   case Decl::Block:
   case Decl::Captured:
   case Decl::OMPDeclareReduction:
   case Decl::OMPDeclareMapper:
   case Decl::RequiresExprBody:
     // There is only one DeclContext for these entities.
     return this;
 
+  case Decl::TranslationUnit:
+    return static_cast<TranslationUnitDecl *>(this)->getFirstDecl();
   case Decl::Namespace:
     // The original namespace is our primary context.
     return static_cast<NamespaceDecl *>(this)->getOriginalNamespace();
 
   case Decl::ObjCMethod:
     return this;
 
   case Decl::ObjCInterface:
default:
 
     assert(getDeclKind() >= Decl::firstFunction &&
            getDeclKind() <= Decl::lastFunction &&
           "Unknown DeclContext kind");
     return this;
   }
 }
 
-void
-DeclContext::collectAllContexts(SmallVectorImpl<DeclContext *> &Contexts){
-  Contexts.clear();
+template <typename T>
+void collectAllContextsImpl(T *Self, SmallVectorImpl<DeclContext *> &Contexts) {
+  for (T *D = Self->getMostRecentDecl(); D; D = D->getPreviousDecl())
+    Contexts.push_back(D);
 
-  if (getDeclKind() != Decl::Namespace) {
-    Contexts.push_back(this);
-    return;
-  }
+  std::reverse(Contexts.begin(), Contexts.end());
 
-  auto *Self = static_cast<NamespaceDecl *>(this);
-  for (NamespaceDecl *N = Self->getMostRecentDecl(); N;
-       N = N->getPreviousDecl())
-    Contexts.push_back(N);
+void DeclContext::collectAllContexts(SmallVectorImpl<DeclContext *> &Contexts) {
+  Contexts.clear();
 
-  std::reverse(Contexts.begin(), Contexts.end());
+  Decl::Kind Kind = getDeclKind();
+
+  if (Kind == Decl::TranslationUnit)
+    collectAllContextsImpl(static_cast<TranslationUnitDecl *>(this), Contexts);
+  else if (Kind == Decl::Namespace)
+    collectAllContextsImpl(static_cast<NamespaceDecl *>(this), Contexts);
+  else
+    Contexts.push_back(this);
 }
 
 std::pair<Decl *, Decl *>
 DeclContext::BuildDeclChain(ArrayRef<Decl *> Decls,
                             bool FieldsAlreadyLoaded) {
   // Build up a chain of declarations via the Decl::NextInContextAndBits field.
   Decl *FirstNewDecl = nullptr;
   Decl *PrevDecl = nullptr;

clang/lib/Frontend/ASTUnit.cpp

AST->PP = std::make_shared<Preprocessor>(
       AST->getSourceManager(), HeaderInfo, AST->ModuleLoader,
       /*IILookup=*/nullptr,
       /*OwnsHeaderSearch=*/false);
   Preprocessor &PP = *AST->PP;
 
   if (ToLoad >= LoadASTOnly)
     AST->Ctx = new ASTContext(*AST->LangOpts, AST->getSourceManager(),
                               PP.getIdentifierTable(), PP.getSelectorTable(),
-                              PP.getBuiltinInfo());
+                              PP.getBuiltinInfo(),
+                              AST->getTranslationUnitKind());
 
   DisableValidationForModuleKind disableValid =
       DisableValidationForModuleKind::None;
   if (::getenv("LIBCLANG_DISABLE_PCH_VALIDATION"))
     disableValid = DisableValidationForModuleKind::All;
   AST->Reader = new ASTReader(
       PP, *AST->ModuleCache, AST->Ctx.get(), PCHContainerRdr, {},
       /*isysroot=*/"",

clang/lib/Frontend/CompilerInstance.cpp

 }
 
 // ASTContext
 
 void CompilerInstance::createASTContext() {
   Preprocessor &PP = getPreprocessor();
   auto *Context = new ASTContext(getLangOpts(), PP.getSourceManager(),
                                  PP.getIdentifierTable(), PP.getSelectorTable(),
-                                 PP.getBuiltinInfo());
+                                 PP.getBuiltinInfo(), PP.TUKind);
   Context->InitBuiltinTypes(getTarget(), getAuxTarget());
   setASTContext(Context);
 }
 
 // ExternalASTSource
 
 void CompilerInstance::createPCHExternalASTSource(
     StringRef Path, DisableValidationForModuleKind DisableValidation,

clang/lib/Interpreter/IncrementalParser.h

 //===--- IncrementalParser.h - Incremental Compilation ----------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file implements the class which performs incremental code compilation.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_LIB_INTERPRETER_INCREMENTALPARSER_H
 #define LLVM_CLANG_LIB_INTERPRETER_INCREMENTALPARSER_H
 
-#include "clang/Interpreter/Transaction.h"
+#include "clang/Interpreter/PartialTranslationUnit.h"
 
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/Error.h"
 
 #include <list>
 #include <memory>
 namespace llvm {
class IncrementalParser {
   /// Consumer to process the produced top level decls. Owned by Act.
   ASTConsumer *Consumer = nullptr;
 
   /// Counts the number of direct user input lines that have been parsed.
   unsigned InputCount = 0;
 
   /// List containing every information about every incrementally parsed piece
   /// of code.
-  std::list<Transaction> Transactions;
+  std::list<PartialTranslationUnit> PTUs;
 
 public:
   IncrementalParser(std::unique_ptr<CompilerInstance> Instance,
                     llvm::LLVMContext &LLVMCtx, llvm::Error &Err);
   ~IncrementalParser();
 
   const CompilerInstance *getCI() const { return CI.get(); }
 
   /// Parses incremental input by creating an in-memory file.
-  ///\returns a \c Transaction which holds information about the \c Decls and
-  /// \c llvm::Module corresponding to the input.
-  llvm::Expected<Transaction &> Parse(llvm::StringRef Input);
+  ///\returns a \c PartialTranslationUnit which holds information about the
+  /// \c TranslationUnitDecl and \c llvm::Module corresponding to the input.
+  llvm::Expected<PartialTranslationUnit &> Parse(llvm::StringRef Input);
 
 private:
-  llvm::Expected<Transaction &> ParseOrWrapTopLevelDecl();
+  llvm::Expected<PartialTranslationUnit &> ParseOrWrapTopLevelDecl();
 };
 } // end namespace clang
 
 #endif // LLVM_CLANG_LIB_INTERPRETER_INCREMENTALPARSER_H

clang/lib/Interpreter/IncrementalParser.cpp

 //===--------- IncrementalParser.cpp - Incremental Compilation  -----------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file implements the class which performs incremental code compilation.
 //
 //===----------------------------------------------------------------------===//
 
 #include "IncrementalParser.h"
 
+#include "clang/AST/DeclContextInternals.h"
 #include "clang/CodeGen/BackendUtil.h"
 #include "clang/CodeGen/CodeGenAction.h"
 #include "clang/CodeGen/ModuleBuilder.h"
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/FrontendAction.h"
 #include "clang/FrontendTool/Utils.h"
 #include "clang/Parse/Parser.h"
 #include "clang/Sema/Sema.h"
IncrementalAction(CompilerInstance &CI, llvm::LLVMContext &LLVMCtx,
             LLVM_FALLTHROUGH;
           case frontend::EmitLLVMOnly:
             Act.reset(new EmitLLVMOnlyAction(&LLVMCtx));
             break;
           }
           return Act;
         }()) {}
   FrontendAction *getWrapped() const { return WrappedAction.get(); }
+  TranslationUnitKind getTranslationUnitKind() override {
+    return TU_Incremental;
+  }
   void ExecuteAction() override {
     CompilerInstance &CI = getCompilerInstance();
     assert(CI.hasPreprocessor() && "No PP!");
 
     // FIXME: Move the truncation aspect of this into Sema, we delayed this till
     // here so the source manager would be initialized.
     if (hasCodeCompletionSupport() &&
         !CI.getFrontendOpts().CodeCompletionAt.FileName.empty())
IncrementalParser::IncrementalParser(std::unique_ptr<CompilerInstance> Instance,
   Consumer = &CI->getASTConsumer();
   P.reset(
       new Parser(CI->getPreprocessor(), CI->getSema(), /*SkipBodies=*/false));
   P->Initialize();
 }
 
 IncrementalParser::~IncrementalParser() { Act->FinalizeAction(); }
 
-llvm::Expected<Transaction &> IncrementalParser::ParseOrWrapTopLevelDecl() {
-  DiagnosticsEngine &Diags = getCI()->getDiagnostics();
-
-  if (Diags.hasErrorOccurred())
-    llvm::report_fatal_error("Previous input had errors, "
-                             "recovery not yet supported",
-                             /*GenCrashDiag=*/false);
-
+llvm::Expected<PartialTranslationUnit &>
+IncrementalParser::ParseOrWrapTopLevelDecl() {
   // Recover resources if we crash before exiting this method.
   Sema &S = CI->getSema();
   llvm::CrashRecoveryContextCleanupRegistrar<Sema> CleanupSema(&S);
   Sema::GlobalEagerInstantiationScope GlobalInstantiations(S, /*Enabled=*/true);
   Sema::LocalEagerInstantiationScope LocalInstantiations(S);
 
+  PTUs.emplace_back(PartialTranslationUnit());
+  PartialTranslationUnit &LastPTU = PTUs.back();
+  // Add a new PTU.
+  ASTContext &C = S.getASTContext();
+  C.addTranslationUnitDecl();
+  LastPTU.TUPart = C.getTranslationUnitDecl();
+
   // Skip previous eof due to last incremental input.
-  if (P->getCurToken().is(tok::eof))
+  if (P->getCurToken().is(tok::eof)) {
     P->ConsumeToken();
-
-  Transactions.emplace_back(Transaction());
-  Transaction &LastTransaction = Transactions.back();
+    // FIXME: Clang does not call ExitScope on finalizing the regular TU, we
+    // might want to do that around HandleEndOfTranslationUnit.
+    P->ExitScope();
+    S.CurContext = nullptr;
+    // Start a new PTU.
+    P->EnterScope(Scope::DeclScope);
+    S.ActOnTranslationUnitScope(P->getCurScope());
+  }
 
   Parser::DeclGroupPtrTy ADecl;
   for (bool AtEOF = P->ParseFirstTopLevelDecl(ADecl); !AtEOF;
        AtEOF = P->ParseTopLevelDecl(ADecl)) {
     // If we got a null return and something *was* parsed, ignore it.  This
     // is due to a top-level semicolon, an action override, or a parse error
     // skipping something.
     if (ADecl && !Consumer->HandleTopLevelDecl(ADecl.get()))
       return llvm::make_error<llvm::StringError>("Parsing failed. "
                                                  "The consumer rejected a decl",
                                                  std::error_code());
-    LastTransaction.Decls.push_back(ADecl.get());
+  }
+
+  DiagnosticsEngine &Diags = getCI()->getDiagnostics();
+  if (Diags.hasErrorOccurred()) {
+    TranslationUnitDecl *MostRecentTU = C.getTranslationUnitDecl();
+    TranslationUnitDecl *PreviousTU = MostRecentTU->getPreviousDecl();
+    assert(PreviousTU && "Must have a TU from the ASTContext initialization!");
+    TranslationUnitDecl *FirstTU = MostRecentTU->getFirstDecl();
+    assert(FirstTU);
+    FirstTU->RedeclLink.setLatest(PreviousTU);
+    C.TUDecl = PreviousTU;
+    S.TUScope->setEntity(PreviousTU);
+
+    // Clean up the lookup table
+    if (StoredDeclsMap *Map = PreviousTU->getLookupPtr()) {
+      for (auto I = Map->begin(); I != Map->end(); ++I) {
+        StoredDeclsList &List = I->second;
+        DeclContextLookupResult R = List.getLookupResult();
+        for (NamedDecl *D : R)
+          if (D->getTranslationUnitDecl() == MostRecentTU)
+            List.remove(D);
+        if (List.isNull())
+          Map->erase(I);
+      }
+    }
+
+    // FIXME: Do not reset the pragma handlers.
+    Diags.Reset();
+    return llvm::make_error<llvm::StringError>("Parsing failed.",
+                                               std::error_code());
   }
 
   // Process any TopLevelDecls generated by #pragma weak.
   for (Decl *D : S.WeakTopLevelDecls()) {
     DeclGroupRef DGR(D);
-    LastTransaction.Decls.push_back(DGR);
     Consumer->HandleTopLevelDecl(DGR);
   }
 
   LocalInstantiations.perform();
   GlobalInstantiations.perform();
 
-  Consumer->HandleTranslationUnit(S.getASTContext());
-
-  if (Diags.hasErrorOccurred())
-    return llvm::make_error<llvm::StringError>("Parsing failed.",
-                                               std::error_code());
+  Consumer->HandleTranslationUnit(C);
 
-  return LastTransaction;
+  return LastPTU;
 }
 
 static CodeGenerator *getCodeGen(FrontendAction *Act) {
   IncrementalAction *IncrAct = static_cast<IncrementalAction *>(Act);
   FrontendAction *WrappedAct = IncrAct->getWrapped();
   if (!WrappedAct->hasIRSupport())
     return nullptr;
   return static_cast<CodeGenAction *>(WrappedAct)->getCodeGenerator();
 }
 
-llvm::Expected<Transaction &> IncrementalParser::Parse(llvm::StringRef input) {
+llvm::Expected<PartialTranslationUnit &>
+IncrementalParser::Parse(llvm::StringRef input) {
   Preprocessor &PP = CI->getPreprocessor();
   assert(PP.isIncrementalProcessingEnabled() && "Not in incremental mode!?");
 
   std::ostringstream SourceName;
   SourceName << "input_line_" << InputCount++;
 
   // Create an uninitialized memory buffer, copy code in and append "\n"
   size_t InputSize = input.size(); // don't include trailing 0
FileID FID = SM.createFileID(std::move(MB), SrcMgr::C_User, /*LoadedID=*/0,
                                /*LoadedOffset=*/0, NewLoc);
 
   // NewLoc only used for diags.
   if (PP.EnterSourceFile(FID, /*DirLookup=*/0, NewLoc))
     return llvm::make_error<llvm::StringError>("Parsing failed. "
                                                "Cannot enter source file.",
                                                std::error_code());
 
-  auto ErrOrTransaction = ParseOrWrapTopLevelDecl();
-  if (auto Err = ErrOrTransaction.takeError())
-    return std::move(Err);
+  auto PTU = ParseOrWrapTopLevelDecl();
+  if (!PTU)
+    return PTU.takeError();
 
   if (PP.getLangOpts().DelayedTemplateParsing) {
     // Microsoft-specific:
     // Late parsed templates can leave unswallowed "macro"-like tokens.
     // They will seriously confuse the Parser when entering the next
     // source file. So lex until we are EOF.
     Token Tok;
     do {
       PP.Lex(Tok);
     } while (Tok.isNot(tok::eof));
   }
 
   Token AssertTok;
   PP.Lex(AssertTok);
   assert(AssertTok.is(tok::eof) &&
          "Lexer must be EOF when starting incremental parse!");
 
   if (CodeGenerator *CG = getCodeGen(Act.get())) {
     std::unique_ptr<llvm::Module> M(CG->ReleaseModule());
-    CG->StartModule("incr_module_" + std::to_string(Transactions.size()),
+    CG->StartModule("incr_module_" + std::to_string(PTUs.size()),
                     M->getContext());
 
-    ErrOrTransaction->TheModule = std::move(M);
+    PTU->TheModule = std::move(M);
   }
 
-  return ErrOrTransaction;
+  return PTU;
 }
 } // end namespace clang

clang/lib/Interpreter/Interpreter.cpp

if (Err)
     return std::move(Err);
   return std::move(Interp);
 }
 
 const CompilerInstance *Interpreter::getCompilerInstance() const {
   return IncrParser->getCI();
 }
 
-llvm::Expected<Transaction &> Interpreter::Parse(llvm::StringRef Code) {
+llvm::Expected<PartialTranslationUnit &>
+Interpreter::Parse(llvm::StringRef Code) {
   return IncrParser->Parse(Code);
 }
 
-llvm::Error Interpreter::Execute(Transaction &T) {
+llvm::Error Interpreter::Execute(PartialTranslationUnit &T) {
   assert(T.TheModule);
   if (!IncrExecutor) {
     const llvm::Triple &Triple =
         getCompilerInstance()->getASTContext().getTargetInfo().getTriple();
     llvm::Error Err = llvm::Error::success();
     IncrExecutor = std::make_unique<IncrementalExecutor>(*TSCtx, Err, Triple);
 
     if (Err)

clang/lib/Sema/Sema.cpp

: ExternalSource(nullptr), isMultiplexExternalSource(false),
       FullyCheckedComparisonCategories(
           static_cast<unsigned>(ComparisonCategoryType::Last) + 1),
       SatisfactionCache(Context), AccessCheckingSFINAE(false),
       InNonInstantiationSFINAEContext(false), NonInstantiationEntries(0),
       ArgumentPackSubstitutionIndex(-1), CurrentInstantiationScope(nullptr),
       DisableTypoCorrection(false), TyposCorrected(0), AnalysisWarnings(*this),
       ThreadSafetyDeclCache(nullptr), VarDataSharingAttributesStack(nullptr),
       CurScope(nullptr), Ident_super(nullptr), Ident___float128(nullptr) {
+  assert(pp.TUKind == TUKind);
   TUScope = nullptr;
   isConstantEvaluatedOverride = false;
 
   LoadedExternalKnownNamespaces = false;
   for (unsigned I = 0; I != NSAPI::NumNSNumberLiteralMethods; ++I)
     NSNumberLiteralMethods[I] = nullptr;
 
   if (getLangOpts().ObjC)

clang/lib/Serialization/ASTReader.cpp

if (NumCurrentElementsDeserializing) {
     // We arrange to not care about the complete redeclaration chain while we're
     // deserializing. Just remember that the AST has marked this one as complete
     // but that it's not actually complete yet, so we know we still need to
     // complete it later.
     PendingIncompleteDeclChains.push_back(const_cast<Decl*>(D));
     return;
   }
 
+  if (!D->getDeclContext()) {
+    assert(isa<TranslationUnitDecl>(D) && "Not a TU?");
+    return;
+  }
+
   const DeclContext *DC = D->getDeclContext()->getRedeclContext();
 
   // If this is a named declaration, complete it by looking it up
   // within its context.
   //
   // FIXME: Merging a function definition should merge
   // all mergeable entities within it.
   if (isa<TranslationUnitDecl>(DC) || isa<NamespaceDecl>(DC) ||

clang/tools/clang-import-test/clang-import-test.cpp

std::unique_ptr<CompilerInstance> BuildCompilerInstance() {
   Ins->createSourceManager(Ins->getFileManager());
   Ins->createPreprocessor(TU_Complete);
 
   return Ins;
 }
 
 std::unique_ptr<ASTContext>
 BuildASTContext(CompilerInstance &CI, SelectorTable &ST, Builtin::Context &BC) {
+  auto &PP = CI.getPreprocessor();
   auto AST = std::make_unique<ASTContext>(
       CI.getLangOpts(), CI.getSourceManager(),
-      CI.getPreprocessor().getIdentifierTable(), ST, BC);
+      PP.getIdentifierTable(), ST, BC, PP.TUKind);
   AST->InitBuiltinTypes(CI.getTarget());
   return AST;
 }
 
 std::unique_ptr<CodeGenerator> BuildCodeGen(CompilerInstance &CI,
                                             llvm::LLVMContext &LLVMCtx) {
   StringRef ModuleName("$__module");
   return std::unique_ptr<CodeGenerator>(CreateLLVMCodeGen(

clang/unittests/AST/ASTVectorTest.cpp

 
 namespace {
 class ASTVectorTest : public ::testing::Test {
 protected:
   ASTVectorTest()
       : FileMgr(FileMgrOpts), DiagID(new DiagnosticIDs()),
         Diags(DiagID, new DiagnosticOptions, new IgnoringDiagConsumer()),
         SourceMgr(Diags, FileMgr), Idents(LangOpts, nullptr),
-        Ctxt(LangOpts, SourceMgr, Idents, Sels, Builtins) {}
+        Ctxt(LangOpts, SourceMgr, Idents, Sels, Builtins, TU_Complete) {}
 
   FileSystemOptions FileMgrOpts;
   FileManager FileMgr;
   IntrusiveRefCntPtr<DiagnosticIDs> DiagID;
   DiagnosticsEngine Diags;
   SourceManager SourceMgr;
   LangOptions LangOpts;
   IdentifierTable Idents;

clang/unittests/Interpreter/IncrementalProcessingTest.cpp

const Function *getGlobalInit(llvm::Module *M) {
   return nullptr;
 }
 
 TEST(IncrementalProcessing, EmitCXXGlobalInitFunc) {
   std::vector<const char *> ClangArgv = {"-Xclang", "-emit-llvm-only"};
   auto CI = llvm::cantFail(IncrementalCompilerBuilder::create(ClangArgv));
   auto Interp = llvm::cantFail(Interpreter::create(std::move(CI)));
 
-  std::array<clang::Transaction *, 2> Transactions;
+  std::array<clang::PartialTranslationUnit *, 2> PTUs;
 
-  Transactions[0] = &llvm::cantFail(Interp->Parse(TestProgram1));
-  ASSERT_TRUE(Transactions[0]->TheModule);
-  ASSERT_TRUE(Transactions[0]->TheModule->getFunction("funcForProg1"));
-
-  Transactions[1] = &llvm::cantFail(Interp->Parse(TestProgram2));
-  ASSERT_TRUE(Transactions[1]->TheModule);
-  ASSERT_TRUE(Transactions[1]->TheModule->getFunction("funcForProg2"));
+  PTUs[0] = &llvm::cantFail(Interp->Parse(TestProgram1));
+  ASSERT_TRUE(PTUs[0]->TheModule);
+  ASSERT_TRUE(PTUs[0]->TheModule->getFunction("funcForProg1"));
+
+  PTUs[1] = &llvm::cantFail(Interp->Parse(TestProgram2));
+  ASSERT_TRUE(PTUs[1]->TheModule);
+  ASSERT_TRUE(PTUs[1]->TheModule->getFunction("funcForProg2"));
   // First code should not end up in second module:
-  ASSERT_FALSE(Transactions[1]->TheModule->getFunction("funcForProg1"));
+  ASSERT_FALSE(PTUs[1]->TheModule->getFunction("funcForProg1"));
 
   // Make sure global inits exist and are unique:
-  const Function *GlobalInit1 = getGlobalInit(Transactions[0]->TheModule.get());
+  const Function *GlobalInit1 = getGlobalInit(PTUs[0]->TheModule.get());
   ASSERT_TRUE(GlobalInit1);
 
-  const Function *GlobalInit2 = getGlobalInit(Transactions[1]->TheModule.get());
+  const Function *GlobalInit2 = getGlobalInit(PTUs[1]->TheModule.get());
   ASSERT_TRUE(GlobalInit2);
 
   ASSERT_FALSE(GlobalInit1->getName() == GlobalInit2->getName());
 }
 
 } // end anonymous namespace

clang/unittests/Interpreter/InterpreterTest.cpp

createInterpreter(const Args &ExtraArgs = {},
   Args ClangArgs = {"-Xclang", "-emit-llvm-only"};
   ClangArgs.insert(ClangArgs.end(), ExtraArgs.begin(), ExtraArgs.end());
   auto CI = cantFail(clang::IncrementalCompilerBuilder::create(ClangArgs));
   if (Client)
     CI->getDiagnostics().setClient(Client, /*ShouldOwnClient=*/false);
   return cantFail(clang::Interpreter::create(std::move(CI)));
 }
 
+static size_t DeclsSize(TranslationUnitDecl *PTUDecl) {
+  return std::distance(PTUDecl->decls().begin(), PTUDecl->decls().end());
+}
+
 TEST(InterpreterTest, Sanity) {
   std::unique_ptr<Interpreter> Interp = createInterpreter();
-  Transaction &R1(cantFail(Interp->Parse("void g(); void g() {}")));
-  EXPECT_EQ(2U, R1.Decls.size());
 
-  Transaction &R2(cantFail(Interp->Parse("int i;")));
-  EXPECT_EQ(1U, R2.Decls.size());
+  using PTU = PartialTranslationUnit;
+
+  PTU &R1(cantFail(Interp->Parse("void g(); void g() {}")));
+  EXPECT_EQ(2U, DeclsSize(R1.TUPart));
+
+  PTU &R2(cantFail(Interp->Parse("int i;")));
+  EXPECT_EQ(1U, DeclsSize(R2.TUPart));
 }
 
-static std::string DeclToString(DeclGroupRef DGR) {
-  return llvm::cast<NamedDecl>(DGR.getSingleDecl())->getQualifiedNameAsString();
+static std::string DeclToString(Decl *D) {
+  return llvm::cast<NamedDecl>(D)->getQualifiedNameAsString();
 }
 
 TEST(InterpreterTest, IncrementalInputTopLevelDecls) {
   std::unique_ptr<Interpreter> Interp = createInterpreter();
-  auto R1OrErr = Interp->Parse("int var1 = 42; int f() { return var1; }");
+  auto R1 = Interp->Parse("int var1 = 42; int f() { return var1; }");
   // gtest doesn't expand into explicit bool conversions.
-  EXPECT_TRUE(!!R1OrErr);
-  auto R1 = R1OrErr->Decls;
-  EXPECT_EQ(2U, R1.size());
-  EXPECT_EQ("var1", DeclToString(R1[0]));
-  EXPECT_EQ("f", DeclToString(R1[1]));
-
-  auto R2OrErr = Interp->Parse("int var2 = f();");
-  EXPECT_TRUE(!!R2OrErr);
-  auto R2 = R2OrErr->Decls;
-  EXPECT_EQ(1U, R2.size());
-  EXPECT_EQ("var2", DeclToString(R2[0]));
+  EXPECT_TRUE(!!R1);
+  auto R1DeclRange = R1->TUPart->decls();
+  EXPECT_EQ(2U, DeclsSize(R1->TUPart));
+  EXPECT_EQ("var1", DeclToString(*R1DeclRange.begin()));
+  EXPECT_EQ("f", DeclToString(*(++R1DeclRange.begin())));
+
+  auto R2 = Interp->Parse("int var2 = f();");
+  EXPECT_TRUE(!!R2);
+  auto R2DeclRange = R2->TUPart->decls();
+  EXPECT_EQ(1U, DeclsSize(R2->TUPart));
+  EXPECT_EQ("var2", DeclToString(*R2DeclRange.begin()));
 }
 
 TEST(InterpreterTest, Errors) {
   Args ExtraArgs = {"-Xclang", "-diagnostic-log-file", "-Xclang", "-"};
 
   // Create the diagnostic engine with unowned consumer.
   std::string DiagnosticOutput;
   llvm::raw_string_ostream DiagnosticsOS(DiagnosticOutput);
   auto DiagPrinter = std::make_unique<TextDiagnosticPrinter>(
       DiagnosticsOS, new DiagnosticOptions());
 
   auto Interp = createInterpreter(ExtraArgs, DiagPrinter.get());
   auto Err = Interp->Parse("intentional_error v1 = 42; ").takeError();
   using ::testing::HasSubstr;
   EXPECT_THAT(DiagnosticsOS.str(),
               HasSubstr("error: unknown type name 'intentional_error'"));
   EXPECT_EQ("Parsing failed.", llvm::toString(std::move(Err)));
 
-#ifdef GTEST_HAS_DEATH_TEST
-  EXPECT_DEATH((void)Interp->Parse("int var1 = 42;"), "");
-#endif
+  auto RecoverErr = Interp->Parse("int var1 = 42;");
+  EXPECT_TRUE(!!RecoverErr);
 }
 
 // Here we test whether the user can mix declarations and statements. The
 // interpreter should be smart enough to recognize the declarations from the
 // statements and wrap the latter into a declaration, producing valid code.
 TEST(InterpreterTest, DeclsAndStatements) {
   Args ExtraArgs = {"-Xclang", "-diagnostic-log-file", "-Xclang", "-"};
 
   // Create the diagnostic engine with unowned consumer.
   std::string DiagnosticOutput;
   llvm::raw_string_ostream DiagnosticsOS(DiagnosticOutput);
   auto DiagPrinter = std::make_unique<TextDiagnosticPrinter>(
       DiagnosticsOS, new DiagnosticOptions());
 
   auto Interp = createInterpreter(ExtraArgs, DiagPrinter.get());
-  auto R1OrErr = Interp->Parse(
+  auto R1 = Interp->Parse(
       "int var1 = 42; extern \"C\" int printf(const char*, ...);");
   // gtest doesn't expand into explicit bool conversions.
-  EXPECT_TRUE(!!R1OrErr);
+  EXPECT_TRUE(!!R1);
 
-  auto R1 = R1OrErr->Decls;
-  EXPECT_EQ(2U, R1.size());
+  auto *PTU1 = R1->TUPart;
+  EXPECT_EQ(2U, DeclsSize(PTU1));
 
   // FIXME: Add support for wrapping and running statements.
-  auto R2OrErr = Interp->Parse("var1++; printf(\"var1 value %d\\n\", var1);");
-  EXPECT_FALSE(!!R2OrErr);
+  auto R2 = Interp->Parse("var1++; printf(\"var1 value %d\\n\", var1);");
+  EXPECT_FALSE(!!R2);
   using ::testing::HasSubstr;
   EXPECT_THAT(DiagnosticsOS.str(),
               HasSubstr("error: unknown type name 'var1'"));
-  auto Err = R2OrErr.takeError();
+  auto Err = R2.takeError();
   EXPECT_EQ("Parsing failed.", llvm::toString(std::move(Err)));
 }
 
 } // end anonymous namespace

clang/unittests/Lex/PPCallbacksTest.cpp

Preprocessor PP(std::make_shared<PreprocessorOptions>(), Diags,
                     /*IILookup =*/nullptr,
                     /*OwnsHeaderSearch =*/false);
     PP.Initialize(*Target);
 
     // parser actually sets correct pragma handlers for preprocessor
     // according to LangOptions, so we init Parser to register opencl
     // pragma handlers
     ASTContext Context(OpenCLLangOpts, SourceMgr, PP.getIdentifierTable(),
-                       PP.getSelectorTable(), PP.getBuiltinInfo());
+                       PP.getSelectorTable(), PP.getBuiltinInfo(), PP.TUKind);
     Context.InitBuiltinTypes(*Target);
 
     ASTConsumer Consumer;
     Sema S(PP, Context, Consumer);
     Parser P(PP, S, false);
     PragmaOpenCLExtensionCallbacks* Callbacks = new PragmaOpenCLExtensionCallbacks;
     PP.addPPCallbacks(std::unique_ptr<PPCallbacks>(Callbacks));