Module: use PCMCache to manage memory buffers for pcm files.

[dexonsmith]

Summary

We create a PCMCache class to manage memory buffers associated with pcm files. With implicit modules, we currently use lock files to make
sure we are making progress when a single clang invocation builds a module file and then loads the module file immediately after. This
implementation requires lock file for correctness and causes all kinds of locking issues.

This patch tries to use PCMCache to share the content between writes and reads within a single clang invocation, so we can remove the
correctness-side of the lock file, and only use lock file for performance i.e to reduce chance of building the same pcm file from multiple
clang invocations.

We also add ThreadContext in PCMCache to diagnose cases where an ancestor validates a module and later on, a child thread tries
to invalidate it. Without the patch, this scenario will cause use-after-free of the FileEntry held by the ancestor.

[aprantl]

include/clang/Basic/DiagnosticSerializationKinds.td

131 ↗

(On Diff #83069)

Is this better?

"module file '%0' was validated as a system module and is now being imported as a non-system module; any differences in diagnostics options will be ignored"

135 ↗

(On Diff #83069)

I wonder if there is a way to rephrase this message such that an end-user could understan how to interpret this error?

[benlangmuir]

Can we test the already-validated diagnostics?

include/clang/Basic/FileManager.h

176 ↗

(On Diff #83069)

Why is this inside the FileManager? It isn't used by the FileManager.

308 ↗

(On Diff #83069)

Can we call this something like "ModuleLoadContext" or maybe "ModuleCompilationContext"? The word thread is misleading, since the threads are not run concurrently, and are only an implementation detail of our crash recovery.

Also, can you explain why it is only necessary to keep information about the the current stack of contexts? In a situation like

A imports B imports C
A imports D imports C

We would no longer have information about C being validated, right? What happens if there's a mismatch there? Can this never happen?

[manmanren]

I forgot to upload the testing case, I will add it now.

Manman

include/clang/Basic/FileManager.h

176 ↗

(On Diff #83069)

I initially had another patch that puts PCMCache object at the top level (i.e whenever we create a FileManager, we create a PCMCache object). The initial patch is much bigger than this patch. PCMCache is cacheing the content of a PCM File, and is shared across threads that we spawned to build modules implicitly, just like FileManager. I had some discussions with Bruno and Adrian, we felt FileManager is a better place. But if you have other suggestions, please let me know!

308 ↗

(On Diff #83069)

Sure, we can call it ModuleCompilationContext because we are spawning threads to build modules implicitly.

The issue we are trying to detect is use-after-free, i.e we don't want a thread to hold on to an invalid FileEntry. We error out before trying to delete the FileEntry that is live in another thread.

In a situation like
A imports B imports C
A imports D imports C

A will start a thread to build B, and B will start a thread to build C, C will be validated in B's compilation thread and A's compilation thread. Later on, if D tries to invalidate C, D knows that an ancestor thread A has validated C, so the compiler will error out. And it is okay for A to invalidate C in A's compilation thread, because we can clean up the references to C's FileEntry.

[manmanren]

Add testing case.

[manmanren]

include/clang/Basic/DiagnosticSerializationKinds.td

131 ↗

(On Diff #83069)

Thanks Adrian. Your wording sounds better.

135 ↗

(On Diff #83069)

I am not sure if we will emit this error diagnostic in some case, that is why I don't have a testing case. Maybe it is better to throw a hard error instead of a diagnostic that user can't understand?

[benlangmuir]

include/clang/Basic/FileManager.h

176 ↗

(On Diff #83069)

I see. It's not ideal, but I see why it would be convenient to use the FileManager this way, since you want to share it in all the same places the FileManager is shared. Since I don't have any better answer, I guess I'm okay with this...

308 ↗

(On Diff #83069)

Okay

[manmanren]

Addressing review comments.

[manmanren]

Ping :]
Hoping to wrap this up this week.
Cheers,
Manman

[benlangmuir]

Sorry for the delay, I though you were still iterating with @aprantl for some reason. LGTM

[bruno]

lib/Serialization/ASTReader.cpp

3692 ↗

(On Diff #84137)

This should honor ARR_OutOfDate, so that the module can be rebuilt in case a user module mismatch via a different diagnostics (lib/Serialization/ASTReader.cpp:4076) and returns OutOfDate instead of Success:

if ((ClientLoadCapabilities & ARR_OutOfDate) == 0)
  for (auto N : ValidationConflicts)
    Diag(diag::err_module_ancestor_conflict) << N;

[dexonsmith]

Taking this over to rebase it.

[dexonsmith]

Rebased on top of the current https://reviews.llvm.org/D27689. This also has some substantive changes:

• Renamed the data structure PCMCache to MemoryBufferCache, and split it out into its own files with tests.

• Simplified logic around whether a MemoryBuffer has already been used. Tracking of what has already been validated now takes O(1) memory instead of O(n^2).

• Moved FileManager::BufferMgr (the instance of PCMCache) to CompilerInstance::PCMCache and ModuleManager::PCMCache (instances of IntrusiveRefCntPtr<MemoryBufferCache>). This took some work to thread through, but since the PCMCache isn't related to the FileManager, this seemed like a cleaner result.

• Added some testcases for bugs we've found in the meantime (thanks to Adrian Prantl). Primarily, we no longer error or cause a use-after-free if a MemoryBuffer was validated by an ancestor.

[dexonsmith]

This has changed enough it needs another review. Richard or Ben, could you take a(nother) look?

[dexonsmith]

Rebased again, and cleaned up tests, and a ping... now that dependencies are resolved.

Richard and/or Ben, can you take a look?

[dexonsmith]

! In D28299#701194, @dexonsmith wrote:
Rebased again, and cleaned up tests, and a ping... now that dependencies are resolved.

Richard and/or Ben, can you take a look?

(Or anyone else that feels comfortable confirming that the ownership semantics of MemoryBufferCache are correct.)

[aprantl]

Drive-by comment.

clang/lib/Basic/MemoryBufferCache.cpp

18

we should be able to replace the std::make_pair() with {}.

[dexonsmith]

Thanks for the reviews! Committed in r298165.

clang/include/clang/Basic/DiagnosticSerializationKinds.td

 def warn_module_uses_date_time : Warning<
   "%select{precompiled header|module}0 uses __DATE__ or __TIME__">,
   InGroup<DiagGroup<"pch-date-time">>;
 
 def warn_duplicate_module_file_extension : Warning<
   "duplicate module file extension block name '%0'">,
   InGroup<ModuleFileExtension>;
 
+def warn_module_system_bit_conflict : Warning<
+  "module file '%0' was validated as a system module and is now being imported "
+  "as a non-system module; any difference in diagnostic options will be ignored">,
+  InGroup<ModuleConflict>;
+
 } // let CategoryName
 } // let Component
 

clang/include/clang/Basic/MemoryBufferCache.h

+//===- MemoryBufferCache.h - Cache for loaded memory buffers ----*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_BASIC_MEMORYBUFFERCACHE_H
+#define LLVM_CLANG_BASIC_MEMORYBUFFERCACHE_H
+
+#include "llvm/ADT/IntrusiveRefCntPtr.h"
+#include "llvm/ADT/StringMap.h"
+#include <memory>
+
+namespace llvm {
+class MemoryBuffer;
+} // end namespace llvm
+
+namespace clang {
+
+/// Manage memory buffers across multiple users.
+///
+/// Ensures that multiple users have a consistent view of each buffer.  This is
+/// used by \a CompilerInstance when building PCMs to ensure that each \a
+/// ModuleManager sees the same files.
+///
+/// \a finalizeCurrentBuffers() should be called before creating a new user.
+/// This locks in the current buffers, ensuring that no buffer that has already
+/// been accessed can be purged, preventing use-after-frees.
+class MemoryBufferCache : public llvm::RefCountedBase<MemoryBufferCache> {
+  struct BufferEntry {
+    std::unique_ptr<llvm::MemoryBuffer> Buffer;
+
+    /// Track the timeline of when this was added to the cache.
+    unsigned Index;
+  };
+
+  /// Cache of buffers.
+  llvm::StringMap<BufferEntry> Buffers;
+
+  /// Monotonically increasing index.
+  unsigned NextIndex = 0;
+
+  /// Bumped to prevent "older" buffers from being removed.
+  unsigned FirstRemovableIndex = 0;
+
+public:
+  /// Store the Buffer under the Filename.
+  ///
+  /// \pre There is not already buffer is not already in the cache.
+  /// \return a reference to the buffer as a convenience.
+  llvm::MemoryBuffer &addBuffer(llvm::StringRef Filename,
+                                std::unique_ptr<llvm::MemoryBuffer> Buffer);
+
+  /// Try to remove a buffer from the cache.
+  ///
+  /// \return false on success, iff \c !isBufferFinal().
+  bool tryToRemoveBuffer(llvm::StringRef Filename);
+
+  /// Get a pointer to the buffer if it exists; else nullptr.
+  llvm::MemoryBuffer *lookupBuffer(llvm::StringRef Filename);
+
+  /// Check whether the buffer is final.
+  ///
+  /// \return true iff \a finalizeCurrentBuffers() has been called since the
+  /// buffer was added.  This prevents buffers from being removed.
+  bool isBufferFinal(llvm::StringRef Filename);
+
+  /// Finalize the current buffers in the cache.
+  ///
+  /// Should be called when creating a new user to ensure previous uses aren't
+  /// invalidated.
+  void finalizeCurrentBuffers();
+};
+
+} // end namespace clang
+
+#endif // LLVM_CLANG_BASIC_MEMORYBUFFERCACHE_H

clang/include/clang/Frontend/ASTUnit.h

 class ASTReader;
 class CompilerInvocation;
 class CompilerInstance;
 class Decl;
 class DiagnosticsEngine;
 class FileEntry;
 class FileManager;
 class HeaderSearch;
+class MemoryBufferCache;
 class Preprocessor;
 class PCHContainerOperations;
 class PCHContainerReader;
 class TargetInfo;
 class FrontendAction;
 class ASTDeserializationListener;
 
 /// \brief Utility class for loading a ASTContext from an AST file.
struct StandaloneDiagnostic {
     std::vector<StandaloneFixIt> FixIts;
   };
 
 private:
   std::shared_ptr<LangOptions>            LangOpts;
   IntrusiveRefCntPtr<DiagnosticsEngine>   Diagnostics;
   IntrusiveRefCntPtr<FileManager>         FileMgr;
   IntrusiveRefCntPtr<SourceManager>       SourceMgr;
+  IntrusiveRefCntPtr<MemoryBufferCache>   PCMCache;
   std::unique_ptr<HeaderSearch>           HeaderInfo;
   IntrusiveRefCntPtr<TargetInfo>          Target;
   std::shared_ptr<Preprocessor>           PP;
   IntrusiveRefCntPtr<ASTContext>          Ctx;
   std::shared_ptr<TargetOptions>          TargetOpts;
   std::shared_ptr<HeaderSearchOptions>    HSOpts;
   IntrusiveRefCntPtr<ASTReader> Reader;
   bool HadModuleLoaderFatalFailure;

clang/include/clang/Frontend/CompilerInstance.h

 class ASTReader;
 class CodeCompleteConsumer;
 class DiagnosticsEngine;
 class DiagnosticConsumer;
 class ExternalASTSource;
 class FileEntry;
 class FileManager;
 class FrontendAction;
+class MemoryBufferCache;
 class Module;
 class Preprocessor;
 class Sema;
 class SourceManager;
 class TargetInfo;
 
 /// CompilerInstance - Helper class for managing a single instance of the Clang
 /// compiler.
class CompilerInstance : public ModuleLoader {
   IntrusiveRefCntPtr<vfs::FileSystem> VirtualFileSystem;
 
   /// The file manager.
   IntrusiveRefCntPtr<FileManager> FileMgr;
 
   /// The source manager.
   IntrusiveRefCntPtr<SourceManager> SourceMgr;
 
+  /// The cache of PCM files.
+  IntrusiveRefCntPtr<MemoryBufferCache> PCMCache;
+
   /// The preprocessor.
   std::shared_ptr<Preprocessor> PP;
 
   /// The AST context.
   IntrusiveRefCntPtr<ASTContext> Context;
 
   /// An optional sema source that will be attached to sema.
   IntrusiveRefCntPtr<ExternalSemaSource> ExternalSemaSrc;
class CompilerInstance : public ModuleLoader {
   std::list<OutputFile> OutputFiles;
 
   CompilerInstance(const CompilerInstance &) = delete;
   void operator=(const CompilerInstance &) = delete;
 public:
   explicit CompilerInstance(
       std::shared_ptr<PCHContainerOperations> PCHContainerOps =
           std::make_shared<PCHContainerOperations>(),
-      bool BuildingModule = false);
+      MemoryBufferCache *SharedPCMCache = nullptr);
   ~CompilerInstance() override;
 
   /// @name High-Level Operations
   /// {
 
   /// ExecuteAction - Execute the provided action against the compiler's
   /// CompilerInvocation object.
   ///
public:
 
   bool lookupMissingImports(StringRef Name, SourceLocation TriggerLoc) override;
 
   void addDependencyCollector(std::shared_ptr<DependencyCollector> Listener) {
     DependencyCollectors.push_back(std::move(Listener));
   }
 
   void setExternalSemaSource(IntrusiveRefCntPtr<ExternalSemaSource> ESS);
+
+  MemoryBufferCache &getPCMCache() const { return *PCMCache; }
 };
 
 } // end namespace clang
 
 #endif

clang/include/clang/Lex/Preprocessor.h

 
 namespace clang {
 
 class SourceManager;
 class ExternalPreprocessorSource;
 class FileManager;
 class FileEntry;
 class HeaderSearch;
+class MemoryBufferCache;
 class PragmaNamespace;
 class PragmaHandler;
 class CommentHandler;
 class ScratchBuffer;
 class TargetInfo;
 class PPCallbacks;
 class CodeCompletionHandler;
 class DirectoryLookup;

 class Preprocessor {
   std::shared_ptr<PreprocessorOptions> PPOpts;
   DiagnosticsEngine        *Diags;
   LangOptions       &LangOpts;
   const TargetInfo  *Target;
   const TargetInfo  *AuxTarget;
   FileManager       &FileMgr;
   SourceManager     &SourceMgr;
+  MemoryBufferCache &PCMCache;
   std::unique_ptr<ScratchBuffer> ScratchBuf;
   HeaderSearch      &HeaderInfo;
   ModuleLoader      &TheModuleLoader;
 
   /// \brief External source of macros.
   ExternalPreprocessorSource *ExternalSource;
 
 
class Preprocessor {
   };
   DeserializedMacroInfoChain *DeserialMIChainHead;
 
   void updateOutOfDateIdentifier(IdentifierInfo &II) const;
 
 public:
   Preprocessor(std::shared_ptr<PreprocessorOptions> PPOpts,
                DiagnosticsEngine &diags, LangOptions &opts, SourceManager &SM,
+               MemoryBufferCache &PCMCache,
                HeaderSearch &Headers, ModuleLoader &TheModuleLoader,
                IdentifierInfoLookup *IILookup = nullptr,
                bool OwnsHeaderSearch = false,
                TranslationUnitKind TUKind = TU_Complete);
 
   ~Preprocessor();
 
   /// \brief Initialize the preprocessor using information about the target.
public:
   DiagnosticsEngine &getDiagnostics() const { return *Diags; }
   void setDiagnostics(DiagnosticsEngine &D) { Diags = &D; }
 
   const LangOptions &getLangOpts() const { return LangOpts; }
   const TargetInfo &getTargetInfo() const { return *Target; }
   const TargetInfo *getAuxTargetInfo() const { return AuxTarget; }
   FileManager &getFileManager() const { return FileMgr; }
   SourceManager &getSourceManager() const { return SourceMgr; }
+  MemoryBufferCache &getPCMCache() const { return PCMCache; }
   HeaderSearch &getHeaderSearchInfo() const { return HeaderInfo; }
 
   IdentifierTable &getIdentifierTable() { return Identifiers; }
   const IdentifierTable &getIdentifierTable() const { return Identifiers; }
   SelectorTable &getSelectorTable() { return Selectors; }
   Builtin::Context &getBuiltinInfo() { return BuiltinInfo; }
   llvm::BumpPtrAllocator &getPreprocessorAllocator() { return BP; }
 

clang/include/clang/Serialization/ASTReader.h

private:
   ASTContext &Context;
 
   /// \brief The AST consumer.
   ASTConsumer *Consumer = nullptr;
 
   /// \brief The module manager which manages modules and their dependencies
   ModuleManager ModuleMgr;
 
+  /// The cache that manages memory buffers for PCM files.
+  MemoryBufferCache &PCMCache;
+
   /// \brief A dummy identifier resolver used to merge TU-scope declarations in
   /// C, for the cases where we don't have a Sema object to provide a real
   /// identifier resolver.
   IdentifierResolver DummyIdResolver;
 
   /// A mapping from extension block names to module file extensions.
   llvm::StringMap<std::shared_ptr<ModuleFileExtension>> ModuleFileExtensions;
 

clang/include/clang/Serialization/ASTWriter.h

 class HeaderSearchOptions;
 class IdentifierResolver;
 class MacroDefinitionRecord;
 class MacroDirective;
 class MacroInfo;
 class OpaqueValueExpr;
 class OpenCLOptions;
 class ASTReader;
+class MemoryBufferCache;
 class Module;
 class ModuleFileExtension;
 class ModuleFileExtensionWriter;
 class PreprocessedEntity;
 class PreprocessingRecord;
 class Preprocessor;
 class RecordDecl;
 class Sema;
typedef llvm::DenseMap<QualType, serialization::TypeIdx,
     TypeIdxMap;
 
   /// \brief The bitstream writer used to emit this precompiled header.
   llvm::BitstreamWriter &Stream;
 
   /// The buffer associated with the bitstream.
   const SmallVectorImpl<char> &Buffer;
 
+  /// \brief The PCM manager which manages memory buffers for pcm files.
+  MemoryBufferCache &PCMCache;
+
   /// \brief The ASTContext we're writing.
   ASTContext *Context = nullptr;
 
   /// \brief The preprocessor we're writing.
   Preprocessor *PP = nullptr;
 
   /// \brief The reader of existing AST files, if we're chaining.
   ASTReader *Chain = nullptr;
private:
   ASTFileSignature WriteASTCore(Sema &SemaRef, StringRef isysroot,
                                 const std::string &OutputFile,
                                 Module *WritingModule);
 
 public:
   /// \brief Create a new precompiled header writer that outputs to
   /// the given bitstream.
   ASTWriter(llvm::BitstreamWriter &Stream, SmallVectorImpl<char> &Buffer,
+            MemoryBufferCache &PCMCache,
             ArrayRef<std::shared_ptr<ModuleFileExtension>> Extensions,
             bool IncludeTimestamps = true);
   ~ASTWriter() override;
 
   const LangOptions &getLangOpts() const;
 
   /// \brief Get a timestamp for output into the AST file. The actual timestamp
   /// of the specified file may be ignored if we have been instructed to not

clang/include/clang/Serialization/Module.h

public:
 
   /// \brief Whether this module has been directly imported by the
   /// user.
   bool DirectlyImported = false;
 
   /// \brief The generation of which this module file is a part.
   unsigned Generation;
   
-  /// \brief The memory buffer that stores the data associated with
-  /// this AST file.
-  std::unique_ptr<llvm::MemoryBuffer> Buffer;
+  /// The memory buffer that stores the data associated with
+  /// this AST file, owned by the PCMCache in the ModuleManager.
+  llvm::MemoryBuffer *Buffer;
 
   /// \brief The size of this file, in bits.
   uint64_t SizeInBits = 0;
 
   /// \brief The global bit offset (or base) of this module
   uint64_t GlobalBitOffset = 0;
 
   /// \brief The serialized bitstream data for this file.

clang/include/clang/Serialization/ModuleManager.h

 #include "clang/Serialization/Module.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/ADT/iterator.h"
 
 namespace clang { 
 
 class GlobalModuleIndex;
+class MemoryBufferCache;
 class ModuleMap;
 class PCHContainerReader;
 
 namespace serialization {
 
 /// \brief Manages the set of modules loaded by an AST reader.
 class ModuleManager {
   /// \brief The chain of AST files, in the order in which we started to load
class ModuleManager {
 
   /// \brief All loaded modules, indexed by name.
   llvm::DenseMap<const FileEntry *, ModuleFile *> Modules;
 
   /// \brief FileManager that handles translating between filenames and
   /// FileEntry *.
   FileManager &FileMgr;
 
+  /// Cache of PCM files.
+  IntrusiveRefCntPtr<MemoryBufferCache> PCMCache;
+
   /// \brief Knows how to unwrap module containers.
   const PCHContainerReader &PCHContainerRdr;
 
   /// \brief A lookup of in-memory (virtual file) buffers
   llvm::DenseMap<const FileEntry *, std::unique_ptr<llvm::MemoryBuffer>>
       InMemoryBuffers;
 
   /// \brief The visitation order.
public:
   typedef llvm::pointee_iterator<
       SmallVectorImpl<std::unique_ptr<ModuleFile>>::const_iterator>
       ModuleConstIterator;
   typedef llvm::pointee_iterator<
       SmallVectorImpl<std::unique_ptr<ModuleFile>>::reverse_iterator>
       ModuleReverseIterator;
   typedef std::pair<uint32_t, StringRef> ModuleOffset;
 
-  explicit ModuleManager(FileManager &FileMgr,
+  explicit ModuleManager(FileManager &FileMgr, MemoryBufferCache &PCMCache,
                          const PCHContainerReader &PCHContainerRdr);
   ~ModuleManager();
 
   /// \brief Forward iterator to traverse all loaded modules.
   ModuleIterator begin() { return Chain.begin(); }
   /// \brief Forward iterator end-point to traverse all loaded modules
   ModuleIterator end() { return Chain.end(); }
   
public:
   /// suitable, or is missing.
   bool lookupModuleFile(StringRef FileName,
                         off_t ExpectedSize,
                         time_t ExpectedModTime,
                         const FileEntry *&File);
 
   /// \brief View the graphviz representation of the module graph.
   void viewGraph();
+
+  MemoryBufferCache &getPCMCache() const { return *PCMCache; }
 };
 
 } } // end namespace clang::serialization
 
 #endif

clang/lib/Basic/CMakeLists.txt

add_clang_library(clangBasic
   Cuda.cpp
   Diagnostic.cpp
   DiagnosticIDs.cpp
   DiagnosticOptions.cpp
   FileManager.cpp
   FileSystemStatCache.cpp
   IdentifierTable.cpp
   LangOptions.cpp
+  MemoryBufferCache.cpp
   Module.cpp
   ObjCRuntime.cpp
   OpenMPKinds.cpp
   OperatorPrecedence.cpp
   SanitizerBlacklist.cpp
   Sanitizers.cpp
   SourceLocation.cpp
   SourceManager.cpp

clang/lib/Basic/MemoryBufferCache.cpp

+//===- MemoryBufferCache.cpp - Cache for loaded memory buffers ------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Basic/MemoryBufferCache.h"
+#include "llvm/Support/MemoryBuffer.h"
+
+using namespace clang;
+
+llvm::MemoryBuffer &
+MemoryBufferCache::addBuffer(llvm::StringRef Filename,
+                             std::unique_ptr<llvm::MemoryBuffer> Buffer) {
+  auto Insertion = Buffers.insert(std::make_pair(
+      Filename, BufferEntry{std::move(Buffer), NextIndex++}));
+  assert(Insertion.second && "Already has a buffer");
+  return *Insertion.first->second.Buffer;
+}
+
+llvm::MemoryBuffer *MemoryBufferCache::lookupBuffer(llvm::StringRef Filename) {
+  auto I = Buffers.find(Filename);
+  if (I == Buffers.end())
+    return nullptr;
+  return I->second.Buffer.get();
+}
+
+bool MemoryBufferCache::isBufferFinal(llvm::StringRef Filename) {
+  auto I = Buffers.find(Filename);
+  if (I == Buffers.end())
+    return false;
+  return I->second.Index < FirstRemovableIndex;
+}
+
+bool MemoryBufferCache::tryToRemoveBuffer(llvm::StringRef Filename) {
+  auto I = Buffers.find(Filename);
+  assert(I != Buffers.end() && "No buffer to remove...");
+  if (I->second.Index < FirstRemovableIndex)
+    return true;
+
+  Buffers.erase(I);
+  return false;
+}
+
+void MemoryBufferCache::finalizeCurrentBuffers() { FirstRemovableIndex = NextIndex; }

clang/lib/Frontend/ASTUnit.cpp

 
 #include "clang/Frontend/ASTUnit.h"
 #include "clang/AST/ASTConsumer.h"
 #include "clang/AST/ASTContext.h"
 #include "clang/AST/DeclVisitor.h"
 #include "clang/AST/StmtVisitor.h"
 #include "clang/AST/TypeOrdering.h"
 #include "clang/Basic/Diagnostic.h"
+#include "clang/Basic/MemoryBufferCache.h"
 #include "clang/Basic/TargetInfo.h"
 #include "clang/Basic/TargetOptions.h"
 #include "clang/Basic/VirtualFileSystem.h"
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/FrontendActions.h"
 #include "clang/Frontend/FrontendDiagnostic.h"
 #include "clang/Frontend/FrontendOptions.h"
 #include "clang/Frontend/MultiplexConsumer.h"
void OnDiskData::Cleanup() {
   CleanPreambleFile();
 }
 
 struct ASTUnit::ASTWriterData {
   SmallString<128> Buffer;
   llvm::BitstreamWriter Stream;
   ASTWriter Writer;
 
-  ASTWriterData() : Stream(Buffer), Writer(Stream, Buffer, {}) {}
+  ASTWriterData(MemoryBufferCache &PCMCache)
+      : Stream(Buffer), Writer(Stream, Buffer, PCMCache, {}) {}
 };
 
 void ASTUnit::clearFileLevelDecls() {
   llvm::DeleteContainerSeconds(FileDecls);
 }
 
 void ASTUnit::CleanTemporaryFiles() {
   getOnDiskData(this).CleanTemporaryFiles();
std::unique_ptr<ASTUnit> ASTUnit::LoadFromASTFile(
   AST->CaptureDiagnostics = CaptureDiagnostics;
   AST->Diagnostics = Diags;
   IntrusiveRefCntPtr<vfs::FileSystem> VFS = vfs::getRealFileSystem();
   AST->FileMgr = new FileManager(FileSystemOpts, VFS);
   AST->UserFilesAreVolatile = UserFilesAreVolatile;
   AST->SourceMgr = new SourceManager(AST->getDiagnostics(),
                                      AST->getFileManager(),
                                      UserFilesAreVolatile);
+  AST->PCMCache = new MemoryBufferCache;
   AST->HSOpts = std::make_shared<HeaderSearchOptions>();
   AST->HSOpts->ModuleFormat = PCHContainerRdr.getFormat();
   AST->HeaderInfo.reset(new HeaderSearch(AST->HSOpts,
                                          AST->getSourceManager(),
                                          AST->getDiagnostics(),
                                          AST->ASTFileLangOpts,
                                          /*Target=*/nullptr));
 
   auto PPOpts = std::make_shared<PreprocessorOptions>();
 
   for (const auto &RemappedFile : RemappedFiles)
     PPOpts->addRemappedFile(RemappedFile.first, RemappedFile.second);
 
   // Gather Info for preprocessor construction later on.
 
   HeaderSearch &HeaderInfo = *AST->HeaderInfo;
   unsigned Counter;
 
   AST->PP = std::make_shared<Preprocessor>(
       std::move(PPOpts), AST->getDiagnostics(), AST->ASTFileLangOpts,
-      AST->getSourceManager(), HeaderInfo, *AST,
+      AST->getSourceManager(), *AST->PCMCache, HeaderInfo, *AST,
       /*IILookup=*/nullptr,
       /*OwnsHeaderSearch=*/false);
   Preprocessor &PP = *AST->PP;
 
   AST->Ctx = new ASTContext(AST->ASTFileLangOpts, AST->getSourceManager(),
                             PP.getIdentifierTable(), PP.getSelectorTable(),
                             PP.getBuiltinInfo());
   ASTContext &Context = *AST->Ctx;
if (!VFS)
     return nullptr;
   AST->Diagnostics = Diags;
   AST->FileSystemOpts = CI->getFileSystemOpts();
   AST->Invocation = std::move(CI);
   AST->FileMgr = new FileManager(AST->FileSystemOpts, VFS);
   AST->UserFilesAreVolatile = UserFilesAreVolatile;
   AST->SourceMgr = new SourceManager(AST->getDiagnostics(), *AST->FileMgr,
                                      UserFilesAreVolatile);
+  AST->PCMCache = new MemoryBufferCache;
 
   return AST;
 }
 
 ASTUnit *ASTUnit::LoadFromCompilerInvocationAction(
     std::shared_ptr<CompilerInvocation> CI,
     std::shared_ptr<PCHContainerOperations> PCHContainerOps,
     IntrusiveRefCntPtr<DiagnosticsEngine> Diags, FrontendAction *Action,
ASTUnit *ASTUnit::LoadFromCommandLine(
   ConfigureDiags(Diags, *AST, CaptureDiagnostics);
   AST->Diagnostics = Diags;
   AST->FileSystemOpts = CI->getFileSystemOpts();
   IntrusiveRefCntPtr<vfs::FileSystem> VFS =
       createVFSFromCompilerInvocation(*CI, *Diags);
   if (!VFS)
     return nullptr;
   AST->FileMgr = new FileManager(AST->FileSystemOpts, VFS);
+  AST->PCMCache = new MemoryBufferCache;
   AST->OnlyLocalDecls = OnlyLocalDecls;
   AST->CaptureDiagnostics = CaptureDiagnostics;
   AST->TUKind = TUKind;
   AST->ShouldCacheCodeCompletionResults = CacheCodeCompletionResults;
   AST->IncludeBriefCommentsInCodeCompletion
     = IncludeBriefCommentsInCodeCompletion;
   AST->UserFilesAreVolatile = UserFilesAreVolatile;
   AST->NumStoredDiagnosticsFromDriver = StoredDiagnostics.size();
   AST->StoredDiagnostics.swap(StoredDiagnostics);
   AST->Invocation = CI;
   if (ForSerialization)
-    AST->WriterData.reset(new ASTWriterData());
+    AST->WriterData.reset(new ASTWriterData(*AST->PCMCache));
   // Zero out now to ease cleanup during crash recovery.
   CI = nullptr;
   Diags = nullptr;
 
   // Recover resources if we crash before exiting this method.
   llvm::CrashRecoveryContextCleanupRegistrar<ASTUnit>
     ASTUnitCleanup(AST.get());
 
bool ASTUnit::serialize(raw_ostream &OS) {
   bool hasErrors = getDiagnostics().hasUncompilableErrorOccurred();
 
   if (WriterData)
     return serializeUnit(WriterData->Writer, WriterData->Buffer,
                          getSema(), hasErrors, OS);
 
   SmallString<128> Buffer;
   llvm::BitstreamWriter Stream(Buffer);
-  ASTWriter Writer(Stream, Buffer, {});
+  MemoryBufferCache PCMCache;
+  ASTWriter Writer(Stream, Buffer, PCMCache, {});
   return serializeUnit(Writer, Buffer, getSema(), hasErrors, OS);
 }
 
 typedef ContinuousRangeMap<unsigned, int, 2> SLocRemap;
 
 void ASTUnit::TranslateStoredDiagnostics(
                           FileManager &FileMgr,
                           SourceManager &SrcMgr,

clang/lib/Frontend/CompilerInstance.cpp

 //===--- CompilerInstance.cpp ---------------------------------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/AST/ASTConsumer.h"
 #include "clang/AST/ASTContext.h"
 #include "clang/AST/Decl.h"
 #include "clang/Basic/Diagnostic.h"
 #include "clang/Basic/FileManager.h"
+#include "clang/Basic/MemoryBufferCache.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Basic/TargetInfo.h"
 #include "clang/Basic/Version.h"
 #include "clang/Config/config.h"
 #include "clang/Frontend/ChainedDiagnosticConsumer.h"
 #include "clang/Frontend/FrontendAction.h"
 #include "clang/Frontend/FrontendActions.h"
 #include "clang/Frontend/FrontendDiagnostic.h"

 #include <system_error>
 #include <time.h>
 #include <utility>
 
 using namespace clang;
 
 CompilerInstance::CompilerInstance(
     std::shared_ptr<PCHContainerOperations> PCHContainerOps,
-    bool BuildingModule)
-    : ModuleLoader(BuildingModule), Invocation(new CompilerInvocation()),
-      ThePCHContainerOperations(std::move(PCHContainerOps)) {}
+    MemoryBufferCache *SharedPCMCache)
+    : ModuleLoader(/* BuildingModule = */ SharedPCMCache),
+      Invocation(new CompilerInvocation()),
+      PCMCache(SharedPCMCache ? SharedPCMCache : new MemoryBufferCache),
+      ThePCHContainerOperations(std::move(PCHContainerOps)) {
+  // Don't allow this to invalidate buffers in use by others.
+  if (SharedPCMCache)
+    getPCMCache().finalizeCurrentBuffers();
+}
 
 CompilerInstance::~CompilerInstance() {
   assert(OutputFiles.empty() && "Still output files in flight?");
 }
 
 void CompilerInstance::setInvocation(
     std::shared_ptr<CompilerInvocation> Value) {
   Invocation = std::move(Value);

 std::unique_ptr<Sema> CompilerInstance::takeSema() {
   return std::move(TheSema);
 }
 
 IntrusiveRefCntPtr<ASTReader> CompilerInstance::getModuleManager() const {
   return ModuleManager;
 }
 void CompilerInstance::setModuleManager(IntrusiveRefCntPtr<ASTReader> Reader) {
+  assert(PCMCache.get() == &Reader->getModuleManager().getPCMCache() &&
+         "Expected ASTReader to use the same PCM cache");
   ModuleManager = std::move(Reader);
 }
 
 std::shared_ptr<ModuleDependencyCollector>
 CompilerInstance::getModuleDepCollector() const {
   return ModuleDepCollector;
 }
 
if (!PPOpts.TokenCache.empty())
     PTHMgr = PTHManager::Create(PPOpts.TokenCache, getDiagnostics());
 
   // Create the Preprocessor.
   HeaderSearch *HeaderInfo =
       new HeaderSearch(getHeaderSearchOptsPtr(), getSourceManager(),
                        getDiagnostics(), getLangOpts(), &getTarget());
   PP = std::make_shared<Preprocessor>(
       Invocation->getPreprocessorOptsPtr(), getDiagnostics(), getLangOpts(),
-      getSourceManager(), *HeaderInfo, *this, PTHMgr,
+      getSourceManager(), getPCMCache(), *HeaderInfo, *this, PTHMgr,
       /*OwnsHeaderSearch=*/true, TUKind);
   PP->Initialize(getTarget(), getAuxTarget());
 
   // Note that this is different then passing PTHMgr to Preprocessor's ctor.
   // That argument is used as the IdentifierInfoLookup argument to
   // IdentifierTable's ctor.
   if (PTHMgr) {
     PTHMgr->setPreprocessor(&*PP);
static bool compileModuleImpl(CompilerInstance &ImportingInstance,
   // Don't free the remapped file buffers; they are owned by our caller.
   PPOpts.RetainRemappedFileBuffers = true;
     
   Invocation->getDiagnosticOpts().VerifyDiagnostics = 0;
   assert(ImportingInstance.getInvocation().getModuleHash() ==
          Invocation->getModuleHash() && "Module hash mismatch!");
   
   // Construct a compiler instance that will be used to actually create the
-  // module.
+  // module.  Since we're sharing a PCMCache,
+  // CompilerInstance::CompilerInstance is responsible for finalizing the
+  // buffers to prevent use-after-frees.
   CompilerInstance Instance(ImportingInstance.getPCHContainerOperations(),
-                            /*BuildingModule=*/true);
+                            &ImportingInstance.getPreprocessor().getPCMCache());
   auto &Inv = *Invocation;
   Instance.setInvocation(std::move(Invocation));
 
   Instance.createDiagnostics(new ForwardingDiagnosticConsumer(
                                    ImportingInstance.getDiagnosticClient()),
                              /*ShouldOwnClient=*/true);
 
   Instance.setVirtualFileSystem(&ImportingInstance.getVirtualFileSystem());

clang/lib/Lex/Preprocessor.cpp

 
 LLVM_INSTANTIATE_REGISTRY(PragmaHandlerRegistry)
 
 //===----------------------------------------------------------------------===//
 ExternalPreprocessorSource::~ExternalPreprocessorSource() { }
 
 Preprocessor::Preprocessor(std::shared_ptr<PreprocessorOptions> PPOpts,
                            DiagnosticsEngine &diags, LangOptions &opts,
-                           SourceManager &SM, HeaderSearch &Headers,
-                           ModuleLoader &TheModuleLoader,
+                           SourceManager &SM, MemoryBufferCache &PCMCache,
+                           HeaderSearch &Headers, ModuleLoader &TheModuleLoader,
                            IdentifierInfoLookup *IILookup, bool OwnsHeaders,
                            TranslationUnitKind TUKind)
     : PPOpts(std::move(PPOpts)), Diags(&diags), LangOpts(opts), Target(nullptr),
       AuxTarget(nullptr), FileMgr(Headers.getFileMgr()), SourceMgr(SM),
-      ScratchBuf(new ScratchBuffer(SourceMgr)), HeaderInfo(Headers),
-      TheModuleLoader(TheModuleLoader), ExternalSource(nullptr),
-      Identifiers(opts, IILookup),
+      PCMCache(PCMCache), ScratchBuf(new ScratchBuffer(SourceMgr)),
+      HeaderInfo(Headers), TheModuleLoader(TheModuleLoader),
+      ExternalSource(nullptr), Identifiers(opts, IILookup),
       PragmaHandlers(new PragmaNamespace(StringRef())),
       IncrementalProcessing(false), TUKind(TUKind), CodeComplete(nullptr),
       CodeCompletionFile(nullptr), CodeCompletionOffset(0),
       LastTokenWasAt(false), ModuleImportExpectsIdentifier(false),
       CodeCompletionReached(false), CodeCompletionII(nullptr),
       MainFileDir(nullptr), SkipMainFilePreamble(0, true), CurPPLexer(nullptr),
       CurDirLookup(nullptr), CurLexerKind(CLK_Lexer), CurSubmodule(nullptr),
       Callbacks(nullptr), CurSubmoduleState(&NullSubmoduleState),

clang/lib/Serialization/ASTReader.cpp

 #include "clang/AST/TypeLocVisitor.h"
 #include "clang/AST/UnresolvedSet.h"
 #include "clang/Basic/CommentOptions.h"
 #include "clang/Basic/DiagnosticOptions.h"
 #include "clang/Basic/ExceptionSpecificationType.h"
 #include "clang/Basic/FileManager.h"
 #include "clang/Basic/FileSystemOptions.h"
 #include "clang/Basic/LangOptions.h"
+#include "clang/Basic/MemoryBufferCache.h"
 #include "clang/Basic/ObjCRuntime.h"
 #include "clang/Basic/OperatorKinds.h"
 #include "clang/Basic/Sanitizers.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Basic/SourceManagerInternals.h"
 #include "clang/Basic/Specifiers.h"
 #include "clang/Basic/TargetInfo.h"
 #include "clang/Basic/TargetOptions.h"
if (isExtHandlingFromDiagsError(Diags) &&
     if (Complain)
       Diags.Report(diag::err_pch_diagopt_mismatch) << "-pedantic-errors";
     return true;
   }
 
   return checkDiagnosticGroupMappings(StoredDiags, Diags, Complain);
 }
 
-bool PCHValidator::ReadDiagnosticOptions(
-    IntrusiveRefCntPtr<DiagnosticOptions> DiagOpts, bool Complain) {
-  DiagnosticsEngine &ExistingDiags = PP.getDiagnostics();
-  IntrusiveRefCntPtr<DiagnosticIDs> DiagIDs(ExistingDiags.getDiagnosticIDs());
-  IntrusiveRefCntPtr<DiagnosticsEngine> Diags(
-      new DiagnosticsEngine(DiagIDs, DiagOpts.get()));
-  // This should never fail, because we would have processed these options
-  // before writing them to an ASTFile.
-  ProcessWarningOptions(*Diags, *DiagOpts, /*Report*/false);
-
-  ModuleManager &ModuleMgr = Reader.getModuleManager();
-  assert(ModuleMgr.size() >= 1 && "what ASTFile is this then");
-
+/// Return the top import module if it is implicit, nullptr otherwise.
+static Module *getTopImportImplicitModule(ModuleManager &ModuleMgr,
+                                          Preprocessor &PP) {
   // If the original import came from a file explicitly generated by the user,
   // don't check the diagnostic mappings.
   // FIXME: currently this is approximated by checking whether this is not a
   // module import of an implicitly-loaded module file.
   // Note: ModuleMgr.rbegin() may not be the current module, but it must be in
   // the transitive closure of its imports, since unrelated modules cannot be
   // imported until after this module finishes validation.
   ModuleFile *TopImport = &*ModuleMgr.rbegin();
   while (!TopImport->ImportedBy.empty())
     TopImport = TopImport->ImportedBy[0];
   if (TopImport->Kind != MK_ImplicitModule)
-    return false;
+    return nullptr;
 
   StringRef ModuleName = TopImport->ModuleName;
   assert(!ModuleName.empty() && "diagnostic options read before module name");
 
   Module *M = PP.getHeaderSearchInfo().lookupModule(ModuleName);
   assert(M && "missing module");
+  return M;
+}
+
+bool PCHValidator::ReadDiagnosticOptions(
+    IntrusiveRefCntPtr<DiagnosticOptions> DiagOpts, bool Complain) {
+  DiagnosticsEngine &ExistingDiags = PP.getDiagnostics();
+  IntrusiveRefCntPtr<DiagnosticIDs> DiagIDs(ExistingDiags.getDiagnosticIDs());
+  IntrusiveRefCntPtr<DiagnosticsEngine> Diags(
+      new DiagnosticsEngine(DiagIDs, DiagOpts.get()));
+  // This should never fail, because we would have processed these options
+  // before writing them to an ASTFile.
+  ProcessWarningOptions(*Diags, *DiagOpts, /*Report*/false);
+
+  ModuleManager &ModuleMgr = Reader.getModuleManager();
+  assert(ModuleMgr.size() >= 1 && "what ASTFile is this then");
+
+  Module *TopM = getTopImportImplicitModule(ModuleMgr, PP);
+  if (!TopM)
+    return false;
 
   // FIXME: if the diagnostics are incompatible, save a DiagnosticOptions that
   // contains the union of their flags.
-  return checkDiagnosticMappings(*Diags, ExistingDiags, M->IsSystem, Complain);
+  return checkDiagnosticMappings(*Diags, ExistingDiags, TopM->IsSystem,
+                                 Complain);
 }
 
 /// \brief Collect the macro definitions provided by the given preprocessor
 /// options.
 static void
 collectMacroDefinitions(const PreprocessorOptions &PPOpts,
                         MacroDefinitionsMap &Macros,
                         SmallVectorImpl<StringRef> *MacroNames = nullptr) {
ASTReader::readUnhashedControlBlock(ModuleFile &F, bool WasImportedBy,
   bool AllowCompatibleConfigurationMismatch =
       F.Kind == MK_ExplicitModule || F.Kind == MK_PrebuiltModule;
 
   ASTReadResult Result = readUnhashedControlBlockImpl(
       &F, F.Data, ClientLoadCapabilities, AllowCompatibleConfigurationMismatch,
       Listener.get(),
       WasImportedBy ? false : HSOpts.ModulesValidateDiagnosticOptions);
 
+  // If F was directly imported by another module, it's implicitly validated by
+  // the importing module.
   if (DisableValidation || WasImportedBy ||
       (AllowConfigurationMismatch && Result == ConfigurationMismatch))
     return Success;
 
-  if (Result == Failure)
+  if (Result == Failure) {
     Error("malformed block record in AST file");
+    return Failure;
+  }
+
+  if (Result == OutOfDate && F.Kind == MK_ImplicitModule) {
+    // If this module has already been finalized in the PCMCache, we're stuck
+    // with it; we can only load a single version of each module.
+    //
+    // This can happen when a module is imported in two contexts: in one, as a
+    // user module; in another, as a system module (due to an import from
+    // another module marked with the [system] flag).  It usually indicates a
+    // bug in the module map: this module should also be marked with [system].
+    //
+    // If -Wno-system-headers (the default), and the first import is as a
+    // system module, then validation will fail during the as-user import,
+    // since -Werror flags won't have been validated.  However, it's reasonable
+    // to treat this consistently as a system module.
+    //
+    // If -Wsystem-headers, the PCM on disk was built with
+    // -Wno-system-headers, and the first import is as a user module, then
+    // validation will fail during the as-system import since the PCM on disk
+    // doesn't guarantee that -Werror was respected.  However, the -Werror
+    // flags were checked during the initial as-user import.
+    if (PCMCache.isBufferFinal(F.FileName)) {
+      Diag(diag::warn_module_system_bit_conflict) << F.FileName;
+      return Success;
+    }
+  }
 
   return Result;
 }
 
 ASTReader::ASTReadResult ASTReader::readUnhashedControlBlockImpl(
     ModuleFile *F, llvm::StringRef StreamData, unsigned ClientLoadCapabilities,
     bool AllowCompatibleConfigurationMismatch, ASTReaderListener *Listener,
     bool ValidateDiagnosticOptions) {
case SIGNATURE: {
         std::copy(Record.begin(), Record.end(), F->Signature.data());
       break;
     }
     case DIAGNOSTIC_OPTIONS: {
       bool Complain = (ClientLoadCapabilities & ARR_OutOfDate) == 0;
       if (Listener && ValidateDiagnosticOptions &&
           !AllowCompatibleConfigurationMismatch &&
           ParseDiagnosticOptions(Record, Complain, *Listener))
-        return OutOfDate;
+        Result = OutOfDate; // Don't return early.  Read the signature.
       break;
     }
     case DIAG_PRAGMA_MAPPINGS:
       if (!F)
         break;
       if (F->PragmaDiagMappings.empty())
         F->PragmaDiagMappings.swap(Record);
       else
LLVM_DUMP_METHOD void ASTReader::dump() {
   for (ModuleFile &M : ModuleMgr)
     M.dump();
 }
 
 /// Return the amount of memory used by memory buffers, breaking down
 /// by heap-backed versus mmap'ed memory.
 void ASTReader::getMemoryBufferSizes(MemoryBufferSizes &sizes) const {
   for (ModuleFile &I : ModuleMgr) {
-    if (llvm::MemoryBuffer *buf = I.Buffer.get()) {
+    if (llvm::MemoryBuffer *buf = I.Buffer) {
       size_t bytes = buf->getBufferSize();
       switch (buf->getBufferKind()) {
         case llvm::MemoryBuffer::MemoryBuffer_Malloc:
           sizes.malloc_bytes += bytes;
           break;
         case llvm::MemoryBuffer::MemoryBuffer_MMap:
           sizes.mmap_bytes += bytes;
           break;
ASTReader::ASTReader(Preprocessor &PP, ASTContext &Context,
                      bool AllowConfigurationMismatch, bool ValidateSystemInputs,
                      bool UseGlobalIndex,
                      std::unique_ptr<llvm::Timer> ReadTimer)
     : Listener(DisableValidation
                    ? cast<ASTReaderListener>(new SimpleASTReaderListener(PP))
                    : cast<ASTReaderListener>(new PCHValidator(PP, *this))),
       SourceMgr(PP.getSourceManager()), FileMgr(PP.getFileManager()),
       PCHContainerRdr(PCHContainerRdr), Diags(PP.getDiagnostics()), PP(PP),
-      Context(Context), ModuleMgr(PP.getFileManager(), PCHContainerRdr),
-      DummyIdResolver(PP), ReadTimer(std::move(ReadTimer)), isysroot(isysroot),
+      Context(Context),
+      ModuleMgr(PP.getFileManager(), PP.getPCMCache(), PCHContainerRdr),
+      PCMCache(PP.getPCMCache()), DummyIdResolver(PP),
+      ReadTimer(std::move(ReadTimer)), isysroot(isysroot),
       DisableValidation(DisableValidation),
       AllowASTWithCompilerErrors(AllowASTWithCompilerErrors),
       AllowConfigurationMismatch(AllowConfigurationMismatch),
       ValidateSystemInputs(ValidateSystemInputs),
       UseGlobalIndex(UseGlobalIndex), CurrSwitchCaseStmts(&SwitchCaseStmts) {
   SourceMgr.setExternalSLocEntrySource(this);
 
   for (const auto &Ext : Extensions) {

clang/lib/Serialization/ASTWriter.cpp

 #include "clang/AST/TemplateName.h"
 #include "clang/AST/Type.h"
 #include "clang/AST/TypeLocVisitor.h"
 #include "clang/Basic/DiagnosticOptions.h"
 #include "clang/Basic/FileManager.h"
 #include "clang/Basic/FileSystemOptions.h"
 #include "clang/Basic/LLVM.h"
 #include "clang/Basic/LangOptions.h"
+#include "clang/Basic/MemoryBufferCache.h"
 #include "clang/Basic/Module.h"
 #include "clang/Basic/ObjCRuntime.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Basic/SourceManagerInternals.h"
 #include "clang/Basic/TargetInfo.h"
 #include "clang/Basic/TargetOptions.h"
 #include "clang/Basic/Version.h"
 #include "clang/Basic/VersionTuple.h"
void ASTWriter::SetSelectorOffset(Selector Sel, uint32_t Offset) {
   // Don't record offsets for selectors that are also available in a different
   // file.
   if (ID < FirstSelectorID)
     return;
   SelectorOffsets[ID - FirstSelectorID] = Offset;
 }
 
 ASTWriter::ASTWriter(llvm::BitstreamWriter &Stream,
-                     SmallVectorImpl<char> &Buffer,
+                     SmallVectorImpl<char> &Buffer, MemoryBufferCache &PCMCache,
                      ArrayRef<std::shared_ptr<ModuleFileExtension>> Extensions,
                      bool IncludeTimestamps)
-    : Stream(Stream), Buffer(Buffer), IncludeTimestamps(IncludeTimestamps) {
+    : Stream(Stream), Buffer(Buffer), PCMCache(PCMCache),
+      IncludeTimestamps(IncludeTimestamps) {
   for (const auto &Ext : Extensions) {
     if (auto Writer = Ext->createExtensionWriter(*this))
       ModuleFileExtensionWriters.push_back(std::move(Writer));
   }
 }
 
 ASTWriter::~ASTWriter() {
   llvm::DeleteContainerSeconds(FileDeclIDs);
ASTFileSignature ASTWriter::WriteAST(Sema &SemaRef,
   ASTFileSignature Signature =
       WriteASTCore(SemaRef, isysroot, OutputFile, WritingModule);
   Context = nullptr;
   PP = nullptr;
   this->WritingModule = nullptr;
   this->BaseDirectory.clear();
 
   WritingAST = false;
+  if (SemaRef.Context.getLangOpts().ImplicitModules && WritingModule) {
+    // Construct MemoryBuffer and update buffer manager.
+    PCMCache.addBuffer(OutputFile,
+                       llvm::MemoryBuffer::getMemBufferCopy(
+                           StringRef(Buffer.begin(), Buffer.size())));
+  }
   return Signature;
 }
 
 template<typename Vector>
 static void AddLazyVectorDecls(ASTWriter &Writer, Vector &Vec,
                                ASTWriter::RecordData &Record) {
   for (typename Vector::iterator I = Vec.begin(nullptr, true), E = Vec.end();
        I != E; ++I) {

clang/lib/Serialization/GeneratePCH.cpp

 
 PCHGenerator::PCHGenerator(
     const Preprocessor &PP, StringRef OutputFile, StringRef isysroot,
     std::shared_ptr<PCHBuffer> Buffer,
     ArrayRef<std::shared_ptr<ModuleFileExtension>> Extensions,
     bool AllowASTWithErrors, bool IncludeTimestamps)
     : PP(PP), OutputFile(OutputFile), isysroot(isysroot.str()),
       SemaPtr(nullptr), Buffer(Buffer), Stream(Buffer->Data),
-      Writer(Stream, Buffer->Data, Extensions, IncludeTimestamps),
+      Writer(Stream, Buffer->Data, PP.getPCMCache(), Extensions,
+             IncludeTimestamps),
       AllowASTWithErrors(AllowASTWithErrors) {
   Buffer->IsComplete = false;
 }
 
 PCHGenerator::~PCHGenerator() {
 }
 
 void PCHGenerator::HandleTranslationUnit(ASTContext &Ctx) {

clang/lib/Serialization/ModuleManager.cpp

 //===--- ModuleManager.cpp - Module Manager ---------------------*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 //
 //  This file defines the ModuleManager class, which manages a set of loaded
 //  modules for the ASTReader.
 //
 //===----------------------------------------------------------------------===//
 #include "clang/Serialization/ModuleManager.h"
+#include "clang/Basic/MemoryBufferCache.h"
 #include "clang/Frontend/PCHContainerOperations.h"
 #include "clang/Lex/HeaderSearch.h"
 #include "clang/Lex/ModuleMap.h"
 #include "clang/Serialization/GlobalModuleIndex.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Path.h"
 #include <system_error>
 
if (NewModule->Kind == MK_ImplicitModule) {
     if (!FileMgr.getNoncachedStatValue(TimestampFilename, Status))
       NewModule->InputFilesValidationTimestamp =
           llvm::sys::toTimeT(Status.getLastModificationTime());
   }
 
   // Load the contents of the module
   if (std::unique_ptr<llvm::MemoryBuffer> Buffer = lookupBuffer(FileName)) {
     // The buffer was already provided for us.
-    NewModule->Buffer = std::move(Buffer);
+    NewModule->Buffer = &PCMCache->addBuffer(FileName, std::move(Buffer));
+  } else if (llvm::MemoryBuffer *Buffer = PCMCache->lookupBuffer(FileName)) {
+    NewModule->Buffer = Buffer;
   } else {
     // Open the AST file.
     llvm::ErrorOr<std::unique_ptr<llvm::MemoryBuffer>> Buf((std::error_code()));
     if (FileName == "-") {
       Buf = llvm::MemoryBuffer::getSTDIN();
     } else {
       // Leave the FileEntry open so if it gets read again by another
       // ModuleManager it must be the same underlying file.
       // FIXME: Because FileManager::getFile() doesn't guarantee that it will
       // give us an open file, this may not be 100% reliable.
       Buf = FileMgr.getBufferForFile(NewModule->File,
                                      /*IsVolatile=*/false,
                                      /*ShouldClose=*/false);
     }
 
     if (!Buf) {
       ErrorStr = Buf.getError().message();
       return Missing;
     }
 
-    NewModule->Buffer = std::move(*Buf);
+    NewModule->Buffer = &PCMCache->addBuffer(FileName, std::move(*Buf));
   }
 
   // Initialize the stream.
   NewModule->Data = PCHContainerRdr.ExtractPCH(*NewModule->Buffer);
 
   // Read the signature eagerly now so that we can check it.  Avoid calling
   // ReadSignature unless there's something to check though.
   if (ExpectedSignature && checkSignature(ReadSignature(NewModule->Data),
-                                          ExpectedSignature, ErrorStr))
+                                          ExpectedSignature, ErrorStr)) {
+    // Try to remove the buffer.  If it can't be removed, then it was already
+    // validated by this process.
+    if (!PCMCache->tryToRemoveBuffer(NewModule->FileName))
+      FileMgr.invalidateCache(NewModule->File);
     return OutOfDate;
+  }
 
   // We're keeping this module.  Store it everywhere.
   Module = Modules[Entry] = NewModule.get();
 
   updateModuleImports(*NewModule, ImportedBy, ImportLoc);
 
   if (!NewModule->isModule())
     PCHChain.push_back(NewModule.get());
if (modMap) {
       if (Module *mod = modMap->findModule(ModuleName)) {
         mod->setASTFile(nullptr);
       }
     }
 
     // Files that didn't make it through ReadASTCore successfully will be
     // rebuilt (or there was an error). Invalidate them so that we can load the
     // new files that will be renamed over the old ones.
-    if (LoadedSuccessfully.count(&*victim) == 0)
+    //
+    // The PCMCache tracks whether the module was succesfully loaded in another
+    // thread/context; in that case, it won't need to be rebuilt (and we can't
+    // safely invalidate it anyway).
+    if (LoadedSuccessfully.count(&*victim) == 0 &&
+        !PCMCache->tryToRemoveBuffer(victim->FileName))
       FileMgr.invalidateCache(victim->File);
   }
 
   // Delete the modules.
   Chain.erase(Chain.begin() + (First - begin()), Chain.end());
 }
 
 void

 
 void ModuleManager::moduleFileAccepted(ModuleFile *MF) {
   if (!GlobalIndex || GlobalIndex->loadedModuleFile(MF))
     return;
 
   ModulesInCommonWithGlobalIndex.push_back(MF);
 }
 
-ModuleManager::ModuleManager(FileManager &FileMgr,
+ModuleManager::ModuleManager(FileManager &FileMgr, MemoryBufferCache &PCMCache,
                              const PCHContainerReader &PCHContainerRdr)
-    : FileMgr(FileMgr), PCHContainerRdr(PCHContainerRdr), GlobalIndex(),
-      FirstVisitState(nullptr) {}
+    : FileMgr(FileMgr), PCMCache(&PCMCache), PCHContainerRdr(PCHContainerRdr),
+      GlobalIndex(), FirstVisitState(nullptr) {}
 
 ModuleManager::~ModuleManager() { delete FirstVisitState; }
 
 void ModuleManager::visit(llvm::function_ref<bool(ModuleFile &M)> Visitor,
                           llvm::SmallPtrSetImpl<ModuleFile *> *ModuleFilesHit) {
   // If the visitation order vector is the wrong size, recompute the order.
   if (VisitOrder.size() != Chain.size()) {
     unsigned N = size();

clang/test/Modules/Inputs/system-out-of-date/X.h

+#import <Y.h>

clang/test/Modules/Inputs/system-out-of-date/Y.h

+//empty

clang/test/Modules/Inputs/system-out-of-date/Z.h

+#import <Y.h>

clang/test/Modules/Inputs/system-out-of-date/module.map

+module X [system] {
+  header "X.h" // imports Y
+  export *
+}
+module Y {
+  header "Y.h"
+  export *
+}
+module Z {
+  header "Z.h" // imports Y
+  export *
+}

clang/test/Modules/Inputs/warning-mismatch/Mismatch.h

+struct Mismatch { int i; };

clang/test/Modules/Inputs/warning-mismatch/System.h

+#import "Mismatch.h"
+struct System { int i; };

clang/test/Modules/Inputs/warning-mismatch/module.modulemap

+module System [system] {
+  header "System.h"
+}
+
+module Mismatch {
+  header "Mismatch.h"
+}

clang/test/Modules/outofdate-rebuild.m

+// RUN: rm -rf %t.cache
+// RUN: echo "@import CoreText;" > %t.m
+// RUN: %clang_cc1 -fdisable-module-hash -fmodules-cache-path=%t.cache \
+// RUN:   -fmodules -fimplicit-module-maps -I%S/Inputs/outofdate-rebuild %s \
+// RUN:   -fsyntax-only -Rmodule-build
+// RUN: %clang_cc1 -DMISMATCH -Werror -fdisable-module-hash \
+// RUN:   -fmodules-cache-path=%t.cache -fmodules -fimplicit-module-maps \
+// RUN:   -I%S/Inputs/outofdate-rebuild %t.m -fsyntax-only -Rmodule-build
+// RUN: %clang_cc1 -fdisable-module-hash -fmodules-cache-path=%t.cache \
+// RUN:   -fmodules -fimplicit-module-maps -I%S/Inputs/outofdate-rebuild %s \
+// RUN:   -fsyntax-only -Rmodule-build
+
+// This testcase reproduces a use-after-free in when ModuleManager removes an
+// entry from the PCMCache without notifying its parent ASTReader.
+@import Cocoa;

clang/test/Modules/system-out-of-date-test.m

+// RUN: rm -rf %t.cache
+// RUN: echo '@import X;' | \
+// RUN:   %clang_cc1 -fmodules -fimplicit-module-maps \
+// RUN:     -fmodules-cache-path=%t.cache -I%S/Inputs/system-out-of-date \
+// RUN:     -fsyntax-only -x objective-c -
+//
+// Build something with different diagnostic options.
+// RUN: %clang_cc1 -fmodules -fimplicit-module-maps \
+// RUN:   -fmodules-cache-path=%t.cache -I%S/Inputs/system-out-of-date \
+// RUN:   -fsyntax-only %s -Wnon-modular-include-in-framework-module \
+// RUN:   -Werror=non-modular-include-in-framework-module 2>&1 \
+// RUN: | FileCheck %s
+@import X;
+
+#import <Z.h>
+// CHECK: While building module 'Z' imported from
+// CHECK: {{.*}}Y-{{.*}}pcm' was validated as a system module and is now being imported as a non-system module

clang/test/Modules/warning-mismatch.m

+// RUN: rm -rf %t.cache
+// RUN: echo "@import Mismatch;" >%t.m
+// RUN: %clang_cc1 -Wno-system-headers -fdisable-module-hash \
+// RUN:   -fmodules-cache-path=%t.cache -fmodules -fimplicit-module-maps \
+// RUN:   -I%S/Inputs/warning-mismatch %t.m -fsyntax-only -Rmodule-build
+// RUN: %clang_cc1 -Wsystem-headers -fdisable-module-hash \
+// RUN:   -fmodules-cache-path=%t.cache -fmodules -fimplicit-module-maps \
+// RUN:   -I%S/Inputs/warning-mismatch %s -fsyntax-only -Rmodule-build
+
+// This testcase triggers a warning flag mismatch in an already validated
+// header.
+@import Mismatch;
+@import System;

clang/unittests/Basic/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   Support
   )
 
 add_clang_unittest(BasicTests
   CharInfoTest.cpp
   DiagnosticTest.cpp
   FileManagerTest.cpp
+  MemoryBufferCacheTest.cpp
   SourceManagerTest.cpp
   VirtualFileSystemTest.cpp
   )
 
 target_link_libraries(BasicTests
   clangBasic
   clangLex
   )

clang/unittests/Basic/MemoryBufferCacheTest.cpp

+//===- MemoryBufferCacheTest.cpp - MemoryBufferCache tests ----------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Basic/MemoryBufferCache.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "gtest/gtest.h"
+
+using namespace llvm;
+using namespace clang;
+
+namespace {
+
+std::unique_ptr<MemoryBuffer> getBuffer(int I) {
+  SmallVector<char, 8> Bytes;
+  raw_svector_ostream(Bytes) << "data:" << I;
+  return MemoryBuffer::getMemBuffer(StringRef(Bytes.data(), Bytes.size()));
+}
+
+TEST(MemoryBufferCacheTest, addBuffer) {
+  auto B1 = getBuffer(1);
+  auto B2 = getBuffer(2);
+  auto B3 = getBuffer(3);
+  auto *RawB1 = B1.get();
+  auto *RawB2 = B2.get();
+  auto *RawB3 = B3.get();
+
+  // Add a few buffers.
+  MemoryBufferCache Cache;
+  EXPECT_EQ(RawB1, &Cache.addBuffer("1", std::move(B1)));
+  EXPECT_EQ(RawB2, &Cache.addBuffer("2", std::move(B2)));
+  EXPECT_EQ(RawB3, &Cache.addBuffer("3", std::move(B3)));
+  EXPECT_EQ(RawB1, Cache.lookupBuffer("1"));
+  EXPECT_EQ(RawB2, Cache.lookupBuffer("2"));
+  EXPECT_EQ(RawB3, Cache.lookupBuffer("3"));
+  EXPECT_FALSE(Cache.isBufferFinal("1"));
+  EXPECT_FALSE(Cache.isBufferFinal("2"));
+  EXPECT_FALSE(Cache.isBufferFinal("3"));
+
+  // Remove the middle buffer.
+  EXPECT_FALSE(Cache.tryToRemoveBuffer("2"));
+  EXPECT_EQ(nullptr, Cache.lookupBuffer("2"));
+  EXPECT_FALSE(Cache.isBufferFinal("2"));
+
+  // Replace the middle buffer.
+  B2 = getBuffer(2);
+  ASSERT_NE(RawB2, B2.get());
+  RawB2 = B2.get();
+  EXPECT_EQ(RawB2, &Cache.addBuffer("2", std::move(B2)));
+
+  // Check that nothing is final.
+  EXPECT_FALSE(Cache.isBufferFinal("1"));
+  EXPECT_FALSE(Cache.isBufferFinal("2"));
+  EXPECT_FALSE(Cache.isBufferFinal("3"));
+}
+
+TEST(MemoryBufferCacheTest, finalizeCurrentBuffers) {
+  // Add a buffer.
+  MemoryBufferCache Cache;
+  auto B1 = getBuffer(1);
+  auto *RawB1 = B1.get();
+  Cache.addBuffer("1", std::move(B1));
+  ASSERT_FALSE(Cache.isBufferFinal("1"));
+
+  // Finalize it.
+  Cache.finalizeCurrentBuffers();
+  EXPECT_TRUE(Cache.isBufferFinal("1"));
+  EXPECT_TRUE(Cache.tryToRemoveBuffer("1"));
+  EXPECT_EQ(RawB1, Cache.lookupBuffer("1"));
+  EXPECT_TRUE(Cache.isBufferFinal("1"));
+
+  // Repeat.
+  auto B2 = getBuffer(2);
+  auto *RawB2 = B2.get();
+  Cache.addBuffer("2", std::move(B2));
+  EXPECT_FALSE(Cache.isBufferFinal("2"));
+
+  Cache.finalizeCurrentBuffers();
+  EXPECT_TRUE(Cache.isBufferFinal("1"));
+  EXPECT_TRUE(Cache.isBufferFinal("2"));
+  EXPECT_TRUE(Cache.tryToRemoveBuffer("1"));
+  EXPECT_TRUE(Cache.tryToRemoveBuffer("2"));
+  EXPECT_EQ(RawB1, Cache.lookupBuffer("1"));
+  EXPECT_EQ(RawB2, Cache.lookupBuffer("2"));
+  EXPECT_TRUE(Cache.isBufferFinal("1"));
+  EXPECT_TRUE(Cache.isBufferFinal("2"));
+}
+
+} // namespace

clang/unittests/Basic/SourceManagerTest.cpp

 //===- unittests/Basic/SourceManagerTest.cpp ------ SourceManager tests ---===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "clang/Basic/SourceManager.h"
 #include "clang/Basic/Diagnostic.h"
 #include "clang/Basic/DiagnosticOptions.h"
 #include "clang/Basic/FileManager.h"
 #include "clang/Basic/LangOptions.h"
+#include "clang/Basic/MemoryBufferCache.h"
 #include "clang/Basic/TargetInfo.h"
 #include "clang/Basic/TargetOptions.h"
 #include "clang/Lex/HeaderSearch.h"
 #include "clang/Lex/HeaderSearchOptions.h"
 #include "clang/Lex/ModuleLoader.h"
 #include "clang/Lex/Preprocessor.h"
 #include "clang/Lex/PreprocessorOptions.h"
 #include "llvm/ADT/SmallString.h"
const char *source =
     "#define M(x) [x]\n"
     "M(foo)";
   std::unique_ptr<llvm::MemoryBuffer> Buf =
       llvm::MemoryBuffer::getMemBuffer(source);
   FileID mainFileID = SourceMgr.createFileID(std::move(Buf));
   SourceMgr.setMainFileID(mainFileID);
 
   VoidModuleLoader ModLoader;
+  MemoryBufferCache PCMCache;
   HeaderSearch HeaderInfo(std::make_shared<HeaderSearchOptions>(), SourceMgr,
                           Diags, LangOpts, &*Target);
   Preprocessor PP(std::make_shared<PreprocessorOptions>(), Diags, LangOpts,
-                  SourceMgr, HeaderInfo, ModLoader,
+                  SourceMgr, PCMCache, HeaderInfo, ModLoader,
                   /*IILookup =*/nullptr,
                   /*OwnsHeaderSearch =*/false);
   PP.Initialize(*Target);
   PP.EnterMainSourceFile();
 
   std::vector<Token> toks;
   while (1) {
     Token tok;
TEST_F(SourceManagerTest, getMacroArgExpandedLocation) {
   FileID mainFileID = SourceMgr.createFileID(std::move(MainBuf));
   SourceMgr.setMainFileID(mainFileID);
 
   const FileEntry *headerFile = FileMgr.getVirtualFile("/test-header.h",
                                                  HeaderBuf->getBufferSize(), 0);
   SourceMgr.overrideFileContents(headerFile, std::move(HeaderBuf));
 
   VoidModuleLoader ModLoader;
+  MemoryBufferCache PCMCache;
   HeaderSearch HeaderInfo(std::make_shared<HeaderSearchOptions>(), SourceMgr,
                           Diags, LangOpts, &*Target);
   Preprocessor PP(std::make_shared<PreprocessorOptions>(), Diags, LangOpts,
-                  SourceMgr, HeaderInfo, ModLoader,
+                  SourceMgr, PCMCache, HeaderInfo, ModLoader,
                   /*IILookup =*/nullptr,
                   /*OwnsHeaderSearch =*/false);
   PP.Initialize(*Target);
   PP.EnterMainSourceFile();
 
   std::vector<Token> toks;
   while (1) {
     Token tok;
std::unique_ptr<llvm::MemoryBuffer> MainBuf =
       llvm::MemoryBuffer::getMemBuffer(main);
   SourceMgr.setMainFileID(SourceMgr.createFileID(std::move(MainBuf)));
 
   const FileEntry *headerFile = FileMgr.getVirtualFile("/test-header.h",
                                                  HeaderBuf->getBufferSize(), 0);
   SourceMgr.overrideFileContents(headerFile, std::move(HeaderBuf));
 
   VoidModuleLoader ModLoader;
+  MemoryBufferCache PCMCache;
   HeaderSearch HeaderInfo(std::make_shared<HeaderSearchOptions>(), SourceMgr,
                           Diags, LangOpts, &*Target);
   Preprocessor PP(std::make_shared<PreprocessorOptions>(), Diags, LangOpts,
-                  SourceMgr, HeaderInfo, ModLoader,
+                  SourceMgr, PCMCache, HeaderInfo, ModLoader,
                   /*IILookup =*/nullptr,
                   /*OwnsHeaderSearch =*/false);
   PP.Initialize(*Target);
 
   std::vector<MacroAction> Macros;
   PP.addPPCallbacks(llvm::make_unique<MacroTracker>(Macros));
 
   PP.EnterMainSourceFile();

clang/unittests/Lex/LexerTest.cpp

 //===- unittests/Lex/LexerTest.cpp ------ Lexer tests ---------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "clang/Lex/Lexer.h"
 #include "clang/Basic/Diagnostic.h"
 #include "clang/Basic/DiagnosticOptions.h"
 #include "clang/Basic/FileManager.h"
 #include "clang/Basic/LangOptions.h"
+#include "clang/Basic/MemoryBufferCache.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Basic/TargetInfo.h"
 #include "clang/Basic/TargetOptions.h"
 #include "clang/Lex/HeaderSearch.h"
 #include "clang/Lex/HeaderSearchOptions.h"
 #include "clang/Lex/ModuleLoader.h"
 #include "clang/Lex/Preprocessor.h"
 #include "clang/Lex/PreprocessorOptions.h"
protected:
   }
 
   std::vector<Token> Lex(StringRef Source) {
     std::unique_ptr<llvm::MemoryBuffer> Buf =
         llvm::MemoryBuffer::getMemBuffer(Source);
     SourceMgr.setMainFileID(SourceMgr.createFileID(std::move(Buf)));
 
     VoidModuleLoader ModLoader;
+    MemoryBufferCache PCMCache;
     HeaderSearch HeaderInfo(std::make_shared<HeaderSearchOptions>(), SourceMgr,
                             Diags, LangOpts, Target.get());
     Preprocessor PP(std::make_shared<PreprocessorOptions>(), Diags, LangOpts,
-                    SourceMgr, HeaderInfo, ModLoader, /*IILookup =*/nullptr,
+                    SourceMgr, PCMCache, HeaderInfo, ModLoader,
+                    /*IILookup =*/nullptr,
                     /*OwnsHeaderSearch =*/false);
     PP.Initialize(*Target);
     PP.EnterMainSourceFile();
 
     std::vector<Token> toks;
     while (1) {
       Token tok;
       PP.Lex(tok);

clang/unittests/Lex/PPCallbacksTest.cpp

 //===- unittests/Lex/PPCallbacksTest.cpp - PPCallbacks tests ------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===--------------------------------------------------------------===//
 
 #include "clang/Lex/Preprocessor.h"
 #include "clang/AST/ASTConsumer.h"
 #include "clang/AST/ASTContext.h"
 #include "clang/Basic/Diagnostic.h"
 #include "clang/Basic/DiagnosticOptions.h"
 #include "clang/Basic/FileManager.h"
 #include "clang/Basic/LangOptions.h"
+#include "clang/Basic/MemoryBufferCache.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Basic/TargetInfo.h"
 #include "clang/Basic/TargetOptions.h"
 #include "clang/Lex/HeaderSearch.h"
 #include "clang/Lex/HeaderSearchOptions.h"
 #include "clang/Lex/ModuleLoader.h"
 #include "clang/Lex/PreprocessorOptions.h"
 #include "clang/Parse/Parser.h"
protected:
   // the InclusionDirective callback.
   CharSourceRange InclusionDirectiveFilenameRange(const char* SourceText, 
       const char* HeaderPath, bool SystemHeader) {
     std::unique_ptr<llvm::MemoryBuffer> Buf =
         llvm::MemoryBuffer::getMemBuffer(SourceText);
     SourceMgr.setMainFileID(SourceMgr.createFileID(std::move(Buf)));
 
     VoidModuleLoader ModLoader;
+    MemoryBufferCache PCMCache;
 
     HeaderSearch HeaderInfo(std::make_shared<HeaderSearchOptions>(), SourceMgr,
                             Diags, LangOpts, Target.get());
     AddFakeHeader(HeaderInfo, HeaderPath, SystemHeader);
 
     Preprocessor PP(std::make_shared<PreprocessorOptions>(), Diags, LangOpts,
-                    SourceMgr, HeaderInfo, ModLoader,
+                    SourceMgr, PCMCache, HeaderInfo, ModLoader,
                     /*IILookup =*/nullptr,
                     /*OwnsHeaderSearch =*/false);
     PP.Initialize(*Target);
     InclusionDirectiveCallbacks* Callbacks = new InclusionDirectiveCallbacks;
     PP.addPPCallbacks(std::unique_ptr<PPCallbacks>(Callbacks));
 
     // Lex source text.
     PP.EnterMainSourceFile();
PragmaOpenCLExtensionCall(const char* SourceText) {
     LangOptions OpenCLLangOpts;
     OpenCLLangOpts.OpenCL = 1;
 
     std::unique_ptr<llvm::MemoryBuffer> SourceBuf =
         llvm::MemoryBuffer::getMemBuffer(SourceText, "test.cl");
     SourceMgr.setMainFileID(SourceMgr.createFileID(std::move(SourceBuf)));
 
     VoidModuleLoader ModLoader;
+    MemoryBufferCache PCMCache;
     HeaderSearch HeaderInfo(std::make_shared<HeaderSearchOptions>(), SourceMgr,
                             Diags, OpenCLLangOpts, Target.get());
 
     Preprocessor PP(std::make_shared<PreprocessorOptions>(), Diags,
-                    OpenCLLangOpts, SourceMgr, HeaderInfo, ModLoader,
+                    OpenCLLangOpts, SourceMgr, PCMCache, HeaderInfo, ModLoader,
                     /*IILookup =*/nullptr,
                     /*OwnsHeaderSearch =*/false);
     PP.Initialize(*Target);
 
     // parser actually sets correct pragma handlers for preprocessor
     // according to LangOptions, so we init Parser to register opencl
     // pragma handlers
     ASTContext Context(OpenCLLangOpts, SourceMgr,

clang/unittests/Lex/PPConditionalDirectiveRecordTest.cpp

 //===- unittests/Lex/PPConditionalDirectiveRecordTest.cpp-PP directive tests =//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "clang/Lex/PPConditionalDirectiveRecord.h"
 #include "clang/Basic/Diagnostic.h"
 #include "clang/Basic/DiagnosticOptions.h"
 #include "clang/Basic/FileManager.h"
 #include "clang/Basic/LangOptions.h"
+#include "clang/Basic/MemoryBufferCache.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Basic/TargetInfo.h"
 #include "clang/Basic/TargetOptions.h"
 #include "clang/Lex/HeaderSearch.h"
 #include "clang/Lex/HeaderSearchOptions.h"
 #include "clang/Lex/ModuleLoader.h"
 #include "clang/Lex/Preprocessor.h"
 #include "clang/Lex/PreprocessorOptions.h"
const char *source =
       "#endif\n"
       "9\n";
 
   std::unique_ptr<llvm::MemoryBuffer> Buf =
       llvm::MemoryBuffer::getMemBuffer(source);
   SourceMgr.setMainFileID(SourceMgr.createFileID(std::move(Buf)));
 
   VoidModuleLoader ModLoader;
+  MemoryBufferCache PCMCache;
   HeaderSearch HeaderInfo(std::make_shared<HeaderSearchOptions>(), SourceMgr,
                           Diags, LangOpts, Target.get());
   Preprocessor PP(std::make_shared<PreprocessorOptions>(), Diags, LangOpts,
-                  SourceMgr, HeaderInfo, ModLoader,
+                  SourceMgr, PCMCache, HeaderInfo, ModLoader,
                   /*IILookup =*/nullptr,
                   /*OwnsHeaderSearch =*/false);
   PP.Initialize(*Target);
   PPConditionalDirectiveRecord *
     PPRec = new PPConditionalDirectiveRecord(SourceMgr);
   PP.addPPCallbacks(std::unique_ptr<PPCallbacks>(PPRec));
   PP.EnterMainSourceFile();