WIP: Frontend: Use an InMemoryFileSystem for the RemappedFiles parameter in ASTUnit

[dexonsmith]

Summary

Re-implement the RemappedFiles parameter to various ASTUnit entry points by storing buffers in an InMemoryFileSystem. This parameter is commonly used by indexing tools to pass in source files that haven't been saved to disk (yet). This is the same strategy ClangTool switched to in 2015.

The previous implementation added them to PreprocessorOptions::RemappedFileBuffers, which eventually triggers calls to FileManager::getVirtualFile and SourceManager::overrideFileContents. That's one of the blockers for sinking the content cache from SourceManager down to FileManager. The new strategy is also a bit cleaner (the old one predates llvm::vfs::FileSystem).

Follow-ups will handle the other two uses of RemappedFileBuffers in ASTUnit. One is for the preamble's serialized AST. The other is for overriding the main file buffer, which is a bit tricky since we need the ASTWriter to save the preamble-only version.

WIP: I'm not totally happy with this patch yet, but tests are finally passing so I'm posting this for early review. In the meantime I'll continue peeling off what I can for prep patches (or there may be changes I can just delete). Among other things, I want to add a test (in a prep patch) for the handling of a file showing up twice in the same RemappedFiles vector, since it was pretty tough to discover I needed special logic for that.

[dexonsmith]

clang/lib/Frontend/PrecompiledPreamble.cpp

571–579

@kadircet, I'm curious if clang-tools-extra/clangd/Preamble.cpp's call to ASTUnit::CanReuse would benefit from this logic as well. I would guess clangd doesn't even try to reuse preambles when unsaved files have changed (since you might see the same crashes or wrong results I saw before adding this), but after this patch lands you might be able to do that safely.

[dexonsmith]

Note, I just discovered this patch changes behaviour in the ASTReader that was only indirectly tested previously via the file-to-file remapping command-line options. I've blocked this on https://reviews.llvm.org/D91342 for now, since that patch reimplements the file-to-file remapping, and whatever we figure out for updating that test should apply here too.

clang/include/clang/Frontend/ASTUnit.h

 
 namespace llvm {
 
 class MemoryBuffer;
 
 namespace vfs {
 
 class FileSystem;
+class InMemoryFileSystem;
 
 } // namespace vfs
 } // namespace llvm
 
 namespace clang {
 
 class ASTContext;
 class ASTDeserializationListener;
struct StandaloneDiagnostic {
     unsigned LocOffset;
     std::vector<std::pair<unsigned, unsigned>> Ranges;
     std::vector<StandaloneFixIt> FixIts;
   };
 
 private:
   std::shared_ptr<LangOptions>            LangOpts;
   IntrusiveRefCntPtr<DiagnosticsEngine>   Diagnostics;
+
+  /// The top-level filesystem.
+  llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem> FS;
+
+  /// Filesystem for remapped file buffers.
+  llvm::IntrusiveRefCntPtr<llvm::vfs::InMemoryFileSystem> RemappedFilesFS;
+
+  /// The base filesystem from the CompilerInvocation.
+  llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem> BaseFS;
+
   IntrusiveRefCntPtr<FileManager>         FileMgr;
   IntrusiveRefCntPtr<SourceManager>       SourceMgr;
   IntrusiveRefCntPtr<InMemoryModuleCache> ModuleCache;
   std::unique_ptr<HeaderSearch>           HeaderInfo;
   IntrusiveRefCntPtr<TargetInfo>          Target;
   std::shared_ptr<Preprocessor>           PP;
   IntrusiveRefCntPtr<ASTContext>          Ctx;
   std::shared_ptr<TargetOptions>          TargetOpts;
private:
   void CacheCodeCompletionResults();
 
   /// Clear out and deallocate
   void ClearCachedCompletionResults();
 
   explicit ASTUnit(bool MainFileIsAST);
 
   bool Parse(std::shared_ptr<PCHContainerOperations> PCHContainerOps,
-             std::unique_ptr<llvm::MemoryBuffer> OverrideMainBuffer,
-             IntrusiveRefCntPtr<llvm::vfs::FileSystem> VFS);
+             std::unique_ptr<llvm::MemoryBuffer> OverrideMainBuffer);
 
   std::unique_ptr<llvm::MemoryBuffer> getMainBufferWithPrecompiledPreamble(
       std::shared_ptr<PCHContainerOperations> PCHContainerOps,
-      CompilerInvocation &PreambleInvocationIn,
-      IntrusiveRefCntPtr<llvm::vfs::FileSystem> VFS, bool AllowRebuild = true,
+      CompilerInvocation &PreambleInvocationIn, bool AllowRebuild = true,
       unsigned MaxLines = 0);
   void RealizeTopLevelDeclsFromPreamble();
 
   /// Transfers ownership of the objects (like SourceManager) from
   /// \param CI to this ASTUnit.
   void transferASTDataFromCompilerInstance(CompilerInstance &CI);
 
   /// Allows us to assert that ASTUnit is not being used concurrently,
public:
   /// Determine what kind of translation unit this AST represents.
   TranslationUnitKind getTranslationUnitKind() const { return TUKind; }
 
   /// Determine the input kind this AST unit represents.
   InputKind getInputKind() const;
 
   /// A mapping from a file name to the memory buffer that stores the
   /// remapped contents of that file.
-  using RemappedFile = std::pair<std::string, llvm::MemoryBuffer *>;
+  using RemappedFile =
+      std::pair<std::string, std::unique_ptr<llvm::MemoryBuffer>>;
 
   /// Create a ASTUnit. Gets ownership of the passed CompilerInvocation.
   static std::unique_ptr<ASTUnit>
   create(std::shared_ptr<CompilerInvocation> CI,
          IntrusiveRefCntPtr<DiagnosticsEngine> Diags,
          CaptureDiagsKind CaptureDiagnostics, bool UserFilesAreVolatile);
 
   enum WhatToLoad {

 private:
   /// Helper function for \c LoadFromCompilerInvocation() and
   /// \c LoadFromCommandLine(), which loads an AST from a compiler invocation.
   ///
   /// \param PrecompilePreambleAfterNParses After how many parses the preamble
   /// of this translation unit should be precompiled, to improve the performance
   /// of reparsing. Set to zero to disable preambles.
   ///
-  /// \param VFS - A llvm::vfs::FileSystem to be used for all file accesses.
-  /// Note that preamble is saved to a temporary directory on a RealFileSystem,
-  /// so in order for it to be loaded correctly, VFS should have access to
-  /// it(i.e., be an overlay over RealFileSystem).
-  ///
   /// \returns \c true if a catastrophic failure occurred (which means that the
   /// \c ASTUnit itself is invalid), or \c false otherwise.
   bool LoadFromCompilerInvocation(
       std::shared_ptr<PCHContainerOperations> PCHContainerOps,
-      unsigned PrecompilePreambleAfterNParses,
-      IntrusiveRefCntPtr<llvm::vfs::FileSystem> VFS);
+      unsigned PrecompilePreambleAfterNParses);
 
 public:
   /// Create an ASTUnit from a source file, via a CompilerInvocation
   /// object, by invoking the optionally provided ASTFrontendAction.
   ///
   /// \param CI - The compiler invocation to use; it must have exactly one input
   /// source file. The ASTUnit takes ownership of the CompilerInvocation object.
   ///
public:
   //
   // FIXME: Move OnlyLocalDecls, UseBumpAllocator to setters on the ASTUnit, we
   // shouldn't need to specify them at construction time.
   static std::unique_ptr<ASTUnit> LoadFromCompilerInvocation(
       std::shared_ptr<CompilerInvocation> CI,
       std::shared_ptr<PCHContainerOperations> PCHContainerOps,
       IntrusiveRefCntPtr<DiagnosticsEngine> Diags, FileManager *FileMgr,
       unsigned PrecompilePreambleAfterNParses = 0,
-      bool UserFilesAreVolatile = false);
+      bool UserFilesAreVolatile = false,
+      Optional<std::vector<RemappedFile>> RemappedFiles = None);
 
   /// LoadFromCommandLine - Create an ASTUnit from a vector of command line
   /// arguments, which must specify exactly one source file.
   ///
   /// \param ArgBegin - The beginning of the argument vector.
   ///
   /// \param ArgEnd - The end of the argument vector.
   ///
   /// \param PCHContainerOps - The PCHContainerOperations to use for loading and
   /// creating modules.
   ///
   /// \param Diags - The diagnostics engine to use for reporting errors; its
   /// lifetime is expected to extend past that of the returned ASTUnit.
   ///
   /// \param ResourceFilesPath - The path to the compiler resource files.
   ///
   /// \param ModuleFormat - If provided, uses the specific module format.
   ///
   /// \param ErrAST - If non-null and parsing failed without any AST to return
   /// (e.g. because the PCH could not be loaded), this accepts the ASTUnit
   /// mainly to allow the caller to see the diagnostics.
   ///
-  /// \param VFS - A llvm::vfs::FileSystem to be used for all file accesses.
-  /// Note that preamble is saved to a temporary directory on a RealFileSystem,
-  /// so in order for it to be loaded correctly, VFS should have access to
-  /// it(i.e., be an overlay over RealFileSystem). RealFileSystem will be used
-  /// if \p VFS is nullptr.
-  ///
   // FIXME: Move OnlyLocalDecls, UseBumpAllocator to setters on the ASTUnit, we
   // shouldn't need to specify them at construction time.
   static ASTUnit *LoadFromCommandLine(
       const char **ArgBegin, const char **ArgEnd,
       std::shared_ptr<PCHContainerOperations> PCHContainerOps,
       IntrusiveRefCntPtr<DiagnosticsEngine> Diags, StringRef ResourceFilesPath,
       bool OnlyLocalDecls = false,
       CaptureDiagsKind CaptureDiagnostics = CaptureDiagsKind::None,
-      ArrayRef<RemappedFile> RemappedFiles = None,
+      Optional<std::vector<RemappedFile>> RemappedFiles = None,
       unsigned PrecompilePreambleAfterNParses = 0,
       TranslationUnitKind TUKind = TU_Complete,
       bool CacheCodeCompletionResults = false,
       bool IncludeBriefCommentsInCodeCompletion = false,
       bool AllowPCHWithCompilerErrors = false,
       SkipFunctionBodiesScope SkipFunctionBodies =
           SkipFunctionBodiesScope::None,
       bool SingleFileParse = false, bool UserFilesAreVolatile = false,
public:
   /// Note that preamble is saved to a temporary directory on a RealFileSystem,
   /// so in order for it to be loaded correctly, VFS should give an access to
   /// this(i.e. be an overlay over RealFileSystem).
   /// FileMgr->getVirtualFileSystem() will be used if \p VFS is nullptr.
   ///
   /// \returns True if a failure occurred that causes the ASTUnit not to
   /// contain any translation-unit information, false otherwise.
   bool Reparse(std::shared_ptr<PCHContainerOperations> PCHContainerOps,
-               ArrayRef<RemappedFile> RemappedFiles = None,
+               Optional<std::vector<RemappedFile>> RemappedFiles = None,
                IntrusiveRefCntPtr<llvm::vfs::FileSystem> VFS = nullptr);
 
   /// Free data that will be re-generated on the next parse.
   ///
   /// Preamble-related data is not affected.
   void ResetForParse();
 
   /// Perform code completion at the given file, line, and
public:
   /// for loop) in the code-completion results.
   ///
   /// \param IncludeBriefComments Whether to include brief documentation within
   /// the set of code completions returned.
   ///
   /// FIXME: The Diag, LangOpts, SourceMgr, FileMgr, StoredDiagnostics, and
   /// OwnedBuffers parameters are all disgusting hacks. They will go away.
   void CodeComplete(StringRef File, unsigned Line, unsigned Column,
-                    ArrayRef<RemappedFile> RemappedFiles, bool IncludeMacros,
-                    bool IncludeCodePatterns, bool IncludeBriefComments,
-                    CodeCompleteConsumer &Consumer,
+                    Optional<std::vector<RemappedFile>> RemappedFiles,
+                    bool IncludeMacros, bool IncludeCodePatterns,
+                    bool IncludeBriefComments, CodeCompleteConsumer &Consumer,
                     std::shared_ptr<PCHContainerOperations> PCHContainerOps,
                     DiagnosticsEngine &Diag, LangOptions &LangOpts,
                     SmallVectorImpl<StoredDiagnostic> &StoredDiagnostics,
                     SmallVectorImpl<const llvm::MemoryBuffer *> &OwnedBuffers);
 
   /// Save this translation unit to a file with the given name.
   ///
   /// \returns true if there was a file error or false if the save was
   /// successful.
   bool Save(StringRef File);
 
   /// Serialize this translation unit with the given output stream.
   ///
   /// \returns True if an error occurred, false otherwise.
   bool serialize(raw_ostream &OS);
+
+private:
+  IntrusiveRefCntPtr<llvm::vfs::FileSystem> computeBaseFS(
+      const CompilerInvocation &CI, DiagnosticsEngine &Diags,
+      IntrusiveRefCntPtr<llvm::vfs::FileSystem> MaybeFS = nullptr) const;
+  void initializeBaseFS(IntrusiveRefCntPtr<llvm::vfs::FileSystem> BaseFS);
+  void computeAndInitializeFS(
+      const CompilerInvocation &CI, DiagnosticsEngine &Diags,
+      IntrusiveRefCntPtr<llvm::vfs::FileSystem> MaybeFS = nullptr);
+  void reinitializeBaseFS(
+      IntrusiveRefCntPtr<llvm::vfs::FileSystem> MaybeFS = nullptr);
+  void remapFiles(Optional<std::vector<RemappedFile>> RemappedFiles);
 };
 
 } // namespace clang
 
 #endif // LLVM_CLANG_FRONTEND_ASTUNIT_H

clang/include/clang/Frontend/PrecompiledPreamble.h

public:
   llvm::StringRef getContents() const {
     return {PreambleBytes.data(), PreambleBytes.size()};
   }
 
   /// Check whether PrecompiledPreamble can be reused for the new contents(\p
   /// MainFileBuffer) of the main file.
   bool CanReuse(const CompilerInvocation &Invocation,
                 const llvm::MemoryBufferRef &MainFileBuffer,
-                PreambleBounds Bounds, llvm::vfs::FileSystem &VFS) const;
+                PreambleBounds Bounds, llvm::vfs::FileSystem &VFS,
+                llvm::vfs::FileSystem *OverriddenFilesFS = nullptr) const;
 
   /// Changes options inside \p CI to use PCH from this preamble. Also remaps
   /// main file to \p MainFileBuffer and updates \p VFS to ensure the preamble
   /// is accessible.
   /// Requires that CanReuse() is true.
   /// For in-memory preambles, PrecompiledPreamble instance continues to own the
   /// MemoryBuffer with the Preamble after this method returns. The caller is
   /// responsible for making sure the PrecompiledPreamble instance outlives the

clang/lib/Frontend/ASTUnit.cpp

llvm::CrashRecoveryContextCleanupRegistrar<DiagnosticsEngine,
     DiagCleanup(Diags.get());
 
   ConfigureDiags(Diags, *AST, CaptureDiagnostics);
 
   AST->LangOpts = std::make_shared<LangOptions>();
   AST->OnlyLocalDecls = OnlyLocalDecls;
   AST->CaptureDiagnostics = CaptureDiagnostics;
   AST->Diagnostics = Diags;
-  IntrusiveRefCntPtr<llvm::vfs::FileSystem> VFS =
-      llvm::vfs::getRealFileSystem();
-  AST->FileMgr = new FileManager(FileSystemOpts, VFS);
+  AST->initializeBaseFS(llvm::vfs::getRealFileSystem());
+  AST->FileMgr = new FileManager(FileSystemOpts, AST->FS);
   AST->UserFilesAreVolatile = UserFilesAreVolatile;
   AST->SourceMgr = new SourceManager(AST->getDiagnostics(),
                                      AST->getFileManager(),
                                      UserFilesAreVolatile);
   AST->ModuleCache = new InMemoryModuleCache;
   AST->HSOpts = std::make_shared<HeaderSearchOptions>();
   AST->HSOpts->ModuleFormat = std::string(PCHContainerRdr.getFormat());
   AST->HeaderInfo.reset(new HeaderSearch(AST->HSOpts,

 }
 
 /// Parse the source file into a translation unit using the given compiler
 /// invocation, replacing the current translation unit.
 ///
 /// \returns True if a failure occurred that causes the ASTUnit not to
 /// contain any translation-unit information, false otherwise.
 bool ASTUnit::Parse(std::shared_ptr<PCHContainerOperations> PCHContainerOps,
-                    std::unique_ptr<llvm::MemoryBuffer> OverrideMainBuffer,
-                    IntrusiveRefCntPtr<llvm::vfs::FileSystem> VFS) {
+                    std::unique_ptr<llvm::MemoryBuffer> OverrideMainBuffer) {
   if (!Invocation)
     return true;
 
-  if (VFS && FileMgr)
-    assert(VFS == &FileMgr->getVirtualFileSystem() &&
-           "VFS passed to Parse and VFS in FileMgr are different");
+  assert((!FileMgr || &FileMgr->getVirtualFileSystem() == FS) &&
+         "ASTUnit has a FileManager without initializing FS");
 
   auto CCInvocation = std::make_shared<CompilerInvocation>(*Invocation);
   if (OverrideMainBuffer) {
     assert(Preamble &&
            "No preamble was built, but OverrideMainBuffer is not null");
-    Preamble->AddImplicitPreamble(*CCInvocation, VFS, OverrideMainBuffer.get());
+    Preamble->AddImplicitPreamble(*CCInvocation, FS, OverrideMainBuffer.get());
     // VFS may have changed...
+    if (FileMgr && &FileMgr->getVirtualFileSystem() != FS)
+      FileMgr->setVirtualFileSystem(FS);
   }
 
   // Create the compiler instance to use for building the AST.
   std::unique_ptr<CompilerInstance> Clang(
       new CompilerInstance(std::move(PCHContainerOps)));
 
   // Clean up on error, disengage it if the function returns successfully.
   auto CleanOnError = llvm::make_scope_exit([&]() {
     // Remove the overridden buffer we used for the preamble.
     SavedMainFileBuffer = nullptr;
 
     // Keep the ownership of the data in the ASTUnit because the client may
     // want to see the diagnostics.
     transferASTDataFromCompilerInstance(*Clang);
     FailedParseDiagnostics.swap(StoredDiagnostics);
     StoredDiagnostics.clear();
     NumStoredDiagnosticsFromDriver = 0;
   });
 
   // Ensure that Clang has a FileManager with the right VFS, which may have
   // changed above in AddImplicitPreamble.  If VFS is nullptr, rely on
   // createFileManager to create one.
-  if (VFS && FileMgr && &FileMgr->getVirtualFileSystem() == VFS)
+  if (FileMgr && &FileMgr->getVirtualFileSystem() == FS)
     Clang->setFileManager(&*FileMgr);
   else
-    FileMgr = Clang->createFileManager(std::move(VFS));
+    FileMgr = Clang->createFileManager(FS);
 
   // Recover resources if we crash before exiting this method.
   llvm::CrashRecoveryContextCleanupRegistrar<CompilerInstance>
     CICleanup(Clang.get());
 
   Clang->setInvocation(CCInvocation);
   OriginalSourceFile =
       std::string(Clang->getFrontendOpts().Inputs[0].getFile());

 /// can occur within the preamble.
 ///
 /// \returns If the precompiled preamble can be used, returns a newly-allocated
 /// buffer that should be used in place of the main file when doing so.
 /// Otherwise, returns a NULL pointer.
 std::unique_ptr<llvm::MemoryBuffer>
 ASTUnit::getMainBufferWithPrecompiledPreamble(
     std::shared_ptr<PCHContainerOperations> PCHContainerOps,
-    CompilerInvocation &PreambleInvocationIn,
-    IntrusiveRefCntPtr<llvm::vfs::FileSystem> VFS, bool AllowRebuild,
+    CompilerInvocation &PreambleInvocationIn, bool AllowRebuild,
     unsigned MaxLines) {
   auto MainFilePath =
       PreambleInvocationIn.getFrontendOpts().Inputs[0].getFile();
   std::unique_ptr<llvm::MemoryBuffer> MainFileBuffer =
-      getBufferForFileHandlingRemapping(PreambleInvocationIn, VFS.get(),
+      getBufferForFileHandlingRemapping(PreambleInvocationIn, FS.get(),
                                         MainFilePath, UserFilesAreVolatile);
   if (!MainFileBuffer)
     return nullptr;
 
   PreambleBounds Bounds = ComputePreambleBounds(
       *PreambleInvocationIn.getLangOpts(), *MainFileBuffer, MaxLines);
   if (!Bounds.Size)
     return nullptr;
 
   if (Preamble) {
     if (Preamble->CanReuse(PreambleInvocationIn, *MainFileBuffer, Bounds,
-                           *VFS)) {
+                           *FS, &*RemappedFilesFS)) {
       // Okay! We can re-use the precompiled preamble.
 
       // Set the state of the diagnostic object to mimic its state
       // after parsing the preamble.
       getDiagnostics().Reset();
       ProcessWarningOptions(getDiagnostics(),
                             PreambleInvocationIn.getDiagnosticOpts());
       getDiagnostics().setNumWarnings(NumWarningsInPreamble);
ASTUnitPreambleCallbacks Callbacks;
     PreambleTimer.setOutput("Precompiling preamble");
 
     const bool PreviousSkipFunctionBodies =
         PreambleInvocationIn.getFrontendOpts().SkipFunctionBodies;
     if (SkipFunctionBodies == SkipFunctionBodiesScope::Preamble)
       PreambleInvocationIn.getFrontendOpts().SkipFunctionBodies = true;
 
     llvm::ErrorOr<PrecompiledPreamble> NewPreamble = PrecompiledPreamble::Build(
-        PreambleInvocationIn, MainFileBuffer.get(), Bounds, *Diagnostics, VFS,
-        PCHContainerOps, /*StoreInMemory=*/false, Callbacks);
+        PreambleInvocationIn, MainFileBuffer.get(), Bounds, *Diagnostics,
+        FS.get(), PCHContainerOps, /*StoreInMemory=*/false, Callbacks);
 
     PreambleInvocationIn.getFrontendOpts().SkipFunctionBodies =
         PreviousSkipFunctionBodies;
 
     if (NewPreamble) {
       Preamble = std::move(*NewPreamble);
       PreambleRebuildCountdown = 1;
     } else {

 
 std::unique_ptr<ASTUnit>
 ASTUnit::create(std::shared_ptr<CompilerInvocation> CI,
                 IntrusiveRefCntPtr<DiagnosticsEngine> Diags,
                 CaptureDiagsKind CaptureDiagnostics,
                 bool UserFilesAreVolatile) {
   std::unique_ptr<ASTUnit> AST(new ASTUnit(false));
   ConfigureDiags(Diags, *AST, CaptureDiagnostics);
-  IntrusiveRefCntPtr<llvm::vfs::FileSystem> VFS =
-      createVFSFromCompilerInvocation(*CI, *Diags);
   AST->Diagnostics = Diags;
   AST->FileSystemOpts = CI->getFileSystemOpts();
   AST->Invocation = std::move(CI);
-  AST->FileMgr = new FileManager(AST->FileSystemOpts, VFS);
+  AST->computeAndInitializeFS(*AST->Invocation, *Diags);
+  AST->FileMgr = new FileManager(AST->FileSystemOpts, AST->FS);
   AST->UserFilesAreVolatile = UserFilesAreVolatile;
   AST->SourceMgr = new SourceManager(AST->getDiagnostics(), *AST->FileMgr,
                                      UserFilesAreVolatile);
   AST->ModuleCache = new InMemoryModuleCache;
 
   return AST;
 }
 
ASTUnit *ASTUnit::LoadFromCompilerInvocationAction(
   if (OwnAST)
     return OwnAST.release();
   else
     return AST;
 }
 
 bool ASTUnit::LoadFromCompilerInvocation(
     std::shared_ptr<PCHContainerOperations> PCHContainerOps,
-    unsigned PrecompilePreambleAfterNParses,
-    IntrusiveRefCntPtr<llvm::vfs::FileSystem> VFS) {
+    unsigned PrecompilePreambleAfterNParses) {
   if (!Invocation)
     return true;
 
-  assert(VFS && "VFS is null");
-
   // We'll manage file buffers ourselves.
   Invocation->getPreprocessorOpts().RetainRemappedFileBuffers = true;
   Invocation->getFrontendOpts().DisableFree = false;
   getDiagnostics().Reset();
   ProcessWarningOptions(getDiagnostics(), Invocation->getDiagnosticOpts());
 
   std::unique_ptr<llvm::MemoryBuffer> OverrideMainBuffer;
   if (PrecompilePreambleAfterNParses > 0) {
     PreambleRebuildCountdown = PrecompilePreambleAfterNParses;
     OverrideMainBuffer =
-        getMainBufferWithPrecompiledPreamble(PCHContainerOps, *Invocation, VFS);
+        getMainBufferWithPrecompiledPreamble(PCHContainerOps, *Invocation);
     getDiagnostics().Reset();
     ProcessWarningOptions(getDiagnostics(), Invocation->getDiagnosticOpts());
   }
 
   SimpleTimer ParsingTimer(WantTiming);
   ParsingTimer.setOutput("Parsing " + getMainFileName());
 
   // Recover resources if we crash before exiting this method.
   llvm::CrashRecoveryContextCleanupRegistrar<llvm::MemoryBuffer>
     MemBufferCleanup(OverrideMainBuffer.get());
 
-  return Parse(std::move(PCHContainerOps), std::move(OverrideMainBuffer), VFS);
+  return Parse(std::move(PCHContainerOps), std::move(OverrideMainBuffer));
 }
 
 std::unique_ptr<ASTUnit> ASTUnit::LoadFromCompilerInvocation(
     std::shared_ptr<CompilerInvocation> CI,
     std::shared_ptr<PCHContainerOperations> PCHContainerOps,
     IntrusiveRefCntPtr<DiagnosticsEngine> Diags, FileManager *FileMgr,
-    unsigned PrecompilePreambleAfterNParses,
-    bool UserFilesAreVolatile) {
+    unsigned PrecompilePreambleAfterNParses, bool UserFilesAreVolatile,
+    Optional<std::vector<RemappedFile>> RemappedFiles) {
   // Create the AST unit.
   std::unique_ptr<ASTUnit> AST(new ASTUnit(false));
   ConfigureDiags(Diags, *AST, CaptureDiagsKind::None);
   AST->Diagnostics = Diags;
   AST->OnlyLocalDecls = false;
   AST->CaptureDiagnostics = CaptureDiagsKind::None;
   AST->TUKind = TU_Complete;
   AST->ShouldCacheCodeCompletionResults = false;
   AST->IncludeBriefCommentsInCodeCompletion = false;
   AST->Invocation = std::move(CI);
   AST->FileSystemOpts = FileMgr->getFileSystemOpts();
   AST->FileMgr = FileMgr;
   AST->UserFilesAreVolatile = UserFilesAreVolatile;
 
+  // If we have a file manager, this will update its VFS.
+  AST->computeAndInitializeFS(*AST->Invocation, *Diags);
+
+  // Override any files that need remapping.
+  AST->remapFiles(std::move(RemappedFiles));
+
   // Recover resources if we crash before exiting this method.
   llvm::CrashRecoveryContextCleanupRegistrar<ASTUnit>
     ASTUnitCleanup(AST.get());
   llvm::CrashRecoveryContextCleanupRegistrar<DiagnosticsEngine,
     llvm::CrashRecoveryContextReleaseRefCleanup<DiagnosticsEngine>>
     DiagCleanup(Diags.get());
 
   if (AST->LoadFromCompilerInvocation(std::move(PCHContainerOps),
-                                      PrecompilePreambleAfterNParses,
-                                      &AST->FileMgr->getVirtualFileSystem()))
+                                      PrecompilePreambleAfterNParses))
     return nullptr;
   return AST;
 }
 
 ASTUnit *ASTUnit::LoadFromCommandLine(
     const char **ArgBegin, const char **ArgEnd,
     std::shared_ptr<PCHContainerOperations> PCHContainerOps,
     IntrusiveRefCntPtr<DiagnosticsEngine> Diags, StringRef ResourceFilesPath,
     bool OnlyLocalDecls, CaptureDiagsKind CaptureDiagnostics,
-    ArrayRef<RemappedFile> RemappedFiles,
+    Optional<std::vector<RemappedFile>> RemappedFiles,
     unsigned PrecompilePreambleAfterNParses, TranslationUnitKind TUKind,
     bool CacheCodeCompletionResults, bool IncludeBriefCommentsInCodeCompletion,
     bool AllowPCHWithCompilerErrors, SkipFunctionBodiesScope SkipFunctionBodies,
     bool SingleFileParse, bool UserFilesAreVolatile, bool ForSerialization,
     bool RetainExcludedConditionalBlocks,
     llvm::Optional<StringRef> ModuleFormat, std::unique_ptr<ASTUnit> *ErrAST) {
   assert(Diags.get() && "no DiagnosticsEngine was provided");
 
   SmallVector<StoredDiagnostic, 4> StoredDiagnostics;
 
   std::shared_ptr<CompilerInvocation> CI;
 
   {
     CaptureDroppedDiagnostics Capture(CaptureDiagnostics, *Diags,
                                       &StoredDiagnostics, nullptr);
 
     CI = createInvocationFromCommandLine(llvm::makeArrayRef(ArgBegin, ArgEnd),
                                          Diags);
     if (!CI)
       return nullptr;
   }
 
-  // Override any files that need remapping
-  for (const auto &RemappedFile : RemappedFiles) {
-    CI->getPreprocessorOpts().addRemappedFile(RemappedFile.first,
-                                              RemappedFile.second);
-  }
   PreprocessorOptions &PPOpts = CI->getPreprocessorOpts();
   PPOpts.RemappedFilesKeepOriginalName = true;
   PPOpts.AllowPCHWithCompilerErrors = AllowPCHWithCompilerErrors;
   PPOpts.SingleFileParseMode = SingleFileParse;
   PPOpts.RetainExcludedConditionalBlocks = RetainExcludedConditionalBlocks;
 
   // Override the resources path.
   CI->getHeaderSearchOpts().ResourceDir = std::string(ResourceFilesPath);
 
   CI->getFrontendOpts().SkipFunctionBodies =
       SkipFunctionBodies == SkipFunctionBodiesScope::PreambleAndMainFile;
 
   if (ModuleFormat)
     CI->getHeaderSearchOpts().ModuleFormat =
         std::string(ModuleFormat.getValue());
 
   // Create the AST unit.
   std::unique_ptr<ASTUnit> AST;
   AST.reset(new ASTUnit(false));
+
+  // Override any files that need remapping
+  AST->initializeBaseFS(createVFSFromCompilerInvocation(*CI, *Diags));
+  AST->remapFiles(std::move(RemappedFiles));
+
   AST->NumStoredDiagnosticsFromDriver = StoredDiagnostics.size();
   AST->StoredDiagnostics.swap(StoredDiagnostics);
   ConfigureDiags(Diags, *AST, CaptureDiagnostics);
   AST->Diagnostics = Diags;
   AST->FileSystemOpts = CI->getFileSystemOpts();
-  IntrusiveRefCntPtr<llvm::vfs::FileSystem> VFS =
-      llvm::vfs::getRealFileSystem();
-  VFS = createVFSFromCompilerInvocation(*CI, *Diags, VFS);
-  AST->FileMgr = new FileManager(AST->FileSystemOpts, VFS);
+  AST->FileMgr = new FileManager(AST->FileSystemOpts, AST->FS);
   AST->ModuleCache = new InMemoryModuleCache;
   AST->OnlyLocalDecls = OnlyLocalDecls;
   AST->CaptureDiagnostics = CaptureDiagnostics;
   AST->TUKind = TUKind;
   AST->ShouldCacheCodeCompletionResults = CacheCodeCompletionResults;
   AST->IncludeBriefCommentsInCodeCompletion
     = IncludeBriefCommentsInCodeCompletion;
   AST->UserFilesAreVolatile = UserFilesAreVolatile;
   AST->Invocation = CI;
   AST->SkipFunctionBodies = SkipFunctionBodies;
   if (ForSerialization)
     AST->WriterData.reset(new ASTWriterData(*AST->ModuleCache));
   // Zero out now to ease cleanup during crash recovery.
   CI = nullptr;
   Diags = nullptr;
 
   // Recover resources if we crash before exiting this method.
   llvm::CrashRecoveryContextCleanupRegistrar<ASTUnit>
     ASTUnitCleanup(AST.get());
 
   if (AST->LoadFromCompilerInvocation(std::move(PCHContainerOps),
-                                      PrecompilePreambleAfterNParses,
-                                      VFS)) {
+                                      PrecompilePreambleAfterNParses)) {
     // Some error occurred, if caller wants to examine diagnostics, pass it the
     // ASTUnit.
     if (ErrAST) {
       AST->StoredDiagnostics.swap(AST->FailedParseDiagnostics);
       ErrAST->swap(AST);
     }
     return nullptr;
   }
 
   return AST.release();
 }
 
 bool ASTUnit::Reparse(std::shared_ptr<PCHContainerOperations> PCHContainerOps,
-                      ArrayRef<RemappedFile> RemappedFiles,
+                      Optional<std::vector<RemappedFile>> RemappedFiles,
                       IntrusiveRefCntPtr<llvm::vfs::FileSystem> VFS) {
   if (!Invocation)
     return true;
 
-  if (!VFS) {
-    assert(FileMgr && "FileMgr is null on Reparse call");
-    VFS = &FileMgr->getVirtualFileSystem();
-  }
-
   clearFileLevelDecls();
 
   SimpleTimer ParsingTimer(WantTiming);
   ParsingTimer.setOutput("Reparsing " + getMainFileName());
 
   // Remap files.
+  reinitializeBaseFS(std::move(VFS));
+  remapFiles(std::move(RemappedFiles));
+
+  // Clear out buffers in the Invocation.
   PreprocessorOptions &PPOpts = Invocation->getPreprocessorOpts();
   for (const auto &RB : PPOpts.RemappedFileBuffers)
     delete RB.second;
 
   Invocation->getPreprocessorOpts().clearRemappedFiles();
-  for (const auto &RemappedFile : RemappedFiles) {
-    Invocation->getPreprocessorOpts().addRemappedFile(RemappedFile.first,
-                                                      RemappedFile.second);
-  }
 
   // If we have a preamble file lying around, or if we might try to
   // build a precompiled preamble, do so now.
   std::unique_ptr<llvm::MemoryBuffer> OverrideMainBuffer;
   if (Preamble || PreambleRebuildCountdown > 0)
     OverrideMainBuffer =
-        getMainBufferWithPrecompiledPreamble(PCHContainerOps, *Invocation, VFS);
+        getMainBufferWithPrecompiledPreamble(PCHContainerOps, *Invocation);
 
   // Clear out the diagnostics state.
   FileMgr.reset();
   getDiagnostics().Reset();
   ProcessWarningOptions(getDiagnostics(), Invocation->getDiagnosticOpts());
   if (OverrideMainBuffer)
     getDiagnostics().setNumWarnings(NumWarningsInPreamble);
 
   // Parse the sources
   bool Result =
-      Parse(std::move(PCHContainerOps), std::move(OverrideMainBuffer), VFS);
+      Parse(std::move(PCHContainerOps), std::move(OverrideMainBuffer));
 
   // If we're caching global code-completion results, and the top-level
   // declarations have changed, clear out the code-completion cache.
   if (!Result && ShouldCacheCodeCompletionResults &&
       CurrentTopLevelHashValue != CompletionCacheTopLevelHashValue)
     CacheCodeCompletionResults();
 
   // We now need to clear out the completion info related to this translation
void AugmentedCodeCompleteConsumer::ProcessCodeCompleteResults(Sema &S,
   }
 
   Next.ProcessCodeCompleteResults(S, Context, AllResults.data(),
                                   AllResults.size());
 }
 
 void ASTUnit::CodeComplete(
     StringRef File, unsigned Line, unsigned Column,
-    ArrayRef<RemappedFile> RemappedFiles, bool IncludeMacros,
+    Optional<std::vector<RemappedFile>> RemappedFiles, bool IncludeMacros,
     bool IncludeCodePatterns, bool IncludeBriefComments,
     CodeCompleteConsumer &Consumer,
     std::shared_ptr<PCHContainerOperations> PCHContainerOps,
     DiagnosticsEngine &Diag, LangOptions &LangOpts,
     SmallVectorImpl<StoredDiagnostic> &StoredDiagnostics,
     SmallVectorImpl<const llvm::MemoryBuffer *> &OwnedBuffers) {
   if (!Invocation)
     return;
assert(Clang->getFrontendOpts().Inputs.size() == 1 &&
          "Invocation must have exactly one source file!");
   assert(Clang->getFrontendOpts().Inputs[0].getKind().getFormat() ==
              InputKind::Source &&
          "FIXME: AST inputs not yet supported here!");
   assert(Clang->getFrontendOpts().Inputs[0].getKind().getLanguage() !=
              Language::LLVM_IR &&
          "IR inputs not support here!");
 
+  // Remap files.
+  reinitializeBaseFS();
+  remapFiles(std::move(RemappedFiles));
+
   // Initialize file and source managers up front so that the caller can access
   // them after.
   Clang->createFileManager(FS);
+  Clang->createSourceManager(Clang->getFileManager());
   FileMgr = &Clang->getFileManager();
-  Clang->createSourceManager(*FileMgr);
   SourceMgr = &Clang->getSourceManager();
 
-  // Remap files.
-  PreprocessorOpts.clearRemappedFiles();
+  // Clear out buffers in the invocation and prepare for any overridden buffers
+  // there by requesting they be retained.
   PreprocessorOpts.RetainRemappedFileBuffers = true;
-  for (const auto &RemappedFile : RemappedFiles) {
-    PreprocessorOpts.addRemappedFile(RemappedFile.first, RemappedFile.second);
-    OwnedBuffers.push_back(RemappedFile.second);
-  }
+  PreprocessorOpts.clearRemappedFiles();
 
   // Use the code completion consumer we were given, but adding any cached
   // code-completion results.
   AugmentedCodeCompleteConsumer *AugmentedConsumer
     = new AugmentedCodeCompleteConsumer(*this, Consumer, CodeCompleteOpts);
   Clang->setCodeCompletionConsumer(AugmentedConsumer);
 
   auto getUniqueID =
-      [&FileMgr](StringRef Filename) -> Optional<llvm::sys::fs::UniqueID> {
-    if (auto Status = FileMgr.getVirtualFileSystem().status(Filename))
+      [this](StringRef Filename) -> Optional<llvm::sys::fs::UniqueID> {
+    if (auto Status = BaseFS->status(Filename))
       return Status->getUniqueID();
     return None;
   };
 
   auto hasSameUniqueID = [getUniqueID](StringRef LHS, StringRef RHS) {
     if (LHS == RHS)
       return true;
     if (auto LHSID = getUniqueID(LHS))
       if (auto RHSID = getUniqueID(RHS))
         return *LHSID == *RHSID;
     return false;
   };
 
   // If we have a precompiled preamble, try to use it. We only allow
   // the use of the precompiled preamble if we're if the completion
   // point is within the main file, after the end of the precompiled
   // preamble.
   std::unique_ptr<llvm::MemoryBuffer> OverrideMainBuffer;
   if (Preamble && Line > 1 && hasSameUniqueID(File, OriginalSourceFile)) {
     OverrideMainBuffer = getMainBufferWithPrecompiledPreamble(
-        PCHContainerOps, Inv, &FileMgr.getVirtualFileSystem(), false, Line - 1);
+        PCHContainerOps, Inv, false, Line - 1);
   }
 
   // If the main file has been overridden due to the use of a preamble,
   // make that override happen and introduce the preamble.
   if (OverrideMainBuffer) {
     assert(Preamble &&
            "No preamble was built, but OverrideMainBuffer is not null");
 
-    IntrusiveRefCntPtr<llvm::vfs::FileSystem> VFS =
-        &FileMgr.getVirtualFileSystem();
-    Preamble->AddImplicitPreamble(Clang->getInvocation(), VFS,
+    Preamble->AddImplicitPreamble(Clang->getInvocation(), FS,
                                   OverrideMainBuffer.get());
     // FIXME: there is no way to update VFS if it was changed by
     // AddImplicitPreamble as FileMgr is accepted as a parameter by this method.
     // We use on-disk preambles instead and rely on FileMgr's VFS to ensure the
     // PCH files are always readable.
     OwnedBuffers.push_back(OverrideMainBuffer.release());
   } else {
     PreprocessorOpts.PrecompiledPreambleBytes.first = 0;
     PreprocessorOpts.PrecompiledPreambleBytes.second = false;
   }
 
   // Disable the preprocessing record if modules are not enabled.
   if (!Clang->getLangOpts().Modules)
     PreprocessorOpts.DetailedRecord = false;
 
   std::unique_ptr<SyntaxOnlyAction> Act;
   Act.reset(new SyntaxOnlyAction);
   if (Act->BeginSourceFile(*Clang.get(), Clang->getFrontendOpts().Inputs[0])) {
     if (llvm::Error Err = Act->Execute()) {
       consumeError(std::move(Err)); // FIXME this drops errors on the floor.
     }
+
+    transferASTDataFromCompilerInstance(*Clang);
+
     Act->EndSourceFile();
   }
 }
 
 bool ASTUnit::Save(StringRef File) {
   if (HadModuleLoaderFatalFailure)
     return true;
 

 #else // NDEBUG
 
 ASTUnit::ConcurrencyState::ConcurrencyState() { Mutex = nullptr; }
 ASTUnit::ConcurrencyState::~ConcurrencyState() {}
 void ASTUnit::ConcurrencyState::start() {}
 void ASTUnit::ConcurrencyState::finish() {}
 
 #endif // NDEBUG
+
+void ASTUnit::initializeBaseFS(IntrusiveRefCntPtr<llvm::vfs::FileSystem> BaseFS) {
+  assert(BaseFS && "Given empty base filesystem?");
+  if (this->BaseFS == BaseFS || this->FS == BaseFS)
+    return;
+
+  llvm::IntrusiveRefCntPtr<llvm::vfs::OverlayFileSystem> OverlayFS(
+      new llvm::vfs::OverlayFileSystem(BaseFS));
+  RemappedFilesFS = new llvm::vfs::InMemoryFileSystem;
+  OverlayFS->pushOverlay(RemappedFilesFS);
+  FS = std::move(OverlayFS);
+  this->BaseFS = std::move(BaseFS);
+  if (FileMgr)
+    FileMgr->setVirtualFileSystem(FS);
+}
+
+void ASTUnit::reinitializeBaseFS(
+    IntrusiveRefCntPtr<llvm::vfs::FileSystem> MaybeFS) {
+  assert(FS && "Resetting without initializing?");
+  assert(BaseFS && "Resetting without initializing?");
+  assert(RemappedFilesFS && "Resetting without initializing?");
+  if (!MaybeFS)
+    MaybeFS = std::move(BaseFS);
+  else
+    BaseFS.reset();
+
+  RemappedFilesFS.reset();
+  FS.reset();
+
+  // Reinitialize.
+  initializeBaseFS(std::move(MaybeFS));
+}
+
+IntrusiveRefCntPtr<llvm::vfs::FileSystem> ASTUnit::computeBaseFS(
+    const CompilerInvocation &CI, DiagnosticsEngine &Diags,
+    IntrusiveRefCntPtr<llvm::vfs::FileSystem> MaybeFS) const {
+  if (MaybeFS)
+    return FS == MaybeFS ? BaseFS : MaybeFS;
+  if (!FileMgr)
+    return FS ? BaseFS : createVFSFromCompilerInvocation(CI, Diags);
+
+  llvm::vfs::FileSystem &FMFS = FileMgr->getVirtualFileSystem();
+  return FS == &FMFS ? BaseFS : &FMFS;
+}
+
+void ASTUnit::computeAndInitializeFS(
+    const CompilerInvocation &CI, DiagnosticsEngine &Diags,
+    IntrusiveRefCntPtr<llvm::vfs::FileSystem> MaybeFS) {
+  initializeBaseFS(computeBaseFS(CI, Diags, std::move(MaybeFS)));
+}
+
+void ASTUnit::remapFiles(Optional<std::vector<RemappedFile>> RemappedFiles) {
+  if (!RemappedFiles || RemappedFiles->empty())
+    return;
+
+  // Add files to the in-memory filesystem in reverse order to ensure the last
+  // mapping "wins". This works because InMemoryFileSystem::addFile ignores
+  // subsequent calls.
+  for (auto &File : llvm::reverse(*RemappedFiles))
+    RemappedFilesFS->addFile(File.first, 0, std::move(File.second));
+}

clang/lib/Frontend/PrecompiledPreamble.cpp

if (Clang->getFrontendOpts().Inputs.size() != 1 ||
           Language::LLVM_IR) {
     return BuildPreambleError::BadInputs;
   }
 
   // Clear out old caches and data.
   Diagnostics.Reset();
   ProcessWarningOptions(Diagnostics, Clang->getDiagnosticOpts());
 
+  // FIXME: should this be deleted? Can we trust that the caller has done this?
   VFS =
       createVFSFromCompilerInvocation(Clang->getInvocation(), Diagnostics, VFS);
 
   // Create a file manager object to provide access to and cache the filesystem.
   Clang->setFileManager(new FileManager(Clang->getFileSystemOpts(), VFS));
 
   // Create the source manager.
   Clang->setSourceManager(
       new SourceManager(Diagnostics, Clang->getFileManager()));
 
   auto PreambleDepCollector = std::make_shared<PreambleDependencyCollector>();
   Clang->addDependencyCollector(PreambleDepCollector);
 
   Clang->getLangOpts().CompilingPCH = true;
 
   // Remap the main source file to the preamble buffer.
+  //
+  // Note: the buffer needs to be overridden in the SourceManager so that the
+  // preamble is serialized the AST file.
   StringRef MainFilePath = FrontendOpts.Inputs[0].getFile();
   auto PreambleInputBuffer = llvm::MemoryBuffer::getMemBufferCopy(
       MainFileBuffer->getBuffer().slice(0, Bounds.Size), MainFilePath);
   if (PreprocessorOpts.RetainRemappedFileBuffers) {
     // MainFileBuffer will be deleted by unique_ptr after leaving the method.
     PreprocessorOpts.addRemappedFile(MainFilePath, PreambleInputBuffer.get());
   } else {
     // In that case, remapped buffer will be deleted by CompilerInstance on
case PCHStorage::Kind::TempFile: {
     assert(Result <= std::numeric_limits<std::size_t>::max() &&
            "file size did not fit into size_t");
     return Result;
   }
   }
   llvm_unreachable("Unhandled storage kind");
 }
 
-bool PrecompiledPreamble::CanReuse(const CompilerInvocation &Invocation,
-                                   const llvm::MemoryBufferRef &MainFileBuffer,
-                                   PreambleBounds Bounds,
-                                   llvm::vfs::FileSystem &VFS) const {
-
+bool PrecompiledPreamble::CanReuse(
+    const CompilerInvocation &Invocation,
+    const llvm::MemoryBufferRef &MainFileBuffer, PreambleBounds Bounds,
+    llvm::vfs::FileSystem &VFS,
+    llvm::vfs::FileSystem *OverriddenFilesFS) const {
   assert(
       Bounds.Size <= MainFileBuffer.getBufferSize() &&
       "Buffer is too large. Bounds were calculated from a different buffer?");
 
   auto PreambleInvocation = std::make_shared<CompilerInvocation>(Invocation);
   PreprocessorOptions &PreprocessorOpts =
       PreambleInvocation->getPreprocessorOpts();
 
for (const auto &RB : PreprocessorOpts.RemappedFileBuffers) {
 
     llvm::SmallString<128> MappedPath(RB.first);
     if (!VFS.makeAbsolute(MappedPath))
       OverriddenAbsPaths.insert(MappedPath);
   }
 
   // Check whether anything has changed.
   for (const auto &F : FilesInPreamble) {
+    if (OverriddenFilesFS) {
+      std::unique_ptr<llvm::MemoryBuffer> Buffer;
+      if (moveOnNoError(OverriddenFilesFS->getBufferForFile(F.first()),
+                        Buffer)) {
+        if (PreambleFileHash::createForMemoryBuffer(*Buffer) != F.second)
+          return false;
+        continue;
+      }
+    }
     auto OverridenFileBuffer = OverridenFileBuffers.find(F.first());
     if (OverridenFileBuffer != OverridenFileBuffers.end()) {
       // The file's buffer was remapped and the file was not found in VFS.
       // Check whether it matches up with the previous mapping.
       if (OverridenFileBuffer->second != F.second)
         return false;
       continue;
     }

 void PrecompiledPreamble::configurePreamble(
     PreambleBounds Bounds, CompilerInvocation &CI,
     IntrusiveRefCntPtr<llvm::vfs::FileSystem> &VFS,
     llvm::MemoryBuffer *MainFileBuffer) const {
   assert(VFS);
 
   auto &PreprocessorOpts = CI.getPreprocessorOpts();
 
-  // Remap main file to point to MainFileBuffer.
+  // Remap main file to point to MainFileBuffer (instead of the preamble).
   auto MainFilePath = CI.getFrontendOpts().Inputs[0].getFile();
   PreprocessorOpts.addRemappedFile(MainFilePath, MainFileBuffer);
 
   // Configure ImpicitPCHInclude.
   PreprocessorOpts.PrecompiledPreambleBytes.first = Bounds.Size;
   PreprocessorOpts.PrecompiledPreambleBytes.second =
       Bounds.PreambleEndsAtStartOfLine;
   PreprocessorOpts.DisablePCHValidation = true;

clang/tools/libclang/CIndex.cpp

clang_parseTranslationUnit_Impl(CXIndex CIdx, const char *source_filename,
 
   std::unique_ptr<std::vector<ASTUnit::RemappedFile>> RemappedFiles(
       new std::vector<ASTUnit::RemappedFile>());
 
   // Recover resources if we crash before exiting this function.
   llvm::CrashRecoveryContextCleanupRegistrar<std::vector<ASTUnit::RemappedFile>>
       RemappedCleanup(RemappedFiles.get());
 
-  for (auto &UF : unsaved_files) {
-    std::unique_ptr<llvm::MemoryBuffer> MB =
-        llvm::MemoryBuffer::getMemBufferCopy(getContents(UF), UF.Filename);
-    RemappedFiles->push_back(std::make_pair(UF.Filename, MB.release()));
-  }
+  for (auto &UF : unsaved_files)
+    RemappedFiles->push_back(std::make_pair(
+        UF.Filename,
+        llvm::MemoryBuffer::getMemBufferCopy(getContents(UF), UF.Filename)));
 
   std::unique_ptr<std::vector<const char *>> Args(
       new std::vector<const char *>());
 
   // Recover resources if we crash before exiting this method.
   llvm::CrashRecoveryContextCleanupRegistrar<std::vector<const char *>>
       ArgsCleanup(Args.get());
 
clang_parseTranslationUnit_Impl(CXIndex CIdx, const char *source_filename,
   LibclangInvocationReporter InvocationReporter(
       *CXXIdx, LibclangInvocationReporter::OperationKind::ParseOperation,
       options, llvm::makeArrayRef(*Args), /*InvocationArgs=*/None,
       unsaved_files);
   std::unique_ptr<ASTUnit> Unit(ASTUnit::LoadFromCommandLine(
       Args->data(), Args->data() + Args->size(),
       CXXIdx->getPCHContainerOperations(), Diags,
       CXXIdx->getClangResourcesPath(), CXXIdx->getOnlyLocalDecls(),
-      CaptureDiagnostics, *RemappedFiles.get(), PrecompilePreambleAfterNParses,
-      TUKind, CacheCodeCompletionResults, IncludeBriefCommentsInCodeCompletion,
+      CaptureDiagnostics, std::move(*RemappedFiles),
+      PrecompilePreambleAfterNParses, TUKind, CacheCodeCompletionResults,
+      IncludeBriefCommentsInCodeCompletion,
       /*AllowPCHWithCompilerErrors=*/true, SkipFunctionBodies, SingleFileParse,
       /*UserFilesAreVolatile=*/true, ForSerialization, RetainExcludedCB,
       CXXIdx->getPCHContainerOperations()->getRawReader().getFormat(),
       &ErrUnit));
 
   // Early failures in LoadFromCommandLine may return with ErrUnit unset.
   if (!Unit && !ErrUnit)
     return CXError_ASTReadError;
clang_reparseTranslationUnit_Impl(CXTranslationUnit TU,
 
   std::unique_ptr<std::vector<ASTUnit::RemappedFile>> RemappedFiles(
       new std::vector<ASTUnit::RemappedFile>());
 
   // Recover resources if we crash before exiting this function.
   llvm::CrashRecoveryContextCleanupRegistrar<std::vector<ASTUnit::RemappedFile>>
       RemappedCleanup(RemappedFiles.get());
 
-  for (auto &UF : unsaved_files) {
-    std::unique_ptr<llvm::MemoryBuffer> MB =
-        llvm::MemoryBuffer::getMemBufferCopy(getContents(UF), UF.Filename);
-    RemappedFiles->push_back(std::make_pair(UF.Filename, MB.release()));
-  }
+  for (auto &UF : unsaved_files)
+    RemappedFiles->push_back(std::make_pair(
+        UF.Filename,
+        llvm::MemoryBuffer::getMemBufferCopy(getContents(UF), UF.Filename)));
 
   if (!CXXUnit->Reparse(CXXIdx->getPCHContainerOperations(),
-                        *RemappedFiles.get()))
+                        std::move(*RemappedFiles)))
     return CXError_Success;
   if (isASTReadError(CXXUnit))
     return CXError_ASTReadError;
   return CXError_Failure;
 }
 
 int clang_reparseTranslationUnit(CXTranslationUnit TU,
                                  unsigned num_unsaved_files,

clang/tools/libclang/CIndexCodeCompletion.cpp

 /// The CXCodeCompleteResults structure we allocate internally;
 /// the client only sees the initial CXCodeCompleteResults structure.
 ///
 /// Normally, clients of CXString shouldn't care whether or not a CXString is
 /// managed by a pool or by explicitly malloc'ed memory.  But
 /// AllocatedCXCodeCompleteResults outlives the CXTranslationUnit, so we can
 /// not rely on the StringPool in the TU.
 struct AllocatedCXCodeCompleteResults : public CXCodeCompleteResults {
-  AllocatedCXCodeCompleteResults(IntrusiveRefCntPtr<FileManager> FileMgr);
+  AllocatedCXCodeCompleteResults();
   ~AllocatedCXCodeCompleteResults();
   
   /// Diagnostics produced while performing code completion.
   SmallVector<StoredDiagnostic, 8> Diagnostics;
 
   /// Allocated API-exposed wrappters for Diagnostics.
   SmallVector<std::unique_ptr<CXStoredDiagnostic>, 8> DiagnosticsWrappers;
 

 }
 
 /// Tracks the number of code-completion result objects that are 
 /// currently active.
 ///
 /// Used for debugging purposes only.
 static std::atomic<unsigned> CodeCompletionResultObjects;
 
-AllocatedCXCodeCompleteResults::AllocatedCXCodeCompleteResults(
-    IntrusiveRefCntPtr<FileManager> FileMgr)
+AllocatedCXCodeCompleteResults::AllocatedCXCodeCompleteResults()
     : CXCodeCompleteResults(), DiagOpts(new DiagnosticOptions),
       Diag(new DiagnosticsEngine(
           IntrusiveRefCntPtr<DiagnosticIDs>(new DiagnosticIDs), &*DiagOpts)),
       CodeCompletionAllocator(
           std::make_shared<clang::GlobalCodeCompletionAllocator>()),
       Contexts(CXCompletionContext_Unknown),
       ContainerKind(CXCursor_InvalidCode), ContainerIsIncomplete(1) {
   if (getenv("LIBCLANG_OBJTRACKING"))
#endif
 
   CIndexer *CXXIdx = TU->CIdx;
   if (CXXIdx->isOptEnabled(CXGlobalOpt_ThreadBackgroundPriorityForEditing))
     setThreadBackgroundPriority();
 
   ASTUnit::ConcurrencyCheck Check(*AST);
 
   // Perform the remapping of source files.
-  SmallVector<ASTUnit::RemappedFile, 4> RemappedFiles;
+  std::vector<ASTUnit::RemappedFile> RemappedFiles;
 
-  for (auto &UF : unsaved_files) {
-    std::unique_ptr<llvm::MemoryBuffer> MB =
-        llvm::MemoryBuffer::getMemBufferCopy(getContents(UF), UF.Filename);
-    RemappedFiles.push_back(std::make_pair(UF.Filename, MB.release()));
-  }
+  for (auto &UF : unsaved_files)
+    RemappedFiles.push_back(std::make_pair(
+        UF.Filename,
+        llvm::MemoryBuffer::getMemBufferCopy(getContents(UF), UF.Filename)));
 
   if (EnableLogging) {
     // FIXME: Add logging.
   }
 
   // Parse the resulting source file to find code-completion results.
-  AllocatedCXCodeCompleteResults *Results = new AllocatedCXCodeCompleteResults(
-      &AST->getFileManager());
+  AllocatedCXCodeCompleteResults *Results = new AllocatedCXCodeCompleteResults;
   Results->Results = nullptr;
   Results->NumResults = 0;
   
   // Create a code-completion consumer to capture the results.
   CodeCompleteOptions Opts;
   Opts.IncludeBriefComments = IncludeBriefComments;
   Opts.LoadExternal = !SkipPreamble;
   Opts.IncludeFixIts = IncludeFixIts;
   CaptureCompletionResults Capture(Opts, *Results, &TU);
 
   // Perform completion.
   std::vector<const char *> CArgs;
   for (const auto &Arg : TU->Arguments)
     CArgs.push_back(Arg.c_str());
   std::string CompletionInvocation =
       llvm::formatv("-code-completion-at={0}:{1}:{2}", complete_filename,
                     complete_line, complete_column)
           .str();
   LibclangInvocationReporter InvocationReporter(
       *CXXIdx, LibclangInvocationReporter::OperationKind::CompletionOperation,
       TU->ParsingOptions, CArgs, CompletionInvocation, unsaved_files);
   AST->CodeComplete(
-      complete_filename, complete_line, complete_column, RemappedFiles,
-      (options & CXCodeComplete_IncludeMacros),
+      complete_filename, complete_line, complete_column,
+      std::move(RemappedFiles), (options & CXCodeComplete_IncludeMacros),
       (options & CXCodeComplete_IncludeCodePatterns), IncludeBriefComments,
       Capture, CXXIdx->getPCHContainerOperations(), *Results->Diag,
       Results->LangOpts, Results->Diagnostics, Results->TemporaryBuffers);
 
   Results->FileMgr = &AST->getFileManager();
   Results->SourceMgr = &AST->getSourceManager();
   Results->DiagnosticsWrappers.resize(Results->Diagnostics.size());
 

clang/unittests/Frontend/PCHPreambleTest.cpp

std::unique_ptr<ASTUnit> ParseAST(const std::string &EntryFile) {
     PCHContainerOpts = std::make_shared<PCHContainerOperations>();
     std::shared_ptr<CompilerInvocation> CI(new CompilerInvocation);
     CI->getFrontendOpts().Inputs.push_back(
       FrontendInputFile(EntryFile, FrontendOptions::getInputKindForExtension(
         llvm::sys::path::extension(EntryFile).substr(1))));
 
     CI->getTargetOpts().Triple = "i386-unknown-linux-gnu";
 
-    CI->getPreprocessorOpts().RemappedFileBuffers = GetRemappedFiles();
-
-    PreprocessorOptions &PPOpts = CI->getPreprocessorOpts();
-    PPOpts.RemappedFilesKeepOriginalName = true;
-
     IntrusiveRefCntPtr<DiagnosticsEngine>
       Diags(CompilerInstance::createDiagnostics(new DiagnosticOptions, new DiagnosticConsumer));
 
     FileManager *FileMgr = new FileManager(FSOpts, VFS);
 
     std::unique_ptr<ASTUnit> AST = ASTUnit::LoadFromCompilerInvocation(
         CI, PCHContainerOpts, Diags, FileMgr,
-        /*PrecompilePreambleAfterNParses=*/1);
+        /*PrecompilePreambleAfterNParses=*/1, /*UserFilesAreVolatile=*/false,
+        GetRemappedFiles());
     return AST;
   }
 
   bool ReparseAST(const std::unique_ptr<ASTUnit> &AST) {
     bool reparseFailed = AST->Reparse(PCHContainerOpts, GetRemappedFiles(), VFS);
     return !reparseFailed;
   }
 
   unsigned GetFileReadCount(const std::string &Filename) const {
     return VFS->GetReadCount(Filename);
   }
 
 private:
-  std::vector<std::pair<std::string, llvm::MemoryBuffer *>>
-  GetRemappedFiles() const {
-    std::vector<std::pair<std::string, llvm::MemoryBuffer *>> Remapped;
+  std::vector<ASTUnit::RemappedFile> GetRemappedFiles() const {
+    std::vector<ASTUnit::RemappedFile> Remapped;
     for (const auto &RemappedFile : RemappedFiles) {
       std::unique_ptr<MemoryBuffer> buf = MemoryBuffer::getMemBufferCopy(
         RemappedFile.second, RemappedFile.first());
       Remapped.emplace_back(std::string(RemappedFile.first()), buf.release());
     }
     return Remapped;
   }
 };