[VFS] Add an in-memory file system implementation.

[bkramer]

Summary

This is a simple file system tree of memory buffers that can be filled by a
client. In conjunction with an OverlayFS it can be used to make virtual
files accessible right next to physical files. This can be used as a
replacement for the virtual file handling in FileManager and which I intend
to remove eventually.

[VFS] Add working directories to every virtual file system.

For RealFileSystem this is getcwd()/chdir(), the synthetic file systems can
make up one for themselves. OverlayFileSystem now synchronizes the working
directories when a new FS is added to the overlay or the overlay working
directory is set. This allows purely artificial file systems that have zero
ties to the underlying disks.

[klimek]

include/clang/Basic/VirtualFileSystem.h

286–288 ↗

(On Diff #36495)

Return errc::success?

lib/Basic/VirtualFileSystem.cpp

80 ↗

(On Diff #36495)

What's the proposed fix? :)

312–316 ↗

(On Diff #36495)

If I read this correctly, it gets and sets the pwd from FS for the first one? That seems superfluous.
(perhaps change the constructor to not use pushOverlay)

437 ↗

(On Diff #36495)

Do we want to make this const?

439 ↗

(On Diff #36495)

This probably wants to be const.

461 ↗

(On Diff #36495)

I think parent is a misleading name here (parent implies a hierarchy, but this is a simple adapter). Perhaps "Node".

476 ↗

(On Diff #36495)

Is this used in the File interface? It seems - strange... is this really a "mv" implementation?

546–550 ↗

(On Diff #36495)

I think we'll want to get some of these into the interface. But that's fine in a follow-up patch. Add a FIXME though.

800 ↗

(On Diff #36495)

Doesn't the YAML FS work on the same pwd as the current process?

unittests/Tooling/RewriterTestContext.h

42–45 ↗

(On Diff #36495)

Perhaps add a FIXME: To make these tests truly in-memory, we need to overlay the builtin headers.

[bkramer]

Rebased and addressed review comments.

[bkramer]

include/clang/Basic/VirtualFileSystem.h

286–288 ↗

(On Diff #36495)

There is no errc::success, the preferred way is to just use error_code(). Fixed the comment.

lib/Basic/VirtualFileSystem.cpp

80 ↗

(On Diff #36495)

Moving it to LLVM. That's done now :)

312–316 ↗

(On Diff #36495)

Good catch!

476 ↗

(On Diff #36495)

Gone it is.

546–550 ↗

(On Diff #36495)

FIXME added.

800 ↗

(On Diff #36495)

Changed the methods to forward to the inner FS of the yaml overlay.

[klimek]

lg

lib/Basic/VirtualFileSystem.cpp

1263–1265 ↗

(On Diff #36509)

Optional: I'd put this into an anonymous namespace. But I'm also fine with waiting how others think about this :)

cfe/trunk/include/clang/Basic/VirtualFileSystem.h

public:
   llvm::ErrorOr<std::unique_ptr<llvm::MemoryBuffer>>
   getBufferForFile(const Twine &Name, int64_t FileSize = -1,
                    bool RequiresNullTerminator = true, bool IsVolatile = false);
 
   /// \brief Get a directory_iterator for \p Dir.
   /// \note The 'end' iterator is directory_iterator().
   virtual directory_iterator dir_begin(const Twine &Dir,
                                        std::error_code &EC) = 0;
+
+  /// Set the working directory. This will affect all following operations on
+  /// this file system and may propagate down for nested file systems.
+  virtual std::error_code setCurrentWorkingDirectory(const Twine &Path) = 0;
+  /// Get the working directory of this file system.
+  virtual llvm::ErrorOr<std::string> getCurrentWorkingDirectory() const = 0;
+
+  /// Make \a Path an absolute path.
+  ///
+  /// Makes \a Path absolute using the current directory if it is not already.
+  /// An empty \a Path will result in the current directory.
+  ///
+  /// /absolute/path   => /absolute/path
+  /// relative/../path => <current-directory>/relative/../path
+  ///
+  /// \param Path A path that is modified to be an absolute path.
+  /// \returns success if \a path has been made absolute, otherwise a
+  ///          platform-specific error_code.
+  std::error_code makeAbsolute(SmallVectorImpl<char> &Path) const;
 };
 
 /// \brief Gets an \p vfs::FileSystem for the 'real' file system, as seen by
 /// the operating system.
 IntrusiveRefCntPtr<FileSystem> getRealFileSystem();
 
 /// \brief A file system that allows overlaying one \p AbstractFileSystem on top
 /// of another.
public:
   OverlayFileSystem(IntrusiveRefCntPtr<FileSystem> Base);
   /// \brief Pushes a file system on top of the stack.
   void pushOverlay(IntrusiveRefCntPtr<FileSystem> FS);
 
   llvm::ErrorOr<Status> status(const Twine &Path) override;
   llvm::ErrorOr<std::unique_ptr<File>>
   openFileForRead(const Twine &Path) override;
   directory_iterator dir_begin(const Twine &Dir, std::error_code &EC) override;
+  llvm::ErrorOr<std::string> getCurrentWorkingDirectory() const override;
+  std::error_code setCurrentWorkingDirectory(const Twine &Path) override;
 
   typedef FileSystemList::reverse_iterator iterator;
   
   /// \brief Get an iterator pointing to the most recently added file system.
   iterator overlays_begin() { return FSList.rbegin(); }
 
   /// \brief Get an iterator pointing one-past the least recently added file
   /// system.
   iterator overlays_end() { return FSList.rend(); }
 };
 
 namespace detail {
 class InMemoryDirectory;
 } // end namespace detail
 
 /// An in-memory file system.
 class InMemoryFileSystem : public FileSystem {
   std::unique_ptr<detail::InMemoryDirectory> Root;
+  std::string WorkingDirectory;
 
 public:
   InMemoryFileSystem();
   ~InMemoryFileSystem() override;
   void addFile(const Twine &Path, time_t ModificationTime,
                std::unique_ptr<llvm::MemoryBuffer> Buffer);
   StringRef toString() const;
 
   llvm::ErrorOr<Status> status(const Twine &Path) override;
   llvm::ErrorOr<std::unique_ptr<File>>
   openFileForRead(const Twine &Path) override;
   directory_iterator dir_begin(const Twine &Dir, std::error_code &EC) override;
+  llvm::ErrorOr<std::string> getCurrentWorkingDirectory() const override {
+    return WorkingDirectory;
+  }
+  std::error_code setCurrentWorkingDirectory(const Twine &Path) override {
+    WorkingDirectory = Path.str();
+    return std::error_code();
+  }
 };
 
 /// \brief Get a globally unique ID for a virtual file or directory.
 llvm::sys::fs::UniqueID getNextVirtualUniqueID();
 
 /// \brief Gets a \p FileSystem for a virtual file system described in YAML
 /// format.
 IntrusiveRefCntPtr<FileSystem>

cfe/trunk/lib/Basic/VirtualFileSystem.cpp

FileSystem::getBufferForFile(const llvm::Twine &Name, int64_t FileSize,
                              bool RequiresNullTerminator, bool IsVolatile) {
   auto F = openFileForRead(Name);
   if (!F)
     return F.getError();
 
   return (*F)->getBuffer(Name, FileSize, RequiresNullTerminator, IsVolatile);
 }
 
+std::error_code FileSystem::makeAbsolute(SmallVectorImpl<char> &Path) const {
+  auto WorkingDir = getCurrentWorkingDirectory();
+  if (!WorkingDir)
+    return WorkingDir.getError();
+
+  return llvm::sys::fs::make_absolute(WorkingDir.get(), Path);
+}
+
 //===-----------------------------------------------------------------------===/
 // RealFileSystem implementation
 //===-----------------------------------------------------------------------===/
 
 namespace {
 /// \brief Wrapper around a raw file descriptor.
 class RealFile : public File {
   int FD;

 
 namespace {
 /// \brief The file system according to your operating system.
 class RealFileSystem : public FileSystem {
 public:
   ErrorOr<Status> status(const Twine &Path) override;
   ErrorOr<std::unique_ptr<File>> openFileForRead(const Twine &Path) override;
   directory_iterator dir_begin(const Twine &Dir, std::error_code &EC) override;
+
+  llvm::ErrorOr<std::string> getCurrentWorkingDirectory() const override;
+  std::error_code setCurrentWorkingDirectory(const Twine &Path) override;
 };
 } // end anonymous namespace
 
 ErrorOr<Status> RealFileSystem::status(const Twine &Path) {
   sys::fs::file_status RealStatus;
   if (std::error_code EC = sys::fs::status(Path, RealStatus))
     return EC;
   return Status::copyWithNewName(RealStatus, Path.str());
 }
 
 ErrorOr<std::unique_ptr<File>>
 RealFileSystem::openFileForRead(const Twine &Name) {
   int FD;
   if (std::error_code EC = sys::fs::openFileForRead(Name, FD))
     return EC;
   return std::unique_ptr<File>(new RealFile(FD, Name.str()));
 }
 
+llvm::ErrorOr<std::string> RealFileSystem::getCurrentWorkingDirectory() const {
+  SmallString<256> Dir;
+  if (std::error_code EC = llvm::sys::fs::current_path(Dir))
+    return EC;
+  return Dir.str().str();
+}
+
+std::error_code RealFileSystem::setCurrentWorkingDirectory(const Twine &Path) {
+  // FIXME: chdir is thread hostile; on the other hand, creating the same
+  // behavior as chdir is complex: chdir resolves the path once, thus
+  // guaranteeing that all subsequent relative path operations work
+  // on the same path the original chdir resulted in. This makes a
+  // difference for example on network filesystems, where symlinks might be
+  // switched during runtime of the tool. Fixing this depends on having a
+  // file system abstraction that allows openat() style interactions.
+  SmallString<256> Storage;
+  StringRef Dir = Path.toNullTerminatedStringRef(Storage);
+  if (int Err = ::chdir(Dir.data()))
+    return std::error_code(Err, std::generic_category());
+  return std::error_code();
+}
+
 IntrusiveRefCntPtr<FileSystem> vfs::getRealFileSystem() {
   static IntrusiveRefCntPtr<FileSystem> FS = new RealFileSystem();
   return FS;
 }
 
 namespace {
 class RealFSDirIter : public clang::vfs::detail::DirIterImpl {
   std::string Path;
directory_iterator RealFileSystem::dir_begin(const Twine &Dir,
                                              std::error_code &EC) {
   return directory_iterator(std::make_shared<RealFSDirIter>(Dir, EC));
 }
 
 //===-----------------------------------------------------------------------===/
 // OverlayFileSystem implementation
 //===-----------------------------------------------------------------------===/
 OverlayFileSystem::OverlayFileSystem(IntrusiveRefCntPtr<FileSystem> BaseFS) {
-  pushOverlay(BaseFS);
+  FSList.push_back(BaseFS);
 }
 
 void OverlayFileSystem::pushOverlay(IntrusiveRefCntPtr<FileSystem> FS) {
   FSList.push_back(FS);
+  // Synchronize added file systems by duplicating the working directory from
+  // the first one in the list.
+  FS->setCurrentWorkingDirectory(getCurrentWorkingDirectory().get());
 }
 
 ErrorOr<Status> OverlayFileSystem::status(const Twine &Path) {
   // FIXME: handle symlinks that cross file systems
   for (iterator I = overlays_begin(), E = overlays_end(); I != E; ++I) {
     ErrorOr<Status> Status = (*I)->status(Path);
     if (Status || Status.getError() != llvm::errc::no_such_file_or_directory)
       return Status;
   }
   return make_error_code(llvm::errc::no_such_file_or_directory);
 }
 
 ErrorOr<std::unique_ptr<File>>
 OverlayFileSystem::openFileForRead(const llvm::Twine &Path) {
   // FIXME: handle symlinks that cross file systems
   for (iterator I = overlays_begin(), E = overlays_end(); I != E; ++I) {
     auto Result = (*I)->openFileForRead(Path);
     if (Result || Result.getError() != llvm::errc::no_such_file_or_directory)
       return Result;
   }
   return make_error_code(llvm::errc::no_such_file_or_directory);
 }
 
+llvm::ErrorOr<std::string>
+OverlayFileSystem::getCurrentWorkingDirectory() const {
+  // All file systems are synchronized, just take the first working directory.
+  return FSList.front()->getCurrentWorkingDirectory();
+}
+std::error_code
+OverlayFileSystem::setCurrentWorkingDirectory(const Twine &Path) {
+  for (auto &FS : FSList)
+    if (std::error_code EC = FS->setCurrentWorkingDirectory(Path))
+      return EC;
+  return std::error_code();
+}
+
 clang::vfs::detail::DirIterImpl::~DirIterImpl() { }
 
 namespace {
 class OverlayFSDirIterImpl : public clang::vfs::detail::DirIterImpl {
   OverlayFileSystem &Overlays;
   std::string Path;
   OverlayFileSystem::iterator CurrentFS;
   directory_iterator CurrentDirIter;

 }
 
 void InMemoryFileSystem::addFile(const Twine &P, time_t ModificationTime,
                                  std::unique_ptr<llvm::MemoryBuffer> Buffer) {
   SmallString<128> Path;
   P.toVector(Path);
 
   // Fix up relative paths. This just prepends the current working directory.
-  std::error_code EC = sys::fs::make_absolute(Path);
+  std::error_code EC = makeAbsolute(Path);
   assert(!EC);
   (void)EC;
 
   detail::InMemoryDirectory *Dir = Root.get();
   auto I = llvm::sys::path::begin(Path), E = llvm::sys::path::end(Path);
   while (true) {
     StringRef Name = *I;
     detail::InMemoryNode *Node = Dir->getChild(Name);
while (true) {
     }
 
     if (auto *NewDir = dyn_cast<detail::InMemoryDirectory>(Node))
       Dir = NewDir;
   }
 }
 
 static ErrorOr<detail::InMemoryNode *>
-lookupInMemoryNode(detail::InMemoryDirectory *Dir, const Twine &P) {
+lookupInMemoryNode(const InMemoryFileSystem &FS, detail::InMemoryDirectory *Dir,
+                   const Twine &P) {
   SmallString<128> Path;
   P.toVector(Path);
 
   // Fix up relative paths. This just prepends the current working directory.
-  std::error_code EC = sys::fs::make_absolute(Path);
+  std::error_code EC = FS.makeAbsolute(Path);
   assert(!EC);
   (void)EC;
 
   auto I = llvm::sys::path::begin(Path), E = llvm::sys::path::end(Path);
   while (true) {
     detail::InMemoryNode *Node = Dir->getChild(*I);
     ++I;
     if (!Node)
while (true) {
     // Traverse directories.
     Dir = cast<detail::InMemoryDirectory>(Node);
     if (I == E)
       return Dir;
   }
 }
 
 llvm::ErrorOr<Status> InMemoryFileSystem::status(const Twine &Path) {
-  auto Node = lookupInMemoryNode(Root.get(), Path);
+  auto Node = lookupInMemoryNode(*this, Root.get(), Path);
   if (Node)
     return (*Node)->getStatus();
   return Node.getError();
 }
 
 llvm::ErrorOr<std::unique_ptr<File>>
 InMemoryFileSystem::openFileForRead(const Twine &Path) {
-  auto Node = lookupInMemoryNode(Root.get(), Path);
+  auto Node = lookupInMemoryNode(*this, Root.get(), Path);
   if (!Node)
     return Node.getError();
 
   // When we have a file provide a heap-allocated wrapper for the memory buffer
   // to match the ownership semantics for File.
   if (auto *F = dyn_cast<detail::InMemoryFile>(*Node))
     return std::unique_ptr<File>(new detail::InMemoryFileAdaptor(*F));
 
std::error_code increment() override {
     CurrentEntry = I != E ? I->second->getStatus() : Status();
     return std::error_code();
   }
 };
 } // end anonymous namespace
 
 directory_iterator InMemoryFileSystem::dir_begin(const Twine &Dir,
                                                  std::error_code &EC) {
-  auto Node = lookupInMemoryNode(Root.get(), Dir);
+  auto Node = lookupInMemoryNode(*this, Root.get(), Dir);
   if (!Node) {
     EC = Node.getError();
     return directory_iterator(std::make_shared<InMemoryDirIterator>());
   }
 
   if (auto *DirNode = dyn_cast<detail::InMemoryDirectory>(*Node))
     return directory_iterator(std::make_shared<InMemoryDirIterator>(*DirNode));
 
public:
   static VFSFromYAML *create(std::unique_ptr<MemoryBuffer> Buffer,
                              SourceMgr::DiagHandlerTy DiagHandler,
                              void *DiagContext,
                              IntrusiveRefCntPtr<FileSystem> ExternalFS);
 
   ErrorOr<Status> status(const Twine &Path) override;
   ErrorOr<std::unique_ptr<File>> openFileForRead(const Twine &Path) override;
 
+  llvm::ErrorOr<std::string> getCurrentWorkingDirectory() const override {
+    return ExternalFS->getCurrentWorkingDirectory();
+  }
+  std::error_code setCurrentWorkingDirectory(const Twine &Path) override {
+    return ExternalFS->setCurrentWorkingDirectory(Path);
+  }
+
   directory_iterator dir_begin(const Twine &Dir, std::error_code &EC) override{
     ErrorOr<Entry *> E = lookupPath(Dir);
     if (!E) {
       EC = E.getError();
       return directory_iterator();
     }
     ErrorOr<Status> S = status(Dir, *E);
     if (!S) {
VFSFromYAML *VFSFromYAML::create(std::unique_ptr<MemoryBuffer> Buffer,
   return FS.release();
 }
 
 ErrorOr<Entry *> VFSFromYAML::lookupPath(const Twine &Path_) {
   SmallString<256> Path;
   Path_.toVector(Path);
 
   // Handle relative paths
-  if (std::error_code EC = sys::fs::make_absolute(Path))
+  if (std::error_code EC = makeAbsolute(Path))
     return EC;
 
   if (Path.empty())
     return make_error_code(llvm::errc::invalid_argument);
 
   sys::path::const_iterator Start = sys::path::begin(Path);
   sys::path::const_iterator End = sys::path::end(Path);
   for (std::vector<Entry *>::iterator I = Roots.begin(), E = Roots.end();

cfe/trunk/unittests/Basic/VirtualFileSystemTest.cpp

ErrorOr<vfs::Status> status(const Twine &Path) override {
     if (I == FilesAndDirs.end())
       return make_error_code(llvm::errc::no_such_file_or_directory);
     return I->second;
   }
   ErrorOr<std::unique_ptr<vfs::File>>
   openFileForRead(const Twine &Path) override {
     llvm_unreachable("unimplemented");
   }
+  llvm::ErrorOr<std::string> getCurrentWorkingDirectory() const override {
+    return std::string();
+  }
+  std::error_code setCurrentWorkingDirectory(const Twine &Path) override {
+    return std::error_code();
+  }
 
   struct DirIterImpl : public clang::vfs::detail::DirIterImpl {
     std::map<std::string, vfs::Status> &FilesAndDirs;
     std::map<std::string, vfs::Status>::iterator I;
     std::string Path;
     bool isInPath(StringRef S) {
       if (Path.size() < S.size() && S.find(Path) == 0) {
         auto LastSep = S.find_last_of('/');