[VFS] Add support to RedirectingFileSystem for mapping a virtual directory to one in the external FS.

[nathawes]

Summary

Previously file entries in the -ivfsoverlay yaml could map to a file in the external file system (specified via the 'external-contents' property, taking a path), but directories had to list their contents in the form of other file or directory entries (via the 'contents' property, taking an array). Allowing directory entries to map to a directory in the external file system (via the same 'external-contents' property as file entries) makes it possible to present an external directory's contents in a different location and, in combination with the 'fallthrough' option, overlay one directory's contents on top of another.

As an example, with the yaml below:

{ 'roots': [
  {
    'type': 'directory',
    'name': '//root/',
    'contents': [
      {
        'type': 'directory-remap',
        'name': 'mappeddir',
        'external-contents': '//root/foo/bar'
      }
    ]
  }
]}

The path 'root/mappeddir/somefile' would map to 'root/foo/bar/somefile' in the external file system. If that file wasn't found in the external file system and the 'fallthrough' configuration option is true, it would fall back to checking for 'root/mappeddir/somefile' in the external file system. This effectively overlays the contents of root/foo/bar/ on top of //root/mappeddir.

Resolves rdar://problem/72485443

[dexonsmith]

This patch also refactors OverlayFileSystem's directory iterator implementation (OverlayFSDirIterImpl) and VFSFromYamlDirIterImpl into a single implementation, addressing a FIXME about their conceptual similarity.

Can the OverlayFSDirIterImpl part be split into a separate prep commit (possibly even NFC) in order to clarify the functional changes here?

[nathawes]

Yeah, certainly. Let me put that up separately and update this diff.

[dexonsmith]

In D94844#2502725, @nathawes wrote:

Yeah, certainly. Let me put that up separately and update this diff.

Thanks! (Optionally, you can link them using the "edit related revisions" feature at the top of the web interface, if you find it useful.)

[nathawes]

Moved the change combining OverlayFSDirIterImpl and VFSFromYamlDirIterImpl in a single implementation into a separate NFC patch (https://reviews.llvm.org/D94857)

[dexonsmith]

I don't see a test for a case where the mapped directory already exists in the external FS, something like:

# external fs
/d1/f1
/d2/f2

# redirect fs from yaml (fallthrough: true)
/d1 -> /d2

Unless I just missed it, can you add one?

My intuition is we'd want to overlay the directory contents, not replace them:

# overlayed contents (my intuition for fallthrough: true)
/d1/f1
/d1/f2
/d2/f2

# replaced contents (I think unexpected)
/d1/f2
/d2/f2

If you agree, it might be good to test that the recursive directory iterator does the right thing.

Might also be valuable to test the non-fallthrough case, which I assume should give this view:

# fallthrough: false
/d1/f2

@JDevlieghere, I'm hoping you can help review the details of the code here, as I don't know the iteration code well. Also, are these API changes okay for LLDB (I seem to remember LLDB subclassing this...)?

llvm/include/llvm/Support/VirtualFileSystem.h

617–619

Is it easy to move this declaration after DirectoryEntry? If so, I think that'd reduce the diff a bit, and make it more clear what parts of this were being used verbatim from the old FileEntry. Up to you though.

812–817

It's not entirely clear what the extra parameters do / when they're filled in / etc; can you add a bit more documentation?

Also, I see that the public caller ignores the ExternalRedirect parameter. I wonder if this should be renamed to lookupPathWithName or lookupPathImpl, leaving behind a lookupPath wrapper that creates the ignored buffer for the public interface... no strong opinion though.

llvm/lib/Support/VirtualFileSystem.cpp

1904

Could this return E, and then the callers return SetExtRedirect(...)? Not sure if it'd actually be cleaner, up to you.

2098

Nit: please keep the early return.

2106

Nit: please use an early return here to avoid unnecessary indentation (I think the rest of this diff goes away...).

2129–2132

This seems a bit unfortunate to have to change, but not a big deal...

[nathawes]

Thanks for reviewing @dexonsmith!

I don't see a test for a case where the mapped directory already exists in the external FS ... Unless I just missed it, can you add one?

I believe the clang/test/VFS/directory.c regression test covers that case. In that one the real file system is initially set up as follows:

%t/Underlying/
    C.h
    B.h
%t/Middle
    C.h
%t/Overlay
    C.h

And it then sets up vfs overlay yaml files with fallthrough set true that redirect Underlying -> Overlay, or Underlying -> Middle and Middle -> Overlay, as specified on the numbered lines in the test (e.g. // 1) Underlying -> Overlay).

My intuition is we'd want to overlay the directory contents, not replace them

I think it behaves as you expect (as an overlay, rather than replacement) when fallthrough is set to true. You can see that happening in first case in the same test (clang/test/VFS/directory.c). It has the real file system set up as above and creates a vfs yaml file that maps %t/Underlying to %t/Overlay in the external/real file system with fallthrough set true. It invokes clang passing that overlay file, an include path of %t/Underlying, and a source file that includes both C.h and B.h, and checks that it finds the C.h from Overlay (after redirecting Underlying -> Overlay) and B.h from Underlying (after redirecting Underlying -> Overlay, not finding it, and falling back to the original path in the underlying file system).

I really should have a unit test showing the directory iterator output directly, though. That makes the overlay behavior much more obvious.

[JDevlieghere]

Overall this looks good. I wonder if abstracting the ExternalRedirect as a small wrapper class around a SmallString would help. There's a few operations that are repeated, like the example below, and it could also house the logic that's currently in the lambda.

if (ExternalRedirect.empty()) {
  assert(RE->getKind() == RedirectingFileSystem::EK_File);
  ExternalRedirect = RE->getExternalContentsPath();
}

One thing I found tricky during the review was differentiating between places where the ExternalRedirect was computed and uses. Looking at the signatures the lookupPath functions are computing it and status is consuming it, although the latter then changes the StringRef (which I guess shouldn't persist?). I don't know if the wrapper could alleviate this as well.

llvm/include/llvm/Support/VirtualFileSystem.h

564

I think we should expand on this a bit more to contrast a directory-remapped directory vs a file-remapped directory (for a lack of better term). Specifically, an empty file-remapped directory could be confusing, as it behaves quite differently but looks pretty similar in YAML.

{
  'type': 'file',
  'name': <string>,
  'external-contents': <path to external file>
}

{
  'type': 'directory',
  'name': <string>,
  'contents': []
}

Looking at the two, I wonder if there would be any benefit to changing the type in the yaml to match the NameKind.

633

nit: s/whether/Whether/

812–817

+1.

[nathawes]

• Updated the yaml for the new kind of virtual directory (that maps to a directory in the external filesystem) to have a distinct value for its 'type' field ('directory-remap') to better distinguish it from the existing 'directory' type that behaves quite differently.
• Updated lookupPath to return a structure giving both the matched Entry and the ExternalRedirect path, rather than returning the matched Entry and setting the ExternalRedirect path via an out parameter. This makes it clearer where the ExternalRedirect path is computed vs used.
• Updated RedirectingFileSystem::dir_begin to respect the 'use-external-names' setting for DirectoryRemapEntry.
• Added unit tests covering directory iteration (dir_begin) with DirectoryRemapEntry support.
• Expanded on the documentation comments for RedirectingFileSystem to hopefully make the difference between 'directory' and 'directory-remap' entries clearer.

[nathawes]

@JDevlieghere and @dexonsmith would you mind taking another look?

I ended up changing the 'lookup' functions to drop the ExternalRedirect out param and supply that via a new 'LookupResult' return value that also gives the matched Entry. Hopefully that avoids any confusion over where it's computed vs used. It also served as a good home for the duplicated code and the logic in the old SetExternalRedirect closure, as Jonas had mentioned. While adding the extra directory iteration tests Duncan suggested I noticed I wasn't respecting the 'use-external-names' setting in the return items, so I've also added a new directory iterator implementation (RedirectingFSDirRemapIterImpl). That one's used to wrap the iterator the external file system gives for the directory being mapped to, and updates the paths in the items it returns to refer to the virtual directory's path instead, before passing them along.

[dexonsmith]

I haven't looked at the code in detail (deferring to @JDevlieghere), but the new tests look great. Thanks!

[JDevlieghere]

In D94844#2524854, @nathawes wrote:

@JDevlieghere and @dexonsmith would you mind taking another look?

I ended up changing the 'lookup' functions to drop the ExternalRedirect out param and supply that via a new 'LookupResult' return value that also gives the matched Entry. Hopefully that avoids any confusion over where it's computed vs used. It also served as a good home for the duplicated code and the logic in the old SetExternalRedirect closure, as Jonas had mentioned. While adding the extra directory iteration tests Duncan suggested I noticed I wasn't respecting the 'use-external-names' setting in the return items, so I've also added a new directory iterator implementation (RedirectingFSDirRemapIterImpl). That one's used to wrap the iterator the external file system gives for the directory being mapped to, and updates the paths in the items it returns to refer to the virtual directory's path instead, before passing them along.

I like the new approach, it adds an indirection but nicely abstracts how we get the ExternalRedirect. I've left an inline comment about a copy I think we can avoid, but other than that this LGTM.

llvm/include/llvm/Support/VirtualFileSystem.h

748

While I prefer Optional over a pointer, this would now incur a copy, so I think it's better to pass a LookupResult* here, or maybe even a RemapEntry*.

[nathawes]

• Updated shouldFallBackToExternalFS() to accept an Entry * rather than an Optional<LookupResult> to avoid an unnecessary copy.

[nathawes]

Made the following changes as a speculative fix for the windows test failures in the pre-merge checks:

• When appending the remaining path components to the external redirect path of a directory-remap entry, use the separator type of the redirect path rather than the host system.
• For a directory-remap entry, when changing the paths returned in the external file system's directory iterator to appear to be in the virtual file system's directory, use the separator type from the virtual directory's path rather than the host system's.

clang/test/VFS/Inputs/vfsoverlay-directory-relative.yaml

+{
+  'version': 0,
+  'fallthrough': true,
+  'overlay-relative': true,
+  'roots': [
+    { 'name': 'OUT_DIR',
+      'type': 'directory-remap',
+      'external-contents': 'INPUT_DIR'
+    }
+  ]
+}

clang/test/VFS/Inputs/vfsoverlay-directory.yaml

+{
+  'version': 0,
+  'fallthrough': true,
+  'roots': [
+    { 'name': 'OUT_DIR',
+      'type': 'directory-remap',
+      'external-contents': 'INPUT_DIR'
+    }
+  ]
+}

clang/test/VFS/directory.c

+// RUN: rm -rf %t
+// RUN: mkdir -p %t/Underlying
+// RUN: mkdir -p %t/Overlay
+// RUN: mkdir -p %t/Middle
+// RUN: echo '// B.h in Underlying' > %t/Underlying/B.h
+// RUN: echo '#ifdef NESTED' >> %t/Underlying/B.h
+// RUN: echo '#include "C.h"' >> %t/Underlying/B.h
+// RUN: echo '#endif' >> %t/Underlying/B.h
+// RUN: echo '// C.h in Underlying' > %t/Underlying/C.h
+// RUN: echo '// C.h in Middle' > %t/Middle/C.h
+// RUN: echo '// C.h in Overlay' > %t/Overlay/C.h
+
+// 1) Underlying -> Overlay (C.h found, B.h falling back to Underlying)
+// RUN: sed -e "s@INPUT_DIR@%{/t:regex_replacement}/Overlay@g" -e "s@OUT_DIR@%{/t:regex_replacement}/Underlying@g" %S/Inputs/vfsoverlay-directory.yaml > %t/vfs.yaml
+// RUN: %clang_cc1 -Werror -I %t/Underlying -ivfsoverlay %t/vfs.yaml -fsyntax-only -E -C %s 2>&1 | FileCheck --check-prefix=DIRECT %s
+// RUN: %clang_cc1 -Werror -I %t/Underlying -ivfsoverlay %t/vfs.yaml -fsyntax-only -DNESTED -E -C %s 2>&1 | FileCheck --check-prefix=DIRECT %s
+// RUN: sed -e "s@INPUT_DIR@Overlay@g" -e "s@OUT_DIR@%{/t:regex_replacement}/Underlying@g" %S/Inputs/vfsoverlay-directory-relative.yaml > %t/vfs-relative.yaml
+// RUN: %clang_cc1 -Werror -I %t/Underlying -ivfsoverlay %t/vfs-relative.yaml -fsyntax-only -E -C %s 2>&1 | FileCheck --check-prefix=DIRECT %s
+
+// DIRECT: {{^}}// B.h in Underlying
+// DIRECT: {{^}}// C.h in Overlay
+
+// 2) Underlying -> Middle -> Overlay (C.h found, B.h falling back to Underlying)
+// RUN: sed -e "s@INPUT_DIR@%{/t:regex_replacement}/Overlay@g" -e "s@OUT_DIR@%{/t:regex_replacement}/Middle@g" %S/Inputs/vfsoverlay-directory.yaml > %t/vfs.yaml
+// RUN: sed -e "s@INPUT_DIR@%{/t:regex_replacement}/Middle@g" -e "s@OUT_DIR@%{/t:regex_replacement}/Underlying@g" %S/Inputs/vfsoverlay-directory.yaml > %t/vfs2.yaml
+// RUN: %clang_cc1 -Werror -I %t/Underlying -ivfsoverlay %t/vfs.yaml -ivfsoverlay %t/vfs2.yaml -fsyntax-only -E -C %s 2>&1 | FileCheck --check-prefix=DIRECT %s
+// RUN: %clang_cc1 -Werror -I %t/Underlying -ivfsoverlay %t/vfs.yaml -ivfsoverlay %t/vfs2.yaml -DNESTED -fsyntax-only -E -C %s 2>&1 | FileCheck --check-prefix=DIRECT %s
+
+// Same as direct above
+
+// 3) Underlying -> Middle -> Overlay (C.h falling back to Middle, B.h falling back to Underlying)
+// RUN: rm -f %t/Overlay/C.h
+// RUN: %clang_cc1 -Werror -I %t/Underlying -ivfsoverlay %t/vfs.yaml -ivfsoverlay %t/vfs2.yaml -fsyntax-only -E -C %s 2>&1 | FileCheck --check-prefix=FALLBACK %s
+
+// FALLBACK: {{^}}// B.h in Underlying
+// FALLBACK: {{^}}// C.h in Middle
+
+// 3) Underlying -> Middle -> Overlay (C.h falling back to Underlying, B.h falling back to Underlying)
+// RUN: rm -f %t/Middle/C.h
+// RUN: %clang_cc1 -Werror -I %t/Underlying -ivfsoverlay %t/vfs.yaml -ivfsoverlay %t/vfs2.yaml -fsyntax-only -E -C %s 2>&1 | FileCheck --check-prefix=FALLBACK2 %s
+
+// FALLBACK2: {{^}}// B.h in Underlying
+// FALLBACK2: {{^}}// C.h in Underlying
+
+#include "B.h"
+#ifndef NESTED
+#include "C.h"
+#endif

lldb/source/Host/common/FileSystem.cpp

Expected<FileUP> FileSystem::Open(const FileSpec &file_spec,
   return std::move(file);
 }
 
 ErrorOr<std::string> FileSystem::GetExternalPath(const llvm::Twine &path) {
   if (!m_mapped)
     return path.str();
 
   // If VFS mapped we know the underlying FS is a RedirectingFileSystem.
-  ErrorOr<vfs::RedirectingFileSystem::Entry *> E =
+  ErrorOr<vfs::RedirectingFileSystem::LookupResult> Result =
       static_cast<vfs::RedirectingFileSystem &>(*m_fs).lookupPath(path.str());
-  if (!E) {
-    if (E.getError() == llvm::errc::no_such_file_or_directory) {
+  if (!Result) {
+    if (Result.getError() == llvm::errc::no_such_file_or_directory) {
       return path.str();
     }
-    return E.getError();
+    return Result.getError();
   }
 
-  auto *F = dyn_cast<vfs::RedirectingFileSystem::FileEntry>(*E);
-  if (!F)
-    return make_error_code(llvm::errc::not_supported);
-
-  return F->getExternalContentsPath().str();
+  if (Optional<StringRef> ExtRedirect = Result->getExternalRedirect())
+    return std::string(*ExtRedirect);
 }
 
 ErrorOr<std::string> FileSystem::GetExternalPath(const FileSpec &file_spec) {
   return GetExternalPath(file_spec.GetPath());
 }
 
 void FileSystem::Collect(const FileSpec &file_spec) {
   Collect(file_spec.GetPath());

llvm/include/llvm/Support/VirtualFileSystem.h

struct YAMLVFSEntry {
   bool IsDirectory = false;
 };
 
 class RedirectingFSDirIterImpl;
 class RedirectingFileSystemParser;
 
 /// A virtual file system parsed from a YAML file.
 ///
-/// Currently, this class allows creating virtual directories and mapping
-/// virtual file paths to existing external files, available in \c ExternalFS.
+/// Currently, this class allows creating virtual files and directories. Virtual
+/// files map to existing external files in \c ExternalFS, and virtual
+/// directories may either map to existing directories in \c ExternalFS or list
+/// their contents in the form of other virtual directories and/or files.
 ///
 /// The basic structure of the parsed file is:
 /// \verbatim
 /// {
 ///   'version': <version number>,
 ///   <optional configuration>
 ///   'roots': [
 ///              <directory entries>
 ///            ]
 /// }
 /// \endverbatim
 ///
 /// All configuration options are optional.
 ///   'case-sensitive': <boolean, default=(true for Posix, false for Windows)>
 ///   'use-external-names': <boolean, default=true>
 ///   'overlay-relative': <boolean, default=false>
 ///   'fallthrough': <boolean, default=true>
 ///
-/// Virtual directories are represented as
+/// Virtual directories that list their contents are represented as
 /// \verbatim
 /// {
 ///   'type': 'directory',
 ///   'name': <string>,
 ///   'contents': [ <file or directory entries> ]
 /// }
 /// \endverbatim
 ///
-/// The default attributes for virtual directories are:
+/// The default attributes for such virtual directories are:
 /// \verbatim
 /// MTime = now() when created
 /// Perms = 0777
 /// User = Group = 0
 /// Size = 0
 /// UniqueID = unspecified unique value
 /// \endverbatim
 ///
+/// When a path prefix matches such a directory, the next component in the path
+/// is matched against the entries in the 'contents' array.
+///
+/// Re-mapped directories, on the other hand, are represented as
+/// /// \verbatim
+/// {
+///   'type': 'directory-remap',
+///   'name': <string>,
+///   'use-external-name': <boolean>, # Optional
+///   'external-contents': <path to external directory>
+/// }
+/// \endverbatim
+///
+/// and inherit their attributes from the external directory. When a path
+/// prefix matches such an entry, the unmatched components are appended to the
+/// 'external-contents' path, and the resulting path is looked up in the
+/// external file system instead.
+///
 /// Re-mapped files are represented as
 /// \verbatim
 /// {
 ///   'type': 'file',
 ///   'name': <string>,
-///   'use-external-name': <boolean> # Optional
+///   'use-external-name': <boolean>, # Optional
 ///   'external-contents': <path to external file>
 /// }
 /// \endverbatim
 ///
-/// and inherit their attributes from the external contents.
+/// Their attributes and file contents are determined by looking up the file at
+/// their 'external-contents' path in the external file system.
 ///
-/// In both cases, the 'name' field may contain multiple path components (e.g.
-/// /path/to/file). However, any directory that contains more than one child
-/// must be uniquely represented by a directory entry.
+/// For 'file', 'directory' and 'directory-remap' entries the 'name' field may
+/// contain multiple path components (e.g. /path/to/file). However, any
+/// directory in such a path that contains more than one child must be uniquely
+/// represented by a 'directory' entry.
 class RedirectingFileSystem : public vfs::FileSystem {
 public:
-  enum EntryKind { EK_Directory, EK_File };
+  enum EntryKind { EK_Directory, EK_DirectoryRemap, EK_File };
+  enum NameKind { NK_NotSet, NK_External, NK_Virtual };
 
   /// A single file or directory in the VFS.
   class Entry {
     EntryKind Kind;
     std::string Name;
 
   public:
     Entry(EntryKind K, StringRef Name) : Kind(K), Name(Name) {}
     virtual ~Entry() = default;
 
     StringRef getName() const { return Name; }
     EntryKind getKind() const { return Kind; }
   };
 
+  /// A directory in the vfs with explicitly specified contents.
   class DirectoryEntry : public Entry {
     std::vector<std::unique_ptr<Entry>> Contents;
     Status S;
 
   public:
     /// Constructs a directory entry with explicitly specified contents.
     DirectoryEntry(StringRef Name, std::vector<std::unique_ptr<Entry>> Contents,
                    Status S)
         : Entry(EK_Directory, Name), Contents(std::move(Contents)),
           S(std::move(S)) {}
 
     /// Constructs an empty directory entry.
     DirectoryEntry(StringRef Name, Status S)
         : Entry(EK_Directory, Name), S(std::move(S)) {}
 
     Status getStatus() { return S; }
 
     void addContent(std::unique_ptr<Entry> Content) {
       Contents.push_back(std::move(Content));
     }
 
     Entry *getLastContent() const { return Contents.back().get(); }
 
     using iterator = decltype(Contents)::iterator;
 
     iterator contents_begin() { return Contents.begin(); }
     iterator contents_end() { return Contents.end(); }
 
     static bool classof(const Entry *E) { return E->getKind() == EK_Directory; }
   };
 
-  class FileEntry : public Entry {
-  public:
-    enum NameKind { NK_NotSet, NK_External, NK_Virtual };
-
-  private:
+  /// A file or directory in the vfs that is mapped to a file or directory in
+  /// the external filesystem.
+  class RemapEntry : public Entry {
     std::string ExternalContentsPath;
     NameKind UseName;
 
-  public:
-    FileEntry(StringRef Name, StringRef ExternalContentsPath, NameKind UseName)
-        : Entry(EK_File, Name), ExternalContentsPath(ExternalContentsPath),
+  protected:
+    RemapEntry(EntryKind K, StringRef Name, StringRef ExternalContentsPath,
+               NameKind UseName)
+        : Entry(K, Name), ExternalContentsPath(ExternalContentsPath),
           UseName(UseName) {}
 
+  public:
     StringRef getExternalContentsPath() const { return ExternalContentsPath; }
 
-    /// whether to use the external path as the name for this file.
+    /// Whether to use the external path as the name for this file or directory.
     bool useExternalName(bool GlobalUseExternalName) const {
       return UseName == NK_NotSet ? GlobalUseExternalName
                                   : (UseName == NK_External);
     }
 
     NameKind getUseName() const { return UseName; }
 
+    static bool classof(const Entry *E) {
+      switch (E->getKind()) {
+      case EK_DirectoryRemap:
+        LLVM_FALLTHROUGH;
+      case EK_File:
+        return true;
+      case EK_Directory:
+        return false;
+      }
+    }
+  };
+
+  /// A directory in the vfs that maps to a directory in the external file
+  /// system.
+  class DirectoryRemapEntry : public RemapEntry {
+  public:
+    DirectoryRemapEntry(StringRef Name, StringRef ExternalContentsPath,
+                        NameKind UseName)
+        : RemapEntry(EK_DirectoryRemap, Name, ExternalContentsPath, UseName) {}
+
+    static bool classof(const Entry *E) {
+      return E->getKind() == EK_DirectoryRemap;
+    }
+  };
+
+  /// A file in the vfs that maps to a file in the external file system.
+  class FileEntry : public RemapEntry {
+  public:
+    FileEntry(StringRef Name, StringRef ExternalContentsPath, NameKind UseName)
+        : RemapEntry(EK_File, Name, ExternalContentsPath, UseName) {}
+
     static bool classof(const Entry *E) { return E->getKind() == EK_File; }
   };
 
+  /// Represents the result of a path lookup into the RedirectingFileSystem.
+  struct LookupResult {
+    /// The entry the looked-up path corresponds to.
+    Entry *E;
+
+  private:
+    /// When the found Entry is a DirectoryRemapEntry, stores the path in the
+    /// external file system that the looked-up path in the virtual file system
+    //  corresponds to.
+    Optional<std::string> ExternalRedirect;
+
+  public:
+    LookupResult(Entry *E, sys::path::const_iterator Start,
+                 sys::path::const_iterator End);
+
+    /// If the found Entry maps the the input path to a path in the external
+    /// file system (i.e. it is a FileEntry or DirectoryRemapEntry), returns
+    /// that path.
+    Optional<StringRef> getExternalRedirect() const {
+      if (isa<DirectoryRemapEntry>(E))
+        return StringRef(*ExternalRedirect);
+      if (auto *FE = dyn_cast<FileEntry>(E))
+        return FE->getExternalContentsPath();
+      return None;
+    }
+  };
+
 private:
   friend class RedirectingFSDirIterImpl;
   friend class RedirectingFileSystemParser;
 
   bool shouldUseExternalFS() const { return IsFallthrough; }
 
   /// Canonicalize path by removing ".", "..", "./", components. This is
   /// a VFS request, do not bother about symlinks in the path components
   /// but canonicalize in order to perform the correct entry search.
   std::error_code makeCanonical(SmallVectorImpl<char> &Path) const;
 
   /// Whether to fall back to the external file system when an operation fails
-  /// with the given error code.
-  bool shouldFallBackToExternalFS(std::error_code EC) const;
+  /// with the given error code on a path associated with the provided Entry.
+  bool shouldFallBackToExternalFS(std::error_code EC, Entry *E = nullptr) const;
 
   // In a RedirectingFileSystem, keys can be specified in Posix or Windows
   // style (or even a mixture of both), so this comparison helper allows
   // slashes (representing a root) to match backslashes (and vice versa).  Note
   // that, other than the root, path components should not contain slashes or
   // backslashes.
   bool pathComponentMatches(llvm::StringRef lhs, llvm::StringRef rhs) const {
     if ((CaseSensitive ? lhs.equals(rhs) : lhs.equals_lower(rhs)))
       return true;
     return (lhs == "/" && rhs == "\\") || (lhs == "\\" && rhs == "/");
#endif
 
   /// Whether to attempt a file lookup in external file system after it wasn't
   /// found in VFS.
   bool IsFallthrough = true;
   /// @}
 
   RedirectingFileSystem(IntrusiveRefCntPtr<FileSystem> ExternalFS);
 
-  /// Looks up the path <tt>[Start, End)</tt> in \p From, possibly
-  /// recursing into the contents of \p From if it is a directory.
-  ErrorOr<Entry *> lookupPath(llvm::sys::path::const_iterator Start,
+  /// Looks up the path <tt>[Start, End)</tt> in \p From, possibly recursing
+  /// into the contents of \p From if it is a directory. Returns a LookupResult
+  /// giving the matched entry and, if that entry is a FileEntry or
+  /// DirectoryRemapEntry, the path it redirects to in the external file system.
+  ErrorOr<LookupResult> lookupPathImpl(llvm::sys::path::const_iterator Start,
                                        llvm::sys::path::const_iterator End,
                                        Entry *From) const;
 
-  /// Get the status of a given an \c Entry.
-  ErrorOr<Status> status(const Twine &Path, Entry *E);
+  /// Get the status for a path with the provided \c LookupResult.
+  ErrorOr<Status> status(const Twine &Path, const LookupResult &Result);
 
 public:
-  /// Looks up \p Path in \c Roots.
-  ErrorOr<Entry *> lookupPath(StringRef Path) const;
+  /// Looks up \p Path in \c Roots and returns a LookupResult giving the
+  /// matched entry and, if the entry was a FileEntry or DirectoryRemapEntry,
+  /// the path it redirects to in the external file system.
+  ErrorOr<LookupResult> lookupPath(StringRef Path) const;
 
   /// Parses \p Buffer, which is expected to be in YAML format and
   /// returns a virtual file system representing its contents.
   static std::unique_ptr<RedirectingFileSystem>
   create(std::unique_ptr<MemoryBuffer> Buffer,
          SourceMgr::DiagHandlerTy DiagHandler, StringRef YAMLFilePath,
          void *DiagContext, IntrusiveRefCntPtr<FileSystem> ExternalFS);
 
   /// Redirect each of the remapped files from first to second.

llvm/lib/Support/VirtualFileSystem.cpp

 } // namespace llvm
 
 //===-----------------------------------------------------------------------===/
 // RedirectingFileSystem implementation
 //===-----------------------------------------------------------------------===/
 
 namespace {
 
-/// Removes leading "./" as well as path components like ".." and ".".
-static llvm::SmallString<256> canonicalize(llvm::StringRef Path) {
-  // First detect the path style in use by checking the first separator.
+static llvm::sys::path::Style getExistingStyle(llvm::StringRef Path) {
+  // Detect the path style in use by checking the first separator.
   llvm::sys::path::Style style = llvm::sys::path::Style::native;
   const size_t n = Path.find_first_of("/\\");
   if (n != static_cast<size_t>(-1))
     style = (Path[n] == '/') ? llvm::sys::path::Style::posix
                              : llvm::sys::path::Style::windows;
+  return style;
+}
+
+/// Removes leading "./" as well as path components like ".." and ".".
+static llvm::SmallString<256> canonicalize(llvm::StringRef Path) {
+  // First detect the path style in use by checking the first separator.
+  llvm::sys::path::Style style = getExistingStyle(Path);
 
   // Now remove the dots.  Explicitly specifying the path style prevents the
   // direction of the slashes from changing.
   llvm::SmallString<256> result =
       llvm::sys::path::remove_leading_dotslash(Path, style);
   llvm::sys::path::remove_dots(result, /*remove_dot_dot=*/true, style);
   return result;
 }
std::error_code incrementImpl(bool IsFirstTime) {
     if (!IsFirstTime)
       ++Current;
     if (Current != End) {
       SmallString<128> PathStr(Dir);
       llvm::sys::path::append(PathStr, (*Current)->getName());
       sys::fs::file_type Type = sys::fs::file_type::type_unknown;
       switch ((*Current)->getKind()) {
       case RedirectingFileSystem::EK_Directory:
+        LLVM_FALLTHROUGH;
+      case RedirectingFileSystem::EK_DirectoryRemap:
         Type = sys::fs::file_type::directory_file;
         break;
       case RedirectingFileSystem::EK_File:
         Type = sys::fs::file_type::regular_file;
         break;
       }
       CurrentEntry = directory_entry(std::string(PathStr.str()), Type);
     } else {
RedirectingFSDirIterImpl(
     EC = incrementImpl(/*IsFirstTime=*/true);
   }
 
   std::error_code increment() override {
     return incrementImpl(/*IsFirstTime=*/false);
   }
 };
 
+/// Directory iterator implementation for \c RedirectingFileSystem's
+/// directory remap entries that maps the paths reported by the external
+/// file system's directory iterator back to the virtual directory's path.
+class RedirectingFSDirRemapIterImpl : public llvm::vfs::detail::DirIterImpl {
+  std::string Dir;
+  llvm::sys::path::Style DirStyle;
+  llvm::vfs::directory_iterator ExternalIter;
+
+public:
+  RedirectingFSDirRemapIterImpl(std::string DirPath,
+                                llvm::vfs::directory_iterator ExtIter)
+      : Dir(std::move(DirPath)), DirStyle(getExistingStyle(Dir)),
+        ExternalIter(ExtIter) {
+    if (ExternalIter != llvm::vfs::directory_iterator())
+      setCurrentEntry();
+  }
+
+  void setCurrentEntry() {
+    StringRef ExternalPath = ExternalIter->path();
+    llvm::sys::path::Style ExternalStyle = getExistingStyle(ExternalPath);
+    StringRef File = llvm::sys::path::filename(ExternalPath, ExternalStyle);
+
+    SmallString<128> NewPath(Dir);
+    llvm::sys::path::append(NewPath, DirStyle, File);
+
+    CurrentEntry = directory_entry(std::string(NewPath), ExternalIter->type());
+  }
+
+  std::error_code increment() override {
+    std::error_code EC;
+    ExternalIter.increment(EC);
+    if (!EC && ExternalIter != llvm::vfs::directory_iterator())
+      setCurrentEntry();
+    else
+      CurrentEntry = directory_entry();
+    return EC;
+  }
+};
+
 llvm::ErrorOr<std::string>
 RedirectingFileSystem::getCurrentWorkingDirectory() const {
   return WorkingDirectory;
 }
 
 std::error_code
 RedirectingFileSystem::setCurrentWorkingDirectory(const Twine &Path) {
   // Don't change the working directory if the path doesn't exist.
directory_iterator RedirectingFileSystem::dir_begin(const Twine &Dir,
                                                     std::error_code &EC) {
   SmallString<256> Path;
   Dir.toVector(Path);
 
   EC = makeCanonical(Path);
   if (EC)
     return {};
 
-  ErrorOr<RedirectingFileSystem::Entry *> E = lookupPath(Path);
-  if (!E) {
-    EC = E.getError();
+  ErrorOr<RedirectingFileSystem::LookupResult> Result = lookupPath(Path);
+  if (!Result) {
+    EC = Result.getError();
     if (shouldFallBackToExternalFS(EC))
       return ExternalFS->dir_begin(Path, EC);
     return {};
   }
-  ErrorOr<Status> S = status(Path, *E);
+
+  // Use status to make sure the path exists and refers to a directory.
+  ErrorOr<Status> S = status(Path, *Result);
   if (!S) {
+    if (shouldFallBackToExternalFS(S.getError(), Result->E))
+      return ExternalFS->dir_begin(Dir, EC);
     EC = S.getError();
     return {};
   }
   if (!S->isDirectory()) {
     EC = std::error_code(static_cast<int>(errc::not_a_directory),
                          std::system_category());
     return {};
   }
 
-  auto *D = cast<RedirectingFileSystem::DirectoryEntry>(*E);
-  auto DirIter = directory_iterator(std::make_shared<RedirectingFSDirIterImpl>(
-      Path, D->contents_begin(), D->contents_end(), EC));
+  // Create the appropriate directory iterator based on whether we found a
+  // DirectoryRemapEntry or DirectoryEntry.
+  directory_iterator DirIter;
+  if (auto ExtRedirect = Result->getExternalRedirect()) {
+    auto RE = cast<RedirectingFileSystem::RemapEntry>(Result->E);
+    DirIter = ExternalFS->dir_begin(*ExtRedirect, EC);
+
+    if (!RE->useExternalName(UseExternalNames)) {
+      // Update the paths in the results to use the virtual directory's path.
+      DirIter =
+          directory_iterator(std::make_shared<RedirectingFSDirRemapIterImpl>(
+              std::string(Path), DirIter));
+    }
+  } else {
+    auto DE = cast<DirectoryEntry>(Result->E);
+    DirIter = directory_iterator(std::make_shared<RedirectingFSDirIterImpl>(
+        Path, DE->contents_begin(), DE->contents_end(), EC));
+  }
 
   if (!shouldUseExternalFS())
     return DirIter;
   return directory_iterator(std::make_shared<CombiningDirIterImpl>(
       DirIter, ExternalFS, std::string(Path), EC));
 }
 
 void RedirectingFileSystem::setExternalContentsPrefixDir(StringRef PrefixDir) {
case RedirectingFileSystem::EK_Directory: {
       // is parsed. This only leads to redundant walks, ignore it.
       if (!Name.empty())
         NewParentE = lookupOrCreateEntry(FS, Name, NewParentE);
       for (std::unique_ptr<RedirectingFileSystem::Entry> &SubEntry :
            llvm::make_range(DE->contents_begin(), DE->contents_end()))
         uniqueOverlayTree(FS, SubEntry.get(), NewParentE);
       break;
     }
+    case RedirectingFileSystem::EK_DirectoryRemap: {
+      assert(NewParentE && "Parent entry must exist");
+      auto *DR = cast<RedirectingFileSystem::DirectoryRemapEntry>(SrcE);
+      auto *DE = cast<RedirectingFileSystem::DirectoryEntry>(NewParentE);
+      DE->addContent(
+          std::make_unique<RedirectingFileSystem::DirectoryRemapEntry>(
+              Name, DR->getExternalContentsPath(), DR->getUseName()));
+      break;
+    }
     case RedirectingFileSystem::EK_File: {
       assert(NewParentE && "Parent entry must exist");
       auto *FE = cast<RedirectingFileSystem::FileEntry>(SrcE);
       auto *DE = cast<RedirectingFileSystem::DirectoryEntry>(NewParentE);
       DE->addContent(std::make_unique<RedirectingFileSystem::FileEntry>(
           Name, FE->getExternalContentsPath(), FE->getUseName()));
       break;
     }
KeyStatusPair Fields[] = {
         KeyStatusPair("type", true),
         KeyStatusPair("contents", false),
         KeyStatusPair("external-contents", false),
         KeyStatusPair("use-external-name", false),
     };
 
     DenseMap<StringRef, KeyStatus> Keys(std::begin(Fields), std::end(Fields));
 
-    bool HasContents = false; // external or otherwise
+    enum { CF_NotSet, CF_List, CF_External } ContentsField = CF_NotSet;
     std::vector<std::unique_ptr<RedirectingFileSystem::Entry>>
         EntryArrayContents;
     SmallString<256> ExternalContentsPath;
     SmallString<256> Name;
     yaml::Node *NameValueNode = nullptr;
-    auto UseExternalName = RedirectingFileSystem::FileEntry::NK_NotSet;
+    auto UseExternalName = RedirectingFileSystem::NK_NotSet;
     RedirectingFileSystem::EntryKind Kind;
 
     for (auto &I : *M) {
       StringRef Key;
       // Reuse the buffer for key and value, since we don't look at key after
       // parsing value.
       SmallString<256> Buffer;
       if (!parseScalarString(I.getKey(), Key, Buffer))
for (auto &I : *M) {
         Name = canonicalize(Value).str();
       } else if (Key == "type") {
         if (!parseScalarString(I.getValue(), Value, Buffer))
           return nullptr;
         if (Value == "file")
           Kind = RedirectingFileSystem::EK_File;
         else if (Value == "directory")
           Kind = RedirectingFileSystem::EK_Directory;
+        else if (Value == "directory-remap")
+          Kind = RedirectingFileSystem::EK_DirectoryRemap;
         else {
           error(I.getValue(), "unknown value for 'type'");
           return nullptr;
         }
       } else if (Key == "contents") {
-        if (HasContents) {
+        if (ContentsField != CF_NotSet) {
           error(I.getKey(),
                 "entry already has 'contents' or 'external-contents'");
           return nullptr;
         }
-        HasContents = true;
+        ContentsField = CF_List;
         auto *Contents = dyn_cast<yaml::SequenceNode>(I.getValue());
         if (!Contents) {
           // FIXME: this is only for directories, what about files?
           error(I.getValue(), "expected array");
           return nullptr;
         }
 
         for (auto &I : *Contents) {
           if (std::unique_ptr<RedirectingFileSystem::Entry> E =
                   parseEntry(&I, FS, /*IsRootEntry*/ false))
             EntryArrayContents.push_back(std::move(E));
           else
             return nullptr;
         }
       } else if (Key == "external-contents") {
-        if (HasContents) {
+        if (ContentsField != CF_NotSet) {
           error(I.getKey(),
                 "entry already has 'contents' or 'external-contents'");
           return nullptr;
         }
-        HasContents = true;
+        ContentsField = CF_External;
         if (!parseScalarString(I.getValue(), Value, Buffer))
           return nullptr;
 
         SmallString<256> FullPath;
         if (FS->IsRelativeOverlay) {
           FullPath = FS->getExternalContentsPrefixDir();
           assert(!FullPath.empty() &&
                  "External contents prefix directory must exist");
           llvm::sys::path::append(FullPath, Value);
         } else {
           FullPath = Value;
         }
 
         // Guarantee that old YAML files containing paths with ".." and "."
         // are properly canonicalized before read into the VFS.
         FullPath = canonicalize(FullPath);
         ExternalContentsPath = FullPath.str();
       } else if (Key == "use-external-name") {
         bool Val;
         if (!parseScalarBool(I.getValue(), Val))
           return nullptr;
-        UseExternalName = Val ? RedirectingFileSystem::FileEntry::NK_External
-                              : RedirectingFileSystem::FileEntry::NK_Virtual;
+        UseExternalName = Val ? RedirectingFileSystem::NK_External
+                              : RedirectingFileSystem::NK_Virtual;
       } else {
         llvm_unreachable("key missing from Keys");
       }
     }
 
     if (Stream.failed())
       return nullptr;
 
     // check for missing keys
-    if (!HasContents) {
+    if (ContentsField == CF_NotSet) {
       error(N, "missing key 'contents' or 'external-contents'");
       return nullptr;
     }
     if (!checkMissingKeys(N, Keys))
       return nullptr;
 
     // check invalid configuration
     if (Kind == RedirectingFileSystem::EK_Directory &&
-        UseExternalName != RedirectingFileSystem::FileEntry::NK_NotSet) {
-      error(N, "'use-external-name' is not supported for directories");
+        UseExternalName != RedirectingFileSystem::NK_NotSet) {
+      error(N, "'use-external-name' is not supported for 'directory' entries");
+      return nullptr;
+    }
+
+    if (Kind == RedirectingFileSystem::EK_DirectoryRemap &&
+        ContentsField == CF_List) {
+      error(N, "'contents' is not supported for 'directory-remap' entries");
       return nullptr;
     }
 
     sys::path::Style path_style = sys::path::Style::native;
     if (IsRootEntry) {
       // VFS root entries may be in either Posix or Windows style.  Figure out
       // which style we have, and use it consistently.
       if (sys::path::is_absolute(Name, sys::path::Style::posix)) {
parseEntry(yaml::Node *N, RedirectingFileSystem *FS, bool IsRootEntry) {
     StringRef LastComponent = sys::path::filename(Trimmed, path_style);
 
     std::unique_ptr<RedirectingFileSystem::Entry> Result;
     switch (Kind) {
     case RedirectingFileSystem::EK_File:
       Result = std::make_unique<RedirectingFileSystem::FileEntry>(
           LastComponent, std::move(ExternalContentsPath), UseExternalName);
       break;
+    case RedirectingFileSystem::EK_DirectoryRemap:
+      Result = std::make_unique<RedirectingFileSystem::DirectoryRemapEntry>(
+          LastComponent, std::move(ExternalContentsPath), UseExternalName);
+      break;
     case RedirectingFileSystem::EK_Directory:
       Result = std::make_unique<RedirectingFileSystem::DirectoryEntry>(
           LastComponent, std::move(EntryArrayContents),
           Status("", getNextVirtualUniqueID(), std::chrono::system_clock::now(),
                  0, 0, 0, file_type::directory_file, sys::fs::all_all));
       break;
     }
 
assert(Parent && "File without a directory?");
       auto EC = ExternalFS.makeAbsolute(To);
       (void)EC;
       assert(!EC && "Could not make absolute path");
     }
 
     // Add the file.
     auto NewFile = std::make_unique<RedirectingFileSystem::FileEntry>(
         llvm::sys::path::filename(From), To,
-        UseExternalNames ? RedirectingFileSystem::FileEntry::NK_External
-                         : RedirectingFileSystem::FileEntry::NK_Virtual);
+        UseExternalNames ? RedirectingFileSystem::NK_External
+                         : RedirectingFileSystem::NK_Virtual);
     ToEntry = NewFile.get();
     cast<RedirectingFileSystem::DirectoryEntry>(Parent)->addContent(
         std::move(NewFile));
   }
 
   return FS;
 }
 
+RedirectingFileSystem::LookupResult::LookupResult(
+    Entry *E, sys::path::const_iterator Start, sys::path::const_iterator End)
+    : E(E) {
+  assert(E != nullptr);
+  // If the matched entry is a DirectoryRemapEntry, set ExternalRedirect to the
+  // path of the directory it maps to in the external file system plus any
+  // remaining path components in the provided iterator.
+  if (auto *DRE = dyn_cast<RedirectingFileSystem::DirectoryRemapEntry>(E)) {
+    SmallString<256> Redirect(DRE->getExternalContentsPath());
+    sys::path::append(Redirect, Start, End,
+                      getExistingStyle(DRE->getExternalContentsPath()));
+    ExternalRedirect = std::string(Redirect);
+  }
+}
+
 bool RedirectingFileSystem::shouldFallBackToExternalFS(
-    std::error_code EC) const {
+    std::error_code EC, RedirectingFileSystem::Entry *E) const {
+  if (E && !isa<RedirectingFileSystem::DirectoryRemapEntry>(E))
+    return false;
   return shouldUseExternalFS() && EC == llvm::errc::no_such_file_or_directory;
 }
 
 std::error_code
 RedirectingFileSystem::makeCanonical(SmallVectorImpl<char> &Path) const {
   if (std::error_code EC = makeAbsolute(Path))
     return EC;
 
   llvm::SmallString<256> CanonicalPath =
       canonicalize(StringRef(Path.data(), Path.size()));
   if (CanonicalPath.empty())
     return make_error_code(llvm::errc::invalid_argument);
 
   Path.assign(CanonicalPath.begin(), CanonicalPath.end());
   return {};
 }
 
-ErrorOr<RedirectingFileSystem::Entry *>
+ErrorOr<RedirectingFileSystem::LookupResult>
 RedirectingFileSystem::lookupPath(StringRef Path) const {
   sys::path::const_iterator Start = sys::path::begin(Path);
   sys::path::const_iterator End = sys::path::end(Path);
   for (const auto &Root : Roots) {
-    ErrorOr<RedirectingFileSystem::Entry *> Result =
-        lookupPath(Start, End, Root.get());
+    ErrorOr<RedirectingFileSystem::LookupResult> Result =
+        lookupPathImpl(Start, End, Root.get());
     if (Result || Result.getError() != llvm::errc::no_such_file_or_directory)
       return Result;
   }
   return make_error_code(llvm::errc::no_such_file_or_directory);
 }
 
-ErrorOr<RedirectingFileSystem::Entry *>
-RedirectingFileSystem::lookupPath(sys::path::const_iterator Start,
-                                  sys::path::const_iterator End,
-                                  RedirectingFileSystem::Entry *From) const {
+ErrorOr<RedirectingFileSystem::LookupResult>
+RedirectingFileSystem::lookupPathImpl(
+    sys::path::const_iterator Start, sys::path::const_iterator End,
   assert(!isTraversalComponent(*Start) &&
          !isTraversalComponent(From->getName()) &&
          "Paths should not contain traversal components");
 
   StringRef FromName = From->getName();
 
   // Forward the search to the next component in case this is an empty one.
   if (!FromName.empty()) {
     if (!pathComponentMatches(*Start, FromName))
       return make_error_code(llvm::errc::no_such_file_or_directory);
 
     ++Start;
 
     if (Start == End) {
       // Match!
-      return From;
+      return LookupResult(From, Start, End);
     }
   }
 
-  auto *DE = dyn_cast<RedirectingFileSystem::DirectoryEntry>(From);
-  if (!DE)
+  if (isa<RedirectingFileSystem::FileEntry>(From))
     return make_error_code(llvm::errc::not_a_directory);
 
+  if (isa<RedirectingFileSystem::DirectoryRemapEntry>(From))
+    return LookupResult(From, Start, End);
+
+  auto *DE = cast<RedirectingFileSystem::DirectoryEntry>(From);
   for (const std::unique_ptr<RedirectingFileSystem::Entry> &DirEntry :
        llvm::make_range(DE->contents_begin(), DE->contents_end())) {
-    ErrorOr<RedirectingFileSystem::Entry *> Result =
-        lookupPath(Start, End, DirEntry.get());
+    ErrorOr<RedirectingFileSystem::LookupResult> Result =
+        lookupPathImpl(Start, End, DirEntry.get());
     if (Result || Result.getError() != llvm::errc::no_such_file_or_directory)
       return Result;
   }
 
   return make_error_code(llvm::errc::no_such_file_or_directory);
 }
 
 static Status getRedirectedFileStatus(const Twine &Path, bool UseExternalNames,
                                       Status ExternalStatus) {
   Status S = ExternalStatus;
   if (!UseExternalNames)
     S = Status::copyWithNewName(S, Path);
   S.IsVFSMapped = true;
   return S;
 }
 
-ErrorOr<Status> RedirectingFileSystem::status(const Twine &Path,
-                                              RedirectingFileSystem::Entry *E) {
-  assert(E != nullptr);
-  if (auto *F = dyn_cast<RedirectingFileSystem::FileEntry>(E)) {
-    ErrorOr<Status> S = ExternalFS->status(F->getExternalContentsPath());
-    assert(!S || S->getName() == F->getExternalContentsPath());
-    if (S)
-      return getRedirectedFileStatus(Path, F->useExternalName(UseExternalNames),
-                                     *S);
+ErrorOr<Status> RedirectingFileSystem::status(
+    const Twine &Path, const RedirectingFileSystem::LookupResult &Result) {
+  if (Optional<StringRef> ExtRedirect = Result.getExternalRedirect()) {
+    ErrorOr<Status> S = ExternalFS->status(*ExtRedirect);
+    if (!S)
       return S;
-  } else { // directory
-    auto *DE = cast<RedirectingFileSystem::DirectoryEntry>(E);
-    return Status::copyWithNewName(DE->getStatus(), Path);
+    auto *RE = cast<RedirectingFileSystem::RemapEntry>(Result.E);
+    return getRedirectedFileStatus(Path, RE->useExternalName(UseExternalNames),
+                                   *S);
   }
+
+  auto *DE = cast<RedirectingFileSystem::DirectoryEntry>(Result.E);
+  return Status::copyWithNewName(DE->getStatus(), Path);
 }
 
 ErrorOr<Status> RedirectingFileSystem::status(const Twine &Path_) {
   SmallString<256> Path;
   Path_.toVector(Path);
 
   if (std::error_code EC = makeCanonical(Path))
     return EC;
 
-  ErrorOr<RedirectingFileSystem::Entry *> Result = lookupPath(Path);
+  ErrorOr<RedirectingFileSystem::LookupResult> Result = lookupPath(Path);
   if (!Result) {
     if (shouldFallBackToExternalFS(Result.getError()))
       return ExternalFS->status(Path);
     return Result.getError();
   }
-  return status(Path, *Result);
+
+  ErrorOr<Status> S = status(Path, *Result);
+  if (!S && shouldFallBackToExternalFS(S.getError(), Result->E))
+    S = ExternalFS->status(Path);
+  return S;
 }
 
 namespace {
 
 /// Provide a file wrapper with an overriden status.
 class FileWithFixedStatus : public File {
   std::unique_ptr<File> InnerFile;
   Status S;

 ErrorOr<std::unique_ptr<File>>
 RedirectingFileSystem::openFileForRead(const Twine &Path_) {
   SmallString<256> Path;
   Path_.toVector(Path);
 
   if (std::error_code EC = makeCanonical(Path))
     return EC;
 
-  ErrorOr<RedirectingFileSystem::Entry *> E = lookupPath(Path);
-  if (!E) {
-    if (shouldFallBackToExternalFS(E.getError()))
+  ErrorOr<RedirectingFileSystem::LookupResult> Result = lookupPath(Path);
+  if (!Result) {
+    if (shouldFallBackToExternalFS(Result.getError()))
       return ExternalFS->openFileForRead(Path);
-    return E.getError();
+    return Result.getError();
   }
 
-  auto *F = dyn_cast<RedirectingFileSystem::FileEntry>(*E);
-  if (!F) // FIXME: errc::not_a_file?
+  if (!Result->getExternalRedirect()) // FIXME: errc::not_a_file?
     return make_error_code(llvm::errc::invalid_argument);
 
-  auto Result = ExternalFS->openFileForRead(F->getExternalContentsPath());
-  if (!Result)
-    return Result;
+  StringRef ExtRedirect = *Result->getExternalRedirect();
+  auto *RE = cast<RedirectingFileSystem::RemapEntry>(Result->E);
+
+  auto ExternalFile = ExternalFS->openFileForRead(ExtRedirect);
+  if (!ExternalFile) {
+    if (shouldFallBackToExternalFS(ExternalFile.getError(), Result->E))
+      return ExternalFS->openFileForRead(Path);
+    return ExternalFile;
+  }
 
-  auto ExternalStatus = (*Result)->status();
+  auto ExternalStatus = (*ExternalFile)->status();
   if (!ExternalStatus)
     return ExternalStatus.getError();
 
   // FIXME: Update the status with the name and VFSMapped.
-  Status S = getRedirectedFileStatus(Path, F->useExternalName(UseExternalNames),
-                                     *ExternalStatus);
+  Status S = getRedirectedFileStatus(
+      Path, RE->useExternalName(UseExternalNames), *ExternalStatus);
   return std::unique_ptr<File>(
-      std::make_unique<FileWithFixedStatus>(std::move(*Result), S));
+      std::make_unique<FileWithFixedStatus>(std::move(*ExternalFile), S));
 }
 
 std::error_code
 RedirectingFileSystem::getRealPath(const Twine &Path_,
                                    SmallVectorImpl<char> &Output) const {
   SmallString<256> Path;
   Path_.toVector(Path);
 
   if (std::error_code EC = makeCanonical(Path))
     return EC;
 
-  ErrorOr<RedirectingFileSystem::Entry *> Result = lookupPath(Path);
+  ErrorOr<RedirectingFileSystem::LookupResult> Result = lookupPath(Path);
   if (!Result) {
     if (shouldFallBackToExternalFS(Result.getError()))
       return ExternalFS->getRealPath(Path, Output);
     return Result.getError();
   }
 
-  if (auto *F = dyn_cast<RedirectingFileSystem::FileEntry>(*Result)) {
-    return ExternalFS->getRealPath(F->getExternalContentsPath(), Output);
+  // If we found FileEntry or DirectoryRemapEntry, look up the mapped
+  // path in the external file system.
+  if (auto ExtRedirect = Result->getExternalRedirect()) {
+    auto P = ExternalFS->getRealPath(*ExtRedirect, Output);
+    if (!P && shouldFallBackToExternalFS(P, Result->E)) {
+      return ExternalFS->getRealPath(Path, Output);
+    }
+    return P;
   }
-  // Even if there is a directory entry, fall back to ExternalFS if allowed,
+
+  // If we found a DirectoryEntry, still fall back to ExternalFS if allowed,
   // because directories don't have a single external contents path.
   return shouldUseExternalFS() ? ExternalFS->getRealPath(Path, Output)
                                : llvm::errc::invalid_argument;
 }
 
 std::unique_ptr<FileSystem>
 vfs::getVFSFromYAML(std::unique_ptr<MemoryBuffer> Buffer,
                     SourceMgr::DiagHandlerTy DiagHandler,
if (Kind == RedirectingFileSystem::EK_Directory) {
     auto *DE = dyn_cast<RedirectingFileSystem::DirectoryEntry>(SrcE);
     assert(DE && "Must be a directory");
     for (std::unique_ptr<RedirectingFileSystem::Entry> &SubEntry :
          llvm::make_range(DE->contents_begin(), DE->contents_end())) {
       Path.push_back(SubEntry->getName());
       getVFSEntries(SubEntry.get(), Path, Entries);
       Path.pop_back();
     }
     return;
   }
 
+  if (Kind == RedirectingFileSystem::EK_DirectoryRemap) {
+    auto *DR = dyn_cast<RedirectingFileSystem::DirectoryRemapEntry>(SrcE);
+    assert(DR && "Must be a directory remap");
+    SmallString<128> VPath;
+    for (auto &Comp : Path)
+      llvm::sys::path::append(VPath, Comp);
+    Entries.push_back(
+        YAMLVFSEntry(VPath.c_str(), DR->getExternalContentsPath()));
+    return;
+  }
+
   assert(Kind == RedirectingFileSystem::EK_File && "Must be a EK_File");
   auto *FE = dyn_cast<RedirectingFileSystem::FileEntry>(SrcE);
   assert(FE && "Must be a file");
   SmallString<128> VPath;
   for (auto &Comp : Path)
     llvm::sys::path::append(VPath, Comp);
   Entries.push_back(YAMLVFSEntry(VPath.c_str(), FE->getExternalContentsPath()));
 }
 
 void vfs::collectVFSFromYAML(std::unique_ptr<MemoryBuffer> Buffer,
                              SourceMgr::DiagHandlerTy DiagHandler,
                              StringRef YAMLFilePath,
                              SmallVectorImpl<YAMLVFSEntry> &CollectedEntries,
                              void *DiagContext,
                              IntrusiveRefCntPtr<FileSystem> ExternalFS) {
   std::unique_ptr<RedirectingFileSystem> VFS = RedirectingFileSystem::create(
       std::move(Buffer), DiagHandler, YAMLFilePath, DiagContext,
       std::move(ExternalFS));
-  ErrorOr<RedirectingFileSystem::Entry *> RootE = VFS->lookupPath("/");
-  if (!RootE)
+  ErrorOr<RedirectingFileSystem::LookupResult> RootResult =
+      VFS->lookupPath("/");
+  if (!RootResult)
     return;
   SmallVector<StringRef, 8> Components;
   Components.push_back("/");
-  getVFSEntries(*RootE, Components, CollectedEntries);
+  getVFSEntries(RootResult->E, Components, CollectedEntries);
 }
 
 UniqueID vfs::getNextVirtualUniqueID() {
   static std::atomic<unsigned> UID;
   unsigned ID = ++UID;
   // The following assumes that uint64_t max will never collide with a real
   // dev_t value from the OS.
   return UniqueID(std::numeric_limits<uint64_t>::max(), ID);

llvm/unittests/Support/VirtualFileSystemTest.cpp

TEST_F(VFSFromYAMLTest, BasicVFSFromYAML) {
   EXPECT_EQ(nullptr, FS.get());
   FS = getFromYAMLString("'string'");
   EXPECT_EQ(nullptr, FS.get());
   EXPECT_EQ(3, NumDiagnostics);
 }
 
 TEST_F(VFSFromYAMLTest, MappedFiles) {
   IntrusiveRefCntPtr<DummyFileSystem> Lower(new DummyFileSystem());
+  Lower->addDirectory("//root/foo/bar");
   Lower->addRegularFile("//root/foo/bar/a");
   IntrusiveRefCntPtr<vfs::FileSystem> FS = getFromYAMLString(
       "{ 'roots': [\n"
       "{\n"
       "  'type': 'directory',\n"
       "  'name': '//root/',\n"
       "  'contents': [ {\n"
       "                  'type': 'file',\n"
       "                  'name': 'file1',\n"
       "                  'external-contents': '//root/foo/bar/a'\n"
       "                },\n"
       "                {\n"
       "                  'type': 'file',\n"
       "                  'name': 'file2',\n"
       "                  'external-contents': '//root/foo/b'\n"
+      "                },\n"
+      "                {\n"
+      "                  'type': 'directory-remap',\n"
+      "                  'name': 'mappeddir',\n"
+      "                  'external-contents': '//root/foo/bar'\n"
+      "                },\n"
+      "                {\n"
+      "                  'type': 'directory-remap',\n"
+      "                  'name': 'mappeddir2',\n"
+      "                  'use-external-name': false,\n"
+      "                  'external-contents': '//root/foo/bar'\n"
       "                }\n"
       "              ]\n"
       "}\n"
       "]\n"
       "}",
       Lower);
   ASSERT_TRUE(FS.get() != nullptr);
 
TEST_F(VFSFromYAMLTest, MappedFiles) {
   EXPECT_TRUE(OpenedS->IsVFSMapped);
 
   // directory
   S = O->status("//root/");
   ASSERT_FALSE(S.getError());
   EXPECT_TRUE(S->isDirectory());
   EXPECT_TRUE(S->equivalent(*O->status("//root/"))); // non-volatile UniqueID
 
+  // remapped directory
+  S = O->status("//root/mappeddir");
+  ASSERT_FALSE(S.getError());
+  EXPECT_TRUE(S->isDirectory());
+  EXPECT_TRUE(S->IsVFSMapped);
+  EXPECT_TRUE(S->equivalent(*O->status("//root/foo/bar")));
+
+  SLower = O->status("//root/foo/bar");
+  EXPECT_EQ("//root/foo/bar", SLower->getName());
+  EXPECT_TRUE(S->equivalent(*SLower));
+  EXPECT_FALSE(SLower->IsVFSMapped);
+
+  // file in remapped directory
+  S = O->status("//root/mappeddir/a");
+  ASSERT_FALSE(S.getError());
+  ASSERT_FALSE(S->isDirectory());
+  ASSERT_TRUE(S->IsVFSMapped);
+  ASSERT_EQ("//root/foo/bar/a", S->getName());
+
+  // file in remapped directory, with use-external-name=false
+  S = O->status("//root/mappeddir2/a");
+  ASSERT_FALSE(S.getError());
+  ASSERT_FALSE(S->isDirectory());
+  ASSERT_TRUE(S->IsVFSMapped);
+  ASSERT_EQ("//root/mappeddir2/a", S->getName());
+
+  // file contents in remapped directory
+  OpenedF = O->openFileForRead("//root/mappeddir/a");
+  ASSERT_FALSE(OpenedF.getError());
+  OpenedS = (*OpenedF)->status();
+  ASSERT_FALSE(OpenedS.getError());
+  EXPECT_EQ("//root/foo/bar/a", OpenedS->getName());
+  EXPECT_TRUE(OpenedS->IsVFSMapped);
+
+  // file contents in remapped directory, with use-external-name=false
+  OpenedF = O->openFileForRead("//root/mappeddir2/a");
+  ASSERT_FALSE(OpenedF.getError());
+  OpenedS = (*OpenedF)->status();
+  ASSERT_FALSE(OpenedS.getError());
+  EXPECT_EQ("//root/mappeddir2/a", OpenedS->getName());
+  EXPECT_TRUE(OpenedS->IsVFSMapped);
+
   // broken mapping
   EXPECT_EQ(O->status("//root/file2").getError(),
             llvm::errc::no_such_file_or_directory);
   EXPECT_EQ(0, NumDiagnostics);
 }
 
+TEST_F(VFSFromYAMLTest, MappedRoot) {
+  IntrusiveRefCntPtr<DummyFileSystem> Lower(new DummyFileSystem());
+  Lower->addDirectory("//root/foo/bar");
+  Lower->addRegularFile("//root/foo/bar/a");
+  IntrusiveRefCntPtr<vfs::FileSystem> FS =
+      getFromYAMLString("{ 'roots': [\n"
+                        "{\n"
+                        "  'type': 'directory-remap',\n"
+                        "  'name': '//mappedroot/',\n"
+                        "  'external-contents': '//root/foo/bar'\n"
+                        "}\n"
+                        "]\n"
+                        "}",
+                        Lower);
+  ASSERT_TRUE(FS.get() != nullptr);
+
+  IntrusiveRefCntPtr<vfs::OverlayFileSystem> O(
+      new vfs::OverlayFileSystem(Lower));
+  O->pushOverlay(FS);
+
+  // file
+  ErrorOr<vfs::Status> S = O->status("//mappedroot/a");
+  ASSERT_FALSE(S.getError());
+  EXPECT_EQ("//root/foo/bar/a", S->getName());
+  EXPECT_TRUE(S->IsVFSMapped);
+
+  ErrorOr<vfs::Status> SLower = O->status("//root/foo/bar/a");
+  EXPECT_EQ("//root/foo/bar/a", SLower->getName());
+  EXPECT_TRUE(S->equivalent(*SLower));
+  EXPECT_FALSE(SLower->IsVFSMapped);
+
+  // file after opening
+  auto OpenedF = O->openFileForRead("//mappedroot/a");
+  ASSERT_FALSE(OpenedF.getError());
+  auto OpenedS = (*OpenedF)->status();
+  ASSERT_FALSE(OpenedS.getError());
+  EXPECT_EQ("//root/foo/bar/a", OpenedS->getName());
+  EXPECT_TRUE(OpenedS->IsVFSMapped);
+
+  EXPECT_EQ(0, NumDiagnostics);
+}
+
+TEST_F(VFSFromYAMLTest, RemappedDirectoryOverlay) {
+  IntrusiveRefCntPtr<DummyFileSystem> Lower(new DummyFileSystem());
+  Lower->addDirectory("//root/foo");
+  Lower->addRegularFile("//root/foo/a");
+  Lower->addDirectory("//root/bar");
+  Lower->addRegularFile("//root/bar/b");
+  Lower->addRegularFile("//root/bar/c");
+  IntrusiveRefCntPtr<vfs::FileSystem> FS =
+      getFromYAMLString("{ 'roots': [\n"
+                        "{\n"
+                        "  'type': 'directory',\n"
+                        "  'name': '//root/',\n"
+                        "  'contents': [ {\n"
+                        "                  'type': 'directory-remap',\n"
+                        "                  'name': 'bar',\n"
+                        "                  'external-contents': '//root/foo'\n"
+                        "                }\n"
+                        "              ]\n"
+                        "}]}",
+                        Lower);
+  ASSERT_TRUE(FS.get() != nullptr);
+
+  IntrusiveRefCntPtr<vfs::OverlayFileSystem> O(
+      new vfs::OverlayFileSystem(Lower));
+  O->pushOverlay(FS);
+
+  ErrorOr<vfs::Status> S = O->status("//root/foo");
+  ASSERT_FALSE(S.getError());
+
+  ErrorOr<vfs::Status> SS = O->status("//root/bar");
+  ASSERT_FALSE(SS.getError());
+  EXPECT_TRUE(S->equivalent(*SS));
+
+  std::error_code EC;
+  checkContents(O->dir_begin("//root/bar", EC),
+                {"//root/foo/a", "//root/bar/b", "//root/bar/c"});
+
+  Lower->addRegularFile("//root/foo/b");
+  checkContents(O->dir_begin("//root/bar", EC),
+                {"//root/foo/a", "//root/foo/b", "//root/bar/c"});
+
+  EXPECT_EQ(0, NumDiagnostics);
+}
+
+TEST_F(VFSFromYAMLTest, RemappedDirectoryOverlayNoExternalNames) {
+  IntrusiveRefCntPtr<DummyFileSystem> Lower(new DummyFileSystem());
+  Lower->addDirectory("//root/foo");
+  Lower->addRegularFile("//root/foo/a");
+  Lower->addDirectory("//root/bar");
+  Lower->addRegularFile("//root/bar/b");
+  Lower->addRegularFile("//root/bar/c");
+  IntrusiveRefCntPtr<vfs::FileSystem> FS =
+      getFromYAMLString("{ 'use-external-names': false,\n"
+                        "  'roots': [\n"
+                        "{\n"
+                        "  'type': 'directory',\n"
+                        "  'name': '//root/',\n"
+                        "  'contents': [ {\n"
+                        "                  'type': 'directory-remap',\n"
+                        "                  'name': 'bar',\n"
+                        "                  'external-contents': '//root/foo'\n"
+                        "                }\n"
+                        "              ]\n"
+                        "}]}",
+                        Lower);
+  ASSERT_TRUE(FS.get() != nullptr);
+
+  ErrorOr<vfs::Status> S = FS->status("//root/foo");
+  ASSERT_FALSE(S.getError());
+
+  ErrorOr<vfs::Status> SS = FS->status("//root/bar");
+  ASSERT_FALSE(SS.getError());
+  EXPECT_TRUE(S->equivalent(*SS));
+
+  std::error_code EC;
+  checkContents(FS->dir_begin("//root/bar", EC),
+                {"//root/bar/a", "//root/bar/b", "//root/bar/c"});
+
+  Lower->addRegularFile("//root/foo/b");
+  checkContents(FS->dir_begin("//root/bar", EC),
+                {"//root/bar/a", "//root/bar/b", "//root/bar/c"});
+
+  EXPECT_EQ(0, NumDiagnostics);
+}
+
+TEST_F(VFSFromYAMLTest, RemappedDirectoryOverlayNoFallthrough) {
+  IntrusiveRefCntPtr<DummyFileSystem> Lower(new DummyFileSystem());
+  Lower->addDirectory("//root/foo");
+  Lower->addRegularFile("//root/foo/a");
+  Lower->addDirectory("//root/bar");
+  Lower->addRegularFile("//root/bar/b");
+  Lower->addRegularFile("//root/bar/c");
+  IntrusiveRefCntPtr<vfs::FileSystem> FS =
+      getFromYAMLString("{ 'fallthrough': false,\n"
+                        "  'roots': [\n"
+                        "{\n"
+                        "  'type': 'directory',\n"
+                        "  'name': '//root/',\n"
+                        "  'contents': [ {\n"
+                        "                  'type': 'directory-remap',\n"
+                        "                  'name': 'bar',\n"
+                        "                  'external-contents': '//root/foo'\n"
+                        "                }\n"
+                        "              ]\n"
+                        "}]}",
+                        Lower);
+  ASSERT_TRUE(FS.get() != nullptr);
+
+  ErrorOr<vfs::Status> S = Lower->status("//root/foo");
+  ASSERT_FALSE(S.getError());
+
+  ErrorOr<vfs::Status> SS = FS->status("//root/bar");
+  ASSERT_FALSE(SS.getError());
+  EXPECT_TRUE(S->equivalent(*SS));
+
+  std::error_code EC;
+  checkContents(FS->dir_begin("//root/bar", EC), {"//root/foo/a"});
+
+  Lower->addRegularFile("//root/foo/b");
+  checkContents(FS->dir_begin("//root/bar", EC),
+                {"//root/foo/a", "//root/foo/b"});
+
+  EXPECT_EQ(0, NumDiagnostics);
+}
+
 TEST_F(VFSFromYAMLTest, CaseInsensitive) {
   IntrusiveRefCntPtr<DummyFileSystem> Lower(new DummyFileSystem());
   Lower->addRegularFile("//root/foo/bar/a");
   IntrusiveRefCntPtr<vfs::FileSystem> FS = getFromYAMLString(
       "{ 'case-sensitive': 'false',\n"
       "  'roots': [\n"
       "{\n"
       "  'type': 'directory',\n"
TEST_F(VFSFromYAMLTest, IllegalVFSFile) {
 
   // bad version number
   FS = getFromYAMLRawString("{ 'version':'foo', 'roots':[] }", Lower);
   EXPECT_EQ(nullptr, FS.get());
   FS = getFromYAMLRawString("{ 'version':-1, 'roots':[] }", Lower);
   EXPECT_EQ(nullptr, FS.get());
   FS = getFromYAMLRawString("{ 'version':100000, 'roots':[] }", Lower);
   EXPECT_EQ(nullptr, FS.get());
-  EXPECT_EQ(24, NumDiagnostics);
+
+  // both 'external-contents' and 'contents' specified
+  Lower->addDirectory("//root/external/dir");
+  FS = getFromYAMLString(
+      "{ 'roots':[ \n"
+      "{ 'type': 'directory', 'name': '//root/A', 'contents': [],\n"
+      "  'external-contents': '//root/external/dir'}]}",
+      Lower);
+  EXPECT_EQ(nullptr, FS.get());
+
+  // 'directory-remap' with 'contents'
+  FS = getFromYAMLString(
+      "{ 'roots':[ \n"
+      "{ 'type': 'directory-remap', 'name': '//root/A', 'contents': [] }]}",
+      Lower);
+  EXPECT_EQ(nullptr, FS.get());
+
+  EXPECT_EQ(26, NumDiagnostics);
 }
 
 TEST_F(VFSFromYAMLTest, UseExternalName) {
   IntrusiveRefCntPtr<DummyFileSystem> Lower(new DummyFileSystem());
   Lower->addRegularFile("//root/external/file");
 
   IntrusiveRefCntPtr<vfs::FileSystem> FS =
       getFromYAMLString("{ 'roots': [\n"