Added Support for StatOnly Files in VFS.

[usaxena95]

Summary

Some files are only Statted by Clang and not read. Clang mostly uses
them for checking the existence of some files and in rare cases uses the
value of the Status to proceed further (for example while loading
module files, it checks the sizes of some files).

This change adds support to represent files that are supposed to be only
statted by Clang.

[sammccall]

Could you give some more detail about what this is for?

• it sounds like this is for record-replay. In the replayed compilation, we assume the accessed files are the same, is it safe to assume we never read files we previously stat()ed? What are these files in practice?
• it seems like this could also be achieved with an overlayFS adding a simple specialized FS that only provides the stat-only files. If this is a relatively niche feature, cramming it into InMemoryFileSystem may not be the best option. Any thoughts on the tradeoff here?

include/clang/Basic/VirtualFileSystem.h

414

Please add some motivation here, e.g. "such files are useful when replaying a recorded compilation, if the original compilation never read the file contents"

lib/Basic/VirtualFileSystem.cpp

495

can't you just represent this by a null buffer?

[kbobyrev]

lib/Basic/VirtualFileSystem.cpp

2097

This used llvm::sort(Container &&C, Compare Comp) before, I would think that the range-based API is the preferred one. There's also a cleanup patch migrating STL-like calls to idiomatic LLVM's STL Extension API: D52576.

[usaxena95]

it sounds like this is for record-replay.

Yes this is for record-replay purpose.

What are these files in practice?

During loading a modules the files that were required to make the module are stat()ed and their sizes are used to verify that the module is still valid. There are occurrences of more stat() only files (apart from the use in modules) which I am not aware of.

In the replayed compilation, we assume the accessed files are the same, is it safe to assume we never read files we previously stat()ed?

I would call it a requirement instead of an assumption. The replay must be exactly the same (even the file stats and reads). If Clang reads the file in replay which was only stat()ed during compilation, it indicates non-determinism or something wrong (in clang or FS).
We currently deal with such files by adding empty buffers for them based on this assumption/requirement only.

it seems like this could also be achieved with an overlayFS adding a simple specialized FS that only provides the stat-only files. If this is a relatively niche feature, cramming it into InMemoryFileSystem may not be the best option. Any thoughts on the tradeoff here?

We can reuse the InMemoryFS to make another FS which just serves stat-only files. A tradeoff I can think of is: we might need to store a random buffer of required size to make final status of the file correct.

[usaxena95]

lib/Basic/VirtualFileSystem.cpp

2097

I think these are some very recent changes and does not belong to my patch. Will sync these. Thanks for pointing out.

[ilya-biryukov]

I would call it a requirement instead of an assumption. The replay must be exactly the same (even the file stats and reads). If Clang reads the file in replay which was only stat()ed during compilation, it indicates non-determinism or something wrong (in clang or FS).
We currently deal with such files by adding empty buffers for them based on this assumption/requirement only.

Does adding a flag to clang to skip the module inputs checks seems feasible/reasonable? +1 to Sam's comments, the mode for FS that has the file sizes, but not the file contents seems too specific for the use-case at hand, does not fit well into the design.

include/clang/Basic/VirtualFileSystem.h

class InMemoryFileSystem : public FileSystem {
 
   /// If HardLinkTarget is non-null, a hardlink is created to the To path which
   /// must be a file. If it is null then it adds the file as the public addFile.
   bool addFile(const Twine &Path, time_t ModificationTime,
                std::unique_ptr<llvm::MemoryBuffer> Buffer,
                Optional<uint32_t> User, Optional<uint32_t> Group,
                Optional<llvm::sys::fs::file_type> Type,
                Optional<llvm::sys::fs::perms> Perms,
-               const detail::InMemoryFile *HardLinkTarget);
+               const detail::InMemoryFile *HardLinkTarget,
+               Optional<size_t> StatOnlyFileSize=None);
 
 public:
   explicit InMemoryFileSystem(bool UseNormalizedPaths = true);
   ~InMemoryFileSystem() override;
 
   /// Add a file containing a buffer or a directory to the VFS with a
   /// path. The VFS owns the buffer.  If present, User, Group, Type
   /// and Perms apply to the newly-created file or directory.
   /// \return true if the file or directory was successfully added,
   /// false if the file or directory already exists in the file system with
   /// different contents.
   bool addFile(const Twine &Path, time_t ModificationTime,
                std::unique_ptr<llvm::MemoryBuffer> Buffer,
                Optional<uint32_t> User = None, Optional<uint32_t> Group = None,
                Optional<llvm::sys::fs::file_type> Type = None,
                Optional<llvm::sys::fs::perms> Perms = None);
 
+  // Add a file with the given size but with no contents. The buffer of this
+  // file must never be requested.
+  /// \return true if the file successfully added, false if the file already
+  /// exists in the file system with different size or is not a StatOnlyFile.
+  bool addStatOnlyFile(const Twine &Path, time_t ModificationTime,
+                           size_t Size, Optional<uint32_t> User = None,
+                           Optional<uint32_t> Group = None,
+                           Optional<llvm::sys::fs::file_type> Type = None,
+                           Optional<llvm::sys::fs::perms> Perms = None);
+
   /// Add a hard link to a file.
   /// Here hard links are not intended to be fully equivalent to the classical
   /// filesystem. Both the hard link and the file share the same buffer and
   /// status (and thus have the same UniqueID). Because of this there is no way
   /// to distinguish between the link and the file after the link has been
   /// added.
   ///
   /// The To path must be an existing file or a hardlink. The From file must not

lib/Basic/VirtualFileSystem.cpp

public:
   StringRef getFileName() const { return FileName; }
   InMemoryNodeKind getKind() const { return Kind; }
   virtual std::string toString(unsigned Indent) const = 0;
 };
 
 class InMemoryFile : public InMemoryNode {
   Status Stat;
   std::unique_ptr<llvm::MemoryBuffer> Buffer;
+  bool StatOnlyFile;
 
 public:
-  InMemoryFile(Status Stat, std::unique_ptr<llvm::MemoryBuffer> Buffer)
+  InMemoryFile(Status Stat, std::unique_ptr<llvm::MemoryBuffer> Buffer,
+               bool StatOnlyFile)
       : InMemoryNode(Stat.getName(), IME_File), Stat(std::move(Stat)),
-        Buffer(std::move(Buffer)) {}
+        Buffer(std::move(Buffer)), StatOnlyFile(StatOnlyFile) {}
 
   /// Return the \p Status for this node. \p RequestedName should be the name
   /// through which the caller referred to this node. It will override
   /// \p Status::Name in the return value, to mimic the behavior of \p RealFile.
   Status getStatus(StringRef RequestedName) const {
     return Status::copyWithNewName(Stat, RequestedName);
   }
-  llvm::MemoryBuffer *getBuffer() const { return Buffer.get(); }
+  llvm::MemoryBuffer *getBuffer() const {
+    assert(!StatOnlyFile && "Cannot get buffer for StatOnlyFile");
+    return Buffer.get();
+  }
 
+  bool IsStatOnlyFile() const { return StatOnlyFile; }
   std::string toString(unsigned Indent) const override {
     return (std::string(Indent, ' ') + Stat.getName() + "\n").str();
   }
 
   static bool classof(const InMemoryNode *N) {
     return N->getKind() == IME_File;
   }
 };

 }
 
 bool InMemoryFileSystem::addFile(const Twine &P, time_t ModificationTime,
                                  std::unique_ptr<llvm::MemoryBuffer> Buffer,
                                  Optional<uint32_t> User,
                                  Optional<uint32_t> Group,
                                  Optional<llvm::sys::fs::file_type> Type,
                                  Optional<llvm::sys::fs::perms> Perms,
-                                 const detail::InMemoryFile *HardLinkTarget) {
+                                 const detail::InMemoryFile *HardLinkTarget,
+                                 Optional<size_t> StatOnlyFileSize) {
   SmallString<128> Path;
   P.toVector(Path);
 
   // Fix up relative paths. This just prepends the current working directory.
   std::error_code EC = makeAbsolute(Path);
   assert(!EC);
   (void)EC;
 
   if (useNormalizedPaths())
     llvm::sys::path::remove_dots(Path, /*remove_dot_dot=*/true);
 
   if (Path.empty())
     return false;
 
   detail::InMemoryDirectory *Dir = Root.get();
   auto I = llvm::sys::path::begin(Path), E = sys::path::end(Path);
   const auto ResolvedUser = User.getValueOr(0);
   const auto ResolvedGroup = Group.getValueOr(0);
   const auto ResolvedType = Type.getValueOr(sys::fs::file_type::regular_file);
   const auto ResolvedPerms = Perms.getValueOr(sys::fs::all_all);
+  assert(!(StatOnlyFileSize.hasValue() && Buffer) &&
+         "StatOnlyFile cannot have a buffer");
   assert(!(HardLinkTarget && Buffer) && "HardLink cannot have a buffer");
+  const auto ResolvedFileSize =
+      StatOnlyFileSize.getValueOr(Buffer ? Buffer->getBufferSize() : 0);
   // Any intermediate directories we create should be accessible by
   // the owner, even if Perms says otherwise for the final path.
   const auto NewDirectoryPerms = ResolvedPerms | sys::fs::owner_all;
   while (true) {
     StringRef Name = *I;
     detail::InMemoryNode *Node = Dir->getChild(Name);
     ++I;
     if (!Node) {
       if (I == E) {
         // End of the path.
         std::unique_ptr<detail::InMemoryNode> Child;
         if (HardLinkTarget)
           Child.reset(new detail::InMemoryHardLink(P.str(), *HardLinkTarget));
         else {
           // Create a new file or directory.
           Status Stat(P.str(), getNextVirtualUniqueID(),
                       llvm::sys::toTimePoint(ModificationTime), ResolvedUser,
-                      ResolvedGroup, Buffer->getBufferSize(), ResolvedType,
+                      ResolvedGroup, ResolvedFileSize, ResolvedType,
                       ResolvedPerms);
           if (ResolvedType == sys::fs::file_type::directory_file) {
             Child.reset(new detail::InMemoryDirectory(std::move(Stat)));
           } else {
-            Child.reset(
-                new detail::InMemoryFile(std::move(Stat), std::move(Buffer)));
+            // Create a new file.
+            Child.reset(new detail::InMemoryFile(std::move(Stat),
+                                                 std::move(Buffer),
+                                                 StatOnlyFileSize.hasValue()));
           }
         }
         Dir->addChild(Name, std::move(Child));
         return true;
       }
 
       // Create a new directory. Use the path up to here.
       Status Stat(
if (auto *NewDir = dyn_cast<detail::InMemoryDirectory>(Node)) {
               isa<detail::InMemoryHardLink>(Node)) &&
              "Must be either file, hardlink or directory!");
 
       // Trying to insert a directory in place of a file.
       if (I != E)
         return false;
 
       // Return false only if the new file is different from the existing one.
+      const detail::InMemoryFile* File;
       if (auto Link = dyn_cast<detail::InMemoryHardLink>(Node)) {
-        return Link->getResolvedFile().getBuffer()->getBuffer() ==
-               Buffer->getBuffer();
+        File = &Link->getResolvedFile();
+      } else {
+        File = cast<detail::InMemoryFile>(Node);
+      }
+      if (File->IsStatOnlyFile() != StatOnlyFileSize.hasValue()) {
+        return false;
+      }
+      if (StatOnlyFileSize.hasValue()) {
+        return File->getStatus(Path).getSize() == StatOnlyFileSize.getValue();
       }
-      return cast<detail::InMemoryFile>(Node)->getBuffer()->getBuffer() ==
-             Buffer->getBuffer();
+      return File->getBuffer()->getBuffer() == Buffer->getBuffer();
     }
   }
 }
 
+bool InMemoryFileSystem::addStatOnlyFile(
+    const Twine &Path, time_t ModificationTime, size_t Size,
+    Optional<uint32_t> User, Optional<uint32_t> Group,
+    Optional<llvm::sys::fs::file_type> Type,
+    Optional<llvm::sys::fs::perms> Perms) {
+  return addFile(Path, ModificationTime,
+                 /*Buffer=*/nullptr, User, Group, Type, Perms,
+                 /*HardLinkTarget=*/nullptr, Size);
+}
+
 bool InMemoryFileSystem::addFile(const Twine &P, time_t ModificationTime,
                                  std::unique_ptr<llvm::MemoryBuffer> Buffer,
                                  Optional<uint32_t> User,
                                  Optional<uint32_t> Group,
                                  Optional<llvm::sys::fs::file_type> Type,
                                  Optional<llvm::sys::fs::perms> Perms) {
   return addFile(P, ModificationTime, std::move(Buffer), User, Group, Type,
                  Perms, /*HardLinkTarget=*/nullptr);
if (!Entries.empty()) {
     OS << "\n";
   }
 
   OS << "  ]\n"
      << "}\n";
 }
 
 void YAMLVFSWriter::write(llvm::raw_ostream &OS) {
-  llvm::sort(Mappings, [](const YAMLVFSEntry &LHS, const YAMLVFSEntry &RHS) {
+  llvm::sort(Mappings.begin(), Mappings.end(),
+             [](const YAMLVFSEntry &LHS, const YAMLVFSEntry &RHS) {
     return LHS.VPath < RHS.VPath;
   });
 
   JSONWriter(OS).write(Mappings, UseExternalNames, IsCaseSensitive,
                        IsOverlayRelative, IgnoreNonExistentContents,
                        OverlayDir);
 }
 

unittests/Basic/VirtualFileSystemTest.cpp

static void checkContents(DirIter I, ArrayRef<StringRef> ExpectedOut) {
   SmallVector<StringRef, 4> Expected(ExpectedOut.begin(), ExpectedOut.end());
   SmallVector<std::string, 4> InputToCheck;
 
   // Do not rely on iteration order to check for contents, sort both
   // content vectors before comparison.
   for (DirIter E; !EC && I != E; I.increment(EC))
     InputToCheck.push_back(I->path());
 
-  llvm::sort(InputToCheck);
-  llvm::sort(Expected);
+  llvm::sort(InputToCheck.begin(), InputToCheck.end());
+  llvm::sort(Expected.begin(), Expected.end());
   EXPECT_EQ(InputToCheck.size(), Expected.size());
 
   unsigned LastElt = std::min(InputToCheck.size(), Expected.size());
   for (unsigned Idx = 0; Idx != LastElt; ++Idx)
     EXPECT_EQ(StringRef(InputToCheck[Idx]), Expected[Idx]);
 }
 
 TEST(VirtualFileSystemTest, OverlayIteration) {
TEST_F(InMemoryFileSystemTest, AddHardLinkUnderAFile) {
   FS.addFile("/a/b", 0, MemoryBuffer::getMemBuffer(CommonContent));
   FS.addFile("/c/d", 0, MemoryBuffer::getMemBuffer(CommonContent));
   EXPECT_FALSE(FS.addHardLink("/c/d/e", "/a/b"));
 }
 
 TEST_F(InMemoryFileSystemTest, RecursiveIterationWithHardLink) {
   std::error_code EC;
   FS.addFile("/a/b", 0, MemoryBuffer::getMemBuffer("content string"));
+  FS.addStatOnlyFile("/a/d", 0, 10);
   EXPECT_TRUE(FS.addHardLink("/c/d", "/a/b"));
   auto I = vfs::recursive_directory_iterator(FS, "/", EC);
   ASSERT_FALSE(EC);
   std::vector<std::string> Nodes;
   for (auto E = vfs::recursive_directory_iterator(); !EC && I != E;
        I.increment(EC)) {
     Nodes.push_back(getPosixPath(I->path()));
   }
-  EXPECT_THAT(Nodes, testing::UnorderedElementsAre("/a", "/a/b", "/c", "/c/d"));
+  EXPECT_THAT(Nodes, testing::UnorderedElementsAre("/a", "/a/d", "/a/b",
+                                                   "/c", "/c/d"));
+}
+
+TEST_F(InMemoryFileSystemTest, AddStatOnlyFileWithCorrectSize) {
+  FS.addStatOnlyFile("/a/b", 0, 10);
+  auto Stat = FS.status("/a/b");
+  EXPECT_EQ(Stat.get().getSize(), 10);
+}
+
+TEST_F(InMemoryFileSystemTest, AddMultipleFilesUnderStatOnlyFile) {
+  FS.addStatOnlyFile("/a/b", 0, 10);
+  EXPECT_FALSE(FS.addStatOnlyFile("/a/b", 0, 11));
+  EXPECT_TRUE(FS.addStatOnlyFile("/a/b", 0, 10));
+  EXPECT_FALSE(FS.addFile("/a/b", 0, MemoryBuffer::getMemBuffer("content")));
+}
+
+TEST_F(InMemoryFileSystemTest, DeathWhenBufferRequestedForStatOnlyFile) {
+  FS.addStatOnlyFile("/a/b", 0, 10);
+  EXPECT_DEATH(FS.getBufferForFile("/a/b"),
+               "Cannot get buffer for StatOnlyFile");
+}
+
+TEST_F(InMemoryFileSystemTest, AddHardLinksToStatOnlyFile) {
+  FS.addStatOnlyFile("/target", 0, 10);
+  FS.addHardLink("/link1", "/target");
+  FS.addHardLink("/link2", "/link1");
+  EXPECT_THAT("/link1", IsHardLinkTo(&FS, "/target"));
+  EXPECT_THAT("/link2", IsHardLinkTo(&FS, "/target"));
+}
+
+TEST_F(InMemoryFileSystemTest, AddFileUnderHardLinkToStatOnlyFile) {
+  FS.addStatOnlyFile("/target", 0, 10);
+  FS.addHardLink("/link", "/target");
+  EXPECT_FALSE(FS.addFile("/link",  0, MemoryBuffer::getMemBuffer("content")));
 }
 
 // NOTE: in the tests below, we use '//root/' as our root directory, since it is
 // a legal *absolute* path on Windows as well as *nix.
 class VFSFromYAMLTest : public ::testing::Test {
 public:
   int NumDiagnostics;