Mark the file entry invalid, until reread. Invalidate SLocEntry cache, readd it on reread. Do not use translateFile, because it pulls in parts of the pch.

[tapaswenipathak]

Summary

Mark the file entry invalid, until reread.

This patch marks the file entry invalid, until reread. Invalidates SLocEntry
cache, readd it on reread. Do not use translateFile, because it pulls
in parts of the pch.

This adds an invalidateCache success test in filemanager and
sourcemanager, using #define public private to access private member in
the test.

Some more related information is available here:
https://github.com/vgvassilev/clang/pull/2.

Co-authored-by: Vassil Vassilev <[e-mail]>

[v.g.vassilev]

Just to add that the invalidateCache is important for cling and clang-repl where we do something like:

clang-repl> #include "file_with_error.h"
// error is printed, we edit the file and include it again:
clang-repl> #include "file_with_error.h"

I am not sure if we can write such a test for clang-repl easily using the lit infrastructure...

[tapaswenipathak]

fix for build failure:

cmdline: git reset --hard
stderr: 'fatal: Unable to create '/var/lib/buildkite-agent/builds/llvm-project-fork/.git/index.lock': File exists.

https://buildkite.com/llvm-project/diff-checks/builds/110904#01818451-4c05-49bb-9f93-ea6d0480f9ec

could be temporary, or could be because i didn't run a git pull for a few days.

[tapaswenipathak]

omit an unrelated change.

[tapaswenipathak]

git clang-format HEAD~1

[v.g.vassilev]

After landing https://reviews.llvm.org/D126682 we might be able to write a test in clang-repl such as:

cpp
clang-repl> #include <iostream>
clang-repl> #include <fstream>
clang-repl>using namespace std;
clang-repl> int write_str (const char* str) {
  ofstream myfile;
  myfile.open ("a.h");
  myfile << str<<"\n";
  myfile.close();
  return 0;
}
clang-repl> auto r1 = write_str("int i = 42");
clang-repl> extern "C" int printf(const char*,...);
clang-repl> #include "a.h"
clang-repl> auto r2 = printf("i=%d\n", i);
clang-repl>%undo
clang-repl>%undo 
clang-repl> auto r3 = write_str("int i = 0");
clang-repl> #include "a.h"
clang-repl> auto r2 = printf("i=%d\n", i);
// Here we should print `i=0`.

[vsapsai]

Hmm, I'm concerned with the pieces added purely for testing. Specifically, FileEntriesToReread and TestHelper friend functions. Need to think how else we can test it.

Do you intend to support only the error cases like

clang-repl> #include "file_with_error.h"
// error is printed, we edit the file and include it again:
clang-repl> #include "file_with_error.h"

or do you want to handle re-including files? Something like

clang-repl> #include "experiments.h"
// edit the file and include it again:
clang-repl> #include "experiments.h"

Asking because maybe in the error case we commit to some state too eagerly and fixing that sticky eagerness is another option (just guessing, have no idea if it is an actual option and if it is "better").

[v.g.vassilev]

In D126266#3629145, @vsapsai wrote:

Hmm, I'm concerned with the pieces added purely for testing. Specifically, FileEntriesToReread and TestHelper friend functions. Need to think how else we can test it.

I am not thrilled about that either. We have discussed this approach here https://reviews.llvm.org/D126183 and we have past experience going that route. I would be happy to drop this test in favor of the mentioned clang-repl test. Should be enough to cover the usecase.

Do you intend to support only the error cases like

clang-repl> #include "file_with_error.h"
// error is printed, we edit the file and include it again:
clang-repl> #include "file_with_error.h"

or do you want to handle re-including files? Something like

clang-repl> #include "experiments.h"
// edit the file and include it again:
clang-repl> #include "experiments.h"

Asking because maybe in the error case we commit to some state too eagerly and fixing that sticky eagerness is another option (just guessing, have no idea if it is an actual option and if it is "better").

We want both. The error case is somewhat easier to deal with. However, in case we have error in the logic in #include "experiments.h" we want to execute and then re-execute the new code.

[v.g.vassilev]

@tapaswenipathak, ping.

[v.g.vassilev]

@tapaswenipathak, ping...

include/clang/Basic/FileManager.h

   std::error_code getNoncachedStatValue(StringRef Path,
                                         llvm::vfs::Status &Result);
 
+  /// Remove the real file \p Entry from the cache.
+  void invalidateCache(FileEntry *Entry);
+
   /// If path is not absolute and FileSystemOptions set the working
   /// directory, the path is modified to be relative to the given
   /// working directory.

include/clang/Basic/SourceManager.h

 
   void clearIDTables();
 
-  void invalidateCache(const FileEntry* Entry);
+  void invalidateCache(FileID FID);
 
   /// Initialize this source manager suitably to replay the compilation
   /// described by \p Old. Requires that \p Old outlive \p *this.

lib/Basic/FileManager.cpp

   return std::error_code();
 }
 
-void FileManager::invalidateCache(const FileEntry *Entry) {
+void FileManager::invalidateCache(FileEntry *Entry) {
   assert(Entry && "Cannot invalidate a NULL FileEntry");
   FileEntriesToReread.insert(Entry->getLastRef());
   Entry->IsValid = false;

lib/Basic/SourceManager.cpp

   return false;
 }
 
-void SourceManager::invalidateCache(const FileEntry* Entry) {
+void SourceManager::invalidateCache(FileID FID) {
+  const FileEntry *Entry = getFileEntryForID(FID);
+  if (!Entry)
+    return;
   if (ContentCache *&E = FileInfos[Entry]) {
-    E->replaceBuffer(0, /*free*/ true);
+    E->setBuffer(nullptr);
     E = 0;
   }
-  getFileManager().invalidateCache(Entry);
+  if (!FID.isInvalid()) {
+    const SrcMgr::SLocEntry &SLocE = getSLocEntry(FID);
+    if (SLocE.isFile()) {
+      SrcMgr::ContentCache &CC =
+          const_cast<SrcMgr::ContentCache &>(SLocE.getFile().getContentCache());
+      CC.setBuffer(nullptr);
+    }
+  }
+  getFileManager().invalidateCache(const_cast<FileEntry *>(Entry));
 }
 
 void SourceManager::initializeForReplay(const SourceManager &Old) {

unittests/Basic/FileManagerTest.cpp

 //
 //===----------------------------------------------------------------------===//
 
+#define private public
 #include "clang/Basic/FileManager.h"
+#undef private
 #include "clang/Basic/FileSystemOptions.h"
 #include "clang/Basic/FileSystemStatCache.h"
 #include "llvm/ADT/STLExtras.h"

   FileManager manager;
 };
 
+// If file entry valid, mark the file entry invalid, until reread.
+TEST_F(FileManagerTest, invalidateCacheSuccess) {
+  manager.setStatCache(std::make_unique<FakeStatCache>());
+
+  manager.getVirtualFile("file.cpp", 100, 0);
+  auto file = manager.getFile("file.cpp");
+
+  // Check for file null assertion success
+  manager.InvalidateCache(NULL);
+
+  manager.InvalidateCache(*file);
+  ASSERT_TRUE(manager.FileEntriesToReread);
+
+  ASSERT_FALSE((*file)->isValid());
+}
+
 // When a virtual file is added, its getDir() field is set correctly
 // (not NULL, correct name).
 TEST_F(FileManagerTest, getVirtualFileSetsTheDirFieldCorrectly) {

unittests/Basic/SourceManagerTest.cpp

 #include "clang/Basic/SourceManager.h"
 #include "clang/Basic/Diagnostic.h"
 #include "clang/Basic/DiagnosticOptions.h"
+#define private public
 #include "clang/Basic/FileManager.h"
+#undef private
 #include "clang/Basic/LangOptions.h"
 #include "clang/Basic/TargetInfo.h"
 #include "clang/Basic/TargetOptions.h"

   IntrusiveRefCntPtr<TargetInfo> Target;
 };
 
+// Test for invalidate cache success, making the file entry invalid, until
+// reread
+TEST_F(SourceManagerTest, invalidateCacheSuccess) {
+  const char *Source = "int x;";
+
+  std::unique_ptr<llvm::MemoryBuffer> Buf =
+      llvm::MemoryBuffer::getMemBuffer(source);
+  const FileEntry *SourceFile =
+      FileMgr.getVirtualFile("mainFile.cpp", Buf->getBufferSize(), 0);
+
+  FileID mainFileID = SourceMgr.createFileID(std::move(Buf));
+  SourceMgr.overrideFileContents(SourceFile, std::move(Buf));
+  SourceMgr.setMainFileID(mainFileID);
+  SourceMgr::invalidateCache(mainFileID);
+
+  EXPECT_FALSE(SrcMgr::ContentCache);
+  EXPECT_TRUE(SourceFile->FileEntriesToReread);
+  EXPECT_FALSE(SourceFile->isValid());
+}
+
 TEST_F(SourceManagerTest, isBeforeInTranslationUnit) {
   const char *source =
     "#define M(x) [x]\n"