[clang][deps] Ensure deterministic filename case

[jansvoboda11]

Summary

The dependency scanner can reuse single FileManager instance across multiple translation units. This may lead to non-deterministic output depending on which TU gets processed first.

One of the problems is that Clang uses DirectoryEntry::getName in the header search algorithm. This function returns the path that was first used to construct the (shared) entry in FileManager. Using DirectoryEntryRef::getName instead preserves the case as it was spelled out for the current "get directory entry" request.

rdar://90647508

[jansvoboda11]

Finish test description.

[dexonsmith]

LGTM.

Once we've removed the last use of DirectoryEntry::getName(), we should drop DirectoryEntry::getName() to avoid picking up other users. Perhaps DirectoryEntry itself could be made private to the FileManager.

The root cause is the fact that Clang is using DirectoryEntry::getName in the header search algorithm. This function returns the path that was first used to construct the (shared) entry in FileManager. Using DirectoryEntryRef::getName instead preserves the case as it was spelled out for the current "get directory entry" request.

For clarity, this is just one of various root causes of FileManager being unsound to reuse. Might be good to reword the commit message to avoid implying that it's the only one.

The obvious thing is that FileEntry is still used in a few places, especially in modules.

The non-obvious thing is that the "redirection hack" in FileManager::getFileRef(), which deals with a VFS remapping an external name, can cause future lookups to succeed that would previously have failed. @benlangmuir hit this recently when he was experimenting with NOT sharing the FileManager during implicit modules scanning; he can share the sequence of steps (I tried just now to remember them but I couldn't work it out).

IMO, we should turn off sharing the FileManager across multiple TUs (except as an experimental option) until we actually believe that it's sound. I'm not convinced there's a lot of benefit anyway:

• Stat failures need to be cleared between TUs
• Stat successes are cached by the dependency scanning filesystem
• The contents are (mostly? entirely?) bump-ptr allocated since https://reviews.llvm.org/D123144

[jansvoboda11]

Windows path separators...

clang/include/clang/Lex/DirectoryLookup.h

public:
   StringRef getName() const;
 
   /// getDir - Return the directory that this entry refers to.
   ///
   const DirectoryEntry *getDir() const {
     return isNormalDir() ? &u.Dir.getDirEntry() : nullptr;
   }
 
+  Optional<DirectoryEntryRef> getDirRef() const {
+    return isNormalDir() ? Optional<DirectoryEntryRef>(u.Dir) : None;
+  }
+
   /// getFrameworkDir - Return the directory that this framework refers to.
   ///
   const DirectoryEntry *getFrameworkDir() const {
     return isFramework() ? &u.Dir.getDirEntry() : nullptr;
   }
 
   Optional<DirectoryEntryRef> getFrameworkDirRef() const {
     return isFramework() ? Optional<DirectoryEntryRef>(u.Dir) : None;

clang/lib/Lex/HeaderSearch.cpp

Optional<FileEntryRef> DirectoryLookup::LookupFile(
     bool &IsInHeaderMap, SmallVectorImpl<char> &MappedName) const {
   InUserSpecifiedSystemFramework = false;
   IsInHeaderMap = false;
   MappedName.clear();
 
   SmallString<1024> TmpDir;
   if (isNormalDir()) {
     // Concatenate the requested file onto the directory.
-    TmpDir = getDir()->getName();
+    TmpDir = getDirRef()->getName();
     llvm::sys::path::append(TmpDir, Filename);
     if (SearchPath) {
-      StringRef SearchPathRef(getDir()->getName());
+      StringRef SearchPathRef(getDirRef()->getName());
       SearchPath->clear();
       SearchPath->append(SearchPathRef.begin(), SearchPathRef.end());
     }
     if (RelativePath) {
       RelativePath->clear();
       RelativePath->append(Filename.begin(), Filename.end());
     }
 

clang/test/ClangScanDeps/header-search-case-sensitivity.c

+// This test checks that reusing FileManager produces deterministic results on case-insensitive filesystems.
+
+// RUN: rm -rf %t
+// RUN: split-file %s %t
+
+//--- dir1/arm/lower.h
+//--- dir2/ARM/upper.h
+//--- t1.c
+#include "upper.h"
+//--- t2.c
+#include "arm/lower.h"
+
+//--- cdb.json.template
+[{
+  "directory": "DIR",
+  "command": "clang -fsyntax-only DIR/t1.c -I DIR/dir2/ARM -I DIR/dir1",
+  "file": "DIR/t1.c"
+},{
+  "directory": "DIR",
+  "command": "clang -fsyntax-only DIR/t2.c -I DIR/dir2     -I DIR/dir1",
+  "file": "DIR/t2.c"
+}]
+
+//--- cdb-rev.json.template
+[{
+  "directory": "DIR",
+  "command": "clang -fsyntax-only DIR/t2.c -I DIR/dir2     -I DIR/dir1",
+  "file": "DIR/t2.c"
+},{
+  "directory": "DIR",
+  "command": "clang -fsyntax-only DIR/t1.c -I DIR/dir2/ARM -I DIR/dir1",
+  "file": "DIR/t1.c"
+}]
+
+// RUN: sed -e "s|DIR|%/t|g" %t/cdb.json.template     > %t/cdb.json
+// RUN: sed -e "s|DIR|%/t|g" %t/cdb-rev.json.template > %t/cdb-rev.json
+
+// RUN: clang-scan-deps -compilation-database=%t/cdb.json     -format make -j 1 | sed 's:\\\\\?:/:g' | FileCheck %s
+
+// In the reversed case, Clang starts by scanning "t2.c". When looking up the "arm/lower.h" header,
+// the string is appended to "DIR/dir2". That file ("DIR/dir2/arm/lower.h") doesn't exist, but when
+// learning so, the FileManager stats and caches the parent directory ("DIR/dir2/arm"), using the
+// UID as the key.
+// When scanning "t1.c" later on, the "DIR/dir2/ARM" search directory is assigned the **same**
+// directory entry (with lowercase "arm"), since they share the UID on case-insensitive filesystems.
+// To preserve the correct case throughout the compiler for any file within that directory, it's
+// important to use the spelling actually used, not just the cached one.
+// RUN: clang-scan-deps -compilation-database=%t/cdb-rev.json -format make -j 1 | sed 's:\\\\\?:/:g' | FileCheck %s
+
+// CHECK: ARM/upper.h