Support case insensitive header searches for MSVCCompat

[abdulras]

Summary

When the compiler is placed into MSVC compatibility, add a fallback mechanism
for supporting case insensitive header search. This is important when dealing
with case sensitive filesystems and building for Windows. In particular, the
Windows SDK in many places mixes up cases for filenames as well as directories.

In general, this codepath does not affect the compilation as it is a simple
boolean check. If MSVC compatibility is enabled, perform an additional stat.
If the path is inaccessible, do a case insensitive match over the contents of
the directory.

If the underlying filesystem is case-insensitive, then the stat will succeed and
the search will not be required.

[thakis]

Does cl.exe to this if you're on a case-sensitive file system? Why are you
using a case-sensitive file system if you want case-insensitive behavior?

[abdulras]

One of the nice things about clang is that it is inherently a cross-compiler :-). Not all file systems are case-insensitive, and unfortunately, as I mentioned in the commit message, the Windows SDK does get the case wrong in many cases. This is an attempt to make the cross-compiler work properly with MS compatibility irrespective of the environment where it is being executed.

[thakis]

The compiler won't be the only place with issues like this though, no? Say
you have a generated header "header.h" and some file accidentally includes
it as "Header.h", then the build system now won't have a correct dependency
edge. There's probably more examples.

Can't you just create a case-insensitive file system even in cross mode?

[majnemer]

Modulo needing some style fixes, this looks fine. However, I'd prefer
for @rsmith look at this at least to make sure it plays nice with modules.

[rsmith]

Have you considered using something ilke ciopfs? =)

This patch needs testcases.

I also suspect that some of this belongs in LLVM's Support library, rather than here, but that can wait until we've figured out exactly what we want.

include/clang/Lex/DirectoryLookup.h

66

Typo of "Microsoft"

lib/Frontend/InitHeaderSearch.cpp

679

We don't want to pay the cost of this if the underlying file system is already case-insensitive. On Windows, GetVolumeInformation can be used to determine this. I don't think there's a good way to determine it for Linux, sadly.

(We should also use this in FileManager rather than making an assumption that Windows is case-insensitive and nothing else is there.)

lib/Lex/HeaderSearch.cpp

36–42

Do you need to compare anything other than the last component here?

257–265

Please factor this out. Should you do something better with EC than ignoring it?

[abdulras]

include/clang/Lex/DirectoryLookup.h

66

Good eye.

lib/Frontend/InitHeaderSearch.cpp

679

Agreed. What volume would you say that we query that on? We could be compiling on a volume different from the compiler. The headers could come from various volumes as well.

lib/Lex/HeaderSearch.cpp

36–42

for a multi-component file path, a non-terminal component may mismatch. If a multi-component path has a matching terminal-component, that would also get incorrectly included otherwise.

257–265

I dont think I can do much with EC. I suppose I could print out a warning (but that seems like it would clutter the output). But, I will pull out the loop into a function.

[rsmith]

lib/Frontend/InitHeaderSearch.cpp

679

We should presumably make the decision on a per-directory basis. I'd expect it can change across junction points, not just across volumes.

lib/Lex/HeaderSearch.cpp

36–42

But directory_iterator only walks a single level at a time, right? If you meant to perform a recursive walk looking for missing directory components, not just the last level, then I think that

1. you're missing the code to do that (or maybe I've just overlooked how you're achieving that), and
2. you still only want to compare the last component at each stage in your tree walk (rather than wasting time walking through the files in directories that don't match).

257–265

I'd be inclined to think that a FatalError is the right way to respond to a directory operation that fails, unless we have a reason to continue past one? Hmm, maybe we want to ignore permissions problems?

[benlangmuir]

Does the case-sensitivity behaviour really only apply to headers found in header search? If not, then how about putting this in the virtual file system layer, or at least FileManager?

Right now this doesn't cover the 'header' declarations in module map files (see ModuleMap.cpp, parseHeaderDecl).

include/clang/Lex/DirectoryLookup.h

   /// \brief Whether we've performed an exhaustive search for module maps
   /// within the subdirectories of this directory.
   unsigned SearchedAllModuleMaps : 1;
+
+  /// \brief Whether the lookup should be case insensitive.  This is required
+  /// for compatibility with case-sensitive file systems and Mcirosoft mode
+  /// which does not include headers with the correct case.
+  unsigned CaseInsensitive : 1;
   
 public:
   /// DirectoryLookup ctor - Note that this ctor *does not take ownership* of
   /// 'dir'.
   DirectoryLookup(const DirectoryEntry *dir, SrcMgr::CharacteristicKind DT,
-                  bool isFramework)
+                  bool isFramework, bool isCaseInsensitive)
     : DirCharacteristic(DT),
       LookupType(isFramework ? LT_Framework : LT_NormalDir),
-      IsIndexHeaderMap(false), SearchedAllModuleMaps(false) {
+      IsIndexHeaderMap(false), SearchedAllModuleMaps(false),
+      CaseInsensitive(isCaseInsensitive) {
     u.Dir = dir;
   }
 
   /// DirectoryLookup ctor - Note that this ctor *does not take ownership* of
   /// 'map'.
   DirectoryLookup(const HeaderMap *map, SrcMgr::CharacteristicKind DT,
-                  bool isIndexHeaderMap)
+                  bool isIndexHeaderMap, bool isCaseInsensitive)
     : DirCharacteristic(DT), LookupType(LT_HeaderMap),
-      IsIndexHeaderMap(isIndexHeaderMap), SearchedAllModuleMaps(false) {
+      IsIndexHeaderMap(isIndexHeaderMap), SearchedAllModuleMaps(false),
+      CaseInsensitive(isCaseInsensitive) {
     u.Map = map;
   }
 

include/clang/Lex/HeaderSearch.h

 
   bool EnabledModules;
 
+  /// \brief Whether we are performing case insensitive header searches
+  bool CaseInsensitive;
+
   // HeaderSearch doesn't support default or copy construction.
   HeaderSearch(const HeaderSearch&) LLVM_DELETED_FUNCTION;
   void operator=(const HeaderSearch&) LLVM_DELETED_FUNCTION;

lib/Frontend/InitHeaderSearch.cpp

   bool Verbose;
   std::string IncludeSysroot;
   bool HasSysroot;
+  bool CaseInsensitive;
 
 public:
 
-  InitHeaderSearch(HeaderSearch &HS, bool verbose, StringRef sysroot)
+  InitHeaderSearch(HeaderSearch &HS, bool verbose, StringRef sysroot,
+                   bool caseInsensitive)
     : Headers(HS), Verbose(verbose), IncludeSysroot(sysroot),
-      HasSysroot(!(sysroot.empty() || sysroot == "/")) {
+      HasSysroot(!(sysroot.empty() || sysroot == "/")),
+      CaseInsensitive(caseInsensitive) {
   }
 
   /// AddPath - Add the specified path to the specified group list, prefixing

 
   // If the directory exists, add it.
   if (const DirectoryEntry *DE = FM.getDirectory(MappedPathStr)) {
-    IncludePath.push_back(
-      std::make_pair(Group, DirectoryLookup(DE, Type, isFramework)));
+    IncludePath.push_back(std::make_pair(Group,
+                                         DirectoryLookup(DE, Type, isFramework,
+                                                         CaseInsensitive)));
     return;
   }
 

     if (const FileEntry *FE = FM.getFile(MappedPathStr)) {
       if (const HeaderMap *HM = Headers.CreateHeaderMap(FE)) {
         // It is a headermap, add it to the search path.
-        IncludePath.push_back(
-          std::make_pair(Group,
-                         DirectoryLookup(HM, Type, Group == IndexHeaderMap)));
+        IncludePath.push_back(std::make_pair(Group,
+                                             DirectoryLookup(HM, Type,
+                                                        Group == IndexHeaderMap,
+                                                             CaseInsensitive)));
         return;
       }
     }

                                      const HeaderSearchOptions &HSOpts,
                                      const LangOptions &Lang,
                                      const llvm::Triple &Triple) {
-  InitHeaderSearch Init(HS, HSOpts.Verbose, HSOpts.Sysroot);
+  InitHeaderSearch Init(HS, HSOpts.Verbose, HSOpts.Sysroot, Lang.MSVCCompat);
 
   // Add the user defined entries.
   for (unsigned i = 0, e = HSOpts.UserEntries.size(); i != e; ++i) {

lib/Lex/HeaderSearch.cpp

 #endif
 using namespace clang;
 
+static bool IsEquivalent(StringRef Path, StringRef LowerFilename) {
+  llvm::sys::path::reverse_iterator PI, PE, FI, FE;
+  const std::string LowerPath = Path.lower();
+
+  for (PI = llvm::sys::path::rbegin(LowerPath),
+       PE = llvm::sys::path::rend(LowerPath),
+       FI = llvm::sys::path::rbegin(LowerFilename),
+       FE = llvm::sys::path::rend(LowerFilename);
+       FI != FE && PI != PE; ++PI, ++FI)
+    if (FI->str() != PI->str())
+      return false;
+
+  return FI == FE;
+}
+
 const IdentifierInfo *
 HeaderFileInfo::getControllingMacro(ExternalIdentifierLookup *External) {
   if (ControllingMacro)

                            SourceManager &SourceMgr, DiagnosticsEngine &Diags,
                            const LangOptions &LangOpts,
                            const TargetInfo *Target)
-    : HSOpts(HSOpts), Diags(Diags), FileMgr(SourceMgr.getFileManager()),
-      FrameworkMap(64), ModMap(SourceMgr, Diags, LangOpts, Target, *this) {
+    FrameworkMap(64), ModMap(SourceMgr, Diags, LangOpts, Target, *this),
+    CaseInsensitive(LangOpts.MSVCCompat) {
   AngledDirIdx = 0;
   SystemDirIdx = 0;
   NoCurDirSearch = false;

     // Concatenate the requested file onto the directory.
     TmpDir = getDir()->getName();
     llvm::sys::path::append(TmpDir, Filename);
+    if (CaseInsensitive && !llvm::sys::fs::exists(TmpDir.str())) {
+      llvm::error_code EC;
+      std::string LowerFilename = Filename.lower();
+      for (llvm::sys::fs::directory_iterator DI(getDir()->getName(), EC), DE;
+           !EC && DI != DE; DI = DI.increment(EC)) {
+        if (IsEquivalent(DI->path(), LowerFilename)) {
+          TmpDir = DI->path();
+          break;
+        }
+      }
+    }
     if (SearchPath != NULL) {
       StringRef SearchPathRef(getDir()->getName());
       SearchPath->clear();

       RelativePath->clear();
       RelativePath->append(Filename.begin(), Filename.end());
     }
+    if (CaseInsensitive && !llvm::sys::fs::exists(Filename)) {
+      llvm::error_code EC;
+      std::string LowerFilename = Filename.lower();
+      for (llvm::sys::fs::directory_iterator
+                             DI(llvm::sys::path::parent_path(Filename), EC), DE;
+           !EC && DI != DE; DI = DI.increment(EC))
+        if (IsEquivalent(DI->path(), LowerFilename))
+          return FileMgr.getFile(DI->path(), /*openFile=*/true);
+    }
     // Otherwise, just return the file.
     return FileMgr.getFile(Filename, /*openFile=*/true);
   }

       TmpDir = Includer->getDir()->getName();
       TmpDir.push_back('/');
       TmpDir.append(Filename.begin(), Filename.end());
+      if (CaseInsensitive && !llvm::sys::fs::exists(TmpDir.str())) {
+        llvm::error_code EC;
+        std::string LowerFilename = Filename.lower();
+        for (llvm::sys::fs::directory_iterator
+                                      DI(Includer->getDir()->getName(), EC), DE;
+             !EC && DI != DE; DI = DI.increment(EC)) {
+          if (IsEquivalent(DI->path(), LowerFilename)) {
+            TmpDir = DI->path();
+            break;
+          }
+        }
+      }
       if (const FileEntry *FE =
               FileMgr.getFile(TmpDir.str(), /*openFile=*/true)) {
         // Leave CurDir unset.

lib/Lex/PPDirectives.cpp

       if (Callbacks->FileNotFound(Filename, RecoveryPath)) {
         if (const DirectoryEntry *DE = FileMgr.getDirectory(RecoveryPath)) {
           // Add the recovery path to the list of search paths.
-          DirectoryLookup DL(DE, SrcMgr::C_User, false);
+          DirectoryLookup DL(DE, SrcMgr::C_User, false, false);
           HeaderInfo.AddSearchPath(DL, isAngled);
           
           // Try the lookup again, skipping the cache.

unittests/Lex/PPCallbacksTest.cpp

       // Add header's parent path to search path.
       StringRef SearchPath = path::parent_path(HeaderPath);
       const DirectoryEntry *DE = FileMgr.getDirectory(SearchPath);
-      DirectoryLookup DL(DE, SrcMgr::C_User, false);
+      DirectoryLookup DL(DE, SrcMgr::C_User, false, false);
       HeaderInfo.AddSearchPath(DL, IsSystemHeader);
   }