New option -fwindows-filesystem, affecting #include paths.

[simon_tatham]

Summary

This option interposes a wrapper implementation of VirtualFileSystem
in front of the one in the CompilerInstance. The wrapper filesystem
differs from the standard one in that it tolerates backslashes as a
path separator even if the native path syntax doesn't, and also looks
up filenames case-insensitively even if the native file system is
case-sensitive.

This is intended to be useful to anyone compiling source code on Unix
whose authors had never prepared it to be compiled on anything but
Windows, so that it uses backslashes and inconsistent filename case in
its #include directives.

One particular example is if you're running clang-cl as a cross-
compiler, to build a Windows application on a Unix build host; another
is if you're developing a Unix port of a Windows code base in a
downstream clone of their repository. In either case, if you can't get
the upstream authors to take patches that make their #include
directives more portable, you might instead use this option to get
clang to tolerate the non-portable ones.

[simon_tatham]

As I mentioned on llvm-dev, this is an unfinished draft. I'm interested in review comments, but I already know things like 'needs more comments / tests / careful error handling' :-)

Some 'known unknowns':

• I'm caching the results of every directory scan, to save effort when more include files are needed from the same directory. That assumes that the lifetime of this filesystem object is short enough that we don't have to worry about new files appearing in directories we've already looked at, which seems like a reasonable assumption in an ordinary compile, but I don't know if the same filesystem abstraction might be used in completely different contexts elsewhere in the LLVM ecosystem and have a longer lifetime. Do I need to take a lot more care with my cache persistence?
• I was wondering if it might be worth having a second command-line option, to turn on only the case-insensitivity and not the backslash-separator tolerance (e.g. for use on source code developed for Mac rather than Windows).
• The option name is the first one that sprang to mind and I'm expecting someone will probably suggest a better one.

[thakis]

There are 2 other patches out there for the case sensitivity. Neither landed, because the performance hit form this approach is pretty big, and it's not necessary: You can either put the Win SDK into a ciopfs mount (example: https://cs.chromium.org/chromium/src/build/vs_toolchain.py?type=cs&q=ciopfs&sq=package:chromium&g=0&l=377 , used by chrome's win/cross build https://chromium.googlesource.com/chromium/src/+/master/docs/win_cross.md) or use -ivfsoverlay (example: http://llvm-cs.pcc.me.uk/cmake/platforms/WinMsvc.cmake#104 , used by llvm's win/cross build).

Have you measured the perf hit from your approach?

[simon_tatham]

No, I haven't measured it. Partly because that was one of the (many) things I was going to leave until after I didn't get feedback on the first draft that discouraged me from the whole idea :-) and also because I've already thought of one thing I can do to speed it up that I haven't done yet.

(Namely, break the pathname down recursively from the RH end instead of iteratively from the LH, so that in the case where the whole pathname is already case-correct, the lookup can succeed immediately. Then any performance hit should be limited to only the cases that actually need fixing up.)

I can see that if you're on Linux specifically then a FUSE filesystem can solve the case insensitivity, albeit in a way that needs setup and teardown. But surely backslash path separators are still problematic in that situation? (Or does ciopfs have an option I didn't spot to fudge those too?)

[thakis]

The things I pointed at only do case-insensitivity. I haven't seen anything using backslashes in the builds I worked on. (Is that for -I flags?) I'd think that the backslash bits can probably be implemented with less overhead.

[mstorsjo]

FWIW, I'm also interested in this. I wasn't aware of the other ways of achieving this used by chromium, but although neat, I'd prefer something that doesn't require that kind of setup.

As for performance - although I haven't studied how the code works - wouldn't it be possible to try the normal straightforward case sensitive open first? Then if that fails, we'd fall back on trying doing a case insensitive search (in either direction). That way, the performance hit only affects compilations that would otherwise fail, so it shouldn't be prohibitive for anything else than the specific headers that are problematic. Or perhaps that's already been taken into consideration?

[smeenai]

Adding some people who are interested in Windows cross-compilation.

[simon_tatham]

@thakis : no, the backslash support is needed for include directives in source files, not just on command lines, because in my experience it's not unusual for Windows-only code bases to be full of things like #include "Subdir\Header.h".

@mstorsjo : in this draft, I traverse the pathname from left to right, and at each level, if the normal case-sensitive lookup succeeds then I don't scan the whole directory looking for other options. As I mentioned above, though, I've since realised I should have done it the other way round, trying the case sensitive lookup on the whole pathname in one go first, and not even trying it one level at a time unless that fails.

Although, come to think of it, that's not good enough, because if you have multiple directories on your include path then you expect a lot of lookups to fail for reasons that have nothing to do with case. Say, if you compile with -Ifoo -Ibar, then every include of a file intended to be found in bar will force lots of pointless directory enumeration in foo when the initial lookup there doesn't find the file. Hmmm. That suggests to me that perhaps this ought not to be a global -ffudge-my-paths type option applying to all include directories, and instead perhaps it should be a new kind of -I option, so that only lookups relative to that particular include-path directory would get the unusual semantics. What do people think?

[mstorsjo]

In D48626#1146282, @simon_tatham wrote:

Although, come to think of it, that's not good enough, because if you have multiple directories on your include path then you expect a lot of lookups to fail for reasons that have nothing to do with case. Say, if you compile with -Ifoo -Ibar, then every include of a file intended to be found in bar will force lots of pointless directory enumeration in foo when the initial lookup there doesn't find the file. Hmmm. That suggests to me that perhaps this ought not to be a global -ffudge-my-paths type option applying to all include directories, and instead perhaps it should be a new kind of -I option, so that only lookups relative to that particular include-path directory would get the unusual semantics. What do people think?

The cases where I would have needed this (the windows sdk), I'm not specifying that path explicitly via -I directives, but implicitly via the INCLUDE env variable, but perhaps limiting this to the directories known to have problematic case could be helpful performance wise.

Otherwise I guess this would need to go on a higher level in the algorithm; first check all directories in a fully case sensitive manner, just like today. And if that fails (and we'd have a real failure that we'd otherwise return to the caller), redo it all with case insensitivity. I think that would be the best performance wise, but that makes for much messier integration...

[simon_tatham]

first check all directories in a fully case sensitive manner, just like today. And if that fails (and we'd have a real failure that we'd otherwise return to the caller), redo it all with case insensitivity.

I agree that the integration would be a bigger headache doing it that way. It would also depart from the expected semantics – a correctly cased header file in an include directory late on the path would override an incorrectly cased one earlier, which isn't how the real Windows compiler would handle the same situation.

The cases where I would have needed this (the windows sdk), I'm not specifying that path explicitly via -I directives, but implicitly via the INCLUDE env variable

*blush* I ought to have thought of that, because I've noticed the same problem!

but perhaps limiting this to the directories known to have problematic case could be helpful performance wise.

Yes. OK, in that case I think my proposed replacement design is to have an option along the lines of -fwindows-paths=prefix (spelling still open to change), with the effect that only pathname lookups starting with that prefix will be subject to this transformation.

include/clang/Basic/VirtualFileSystem.h

public:
   void setOverlayDir(StringRef OverlayDirectory) {
     IsOverlayRelative = true;
     OverlayDir.assign(OverlayDirectory.str());
   }
 
   void write(llvm::raw_ostream &OS);
 };
 
+/// Wrap an existing filesystem to create one that tolerates
+/// case-incorrect file names and backslash path separators, even if
+/// the underlying filesystem is not Windows-like.
+IntrusiveRefCntPtr<vfs::FileSystem>
+createWindowsFileSystemWrapper(IntrusiveRefCntPtr<vfs::FileSystem> BaseFS);
+
 } // namespace vfs
 } // namespace clang
 
 #endif // LLVM_CLANG_BASIC_VIRTUALFILESYSTEM_H

include/clang/Driver/Options.td

def fvisibility_inlines_hidden : Flag<["-"], "fvisibility-inlines-hidden">, Group<f_Group>,
   HelpText<"Give inline C++ member functions default visibility by default">,
   Flags<[CC1Option]>;
 def fvisibility_ms_compat : Flag<["-"], "fvisibility-ms-compat">, Group<f_Group>,
   HelpText<"Give global types 'default' visibility and global functions and "
            "variables 'hidden' visibility by default">;
 def fwhole_program_vtables : Flag<["-"], "fwhole-program-vtables">, Group<f_Group>,
   Flags<[CoreOption, CC1Option]>,
   HelpText<"Enables whole-program vtable optimization. Requires -flto">;
+def fwindows_filesystem : Flag<["-"], "fwindows-filesystem">, Group<f_Group>,
+  Flags<[CoreOption, CC1Option]>,
+  HelpText<"Permits case-insensitive filenames and \\ path separators">;
 def fno_whole_program_vtables : Flag<["-"], "fno-whole-program-vtables">, Group<f_Group>,
   Flags<[CoreOption]>;
 def fforce_emit_vtables : Flag<["-"], "fforce-emit-vtables">, Group<f_Group>,
     Flags<[CC1Option]>,
     HelpText<"Emits more virtual tables to improve devirtualization">;
 def fno_force_emit_vtables : Flag<["-"], "fno-force-emit-vtables">, Group<f_Group>,
   Flags<[CoreOption]>;
 def fwrapv : Flag<["-"], "fwrapv">, Group<f_Group>, Flags<[CC1Option]>,

include/clang/Lex/HeaderSearchOptions.h

public:
 
   /// Whether the module includes debug information (-gmodules).
   unsigned UseDebugInfo : 1;
 
   unsigned ModulesValidateDiagnosticOptions : 1;
 
   unsigned ModulesHashContent : 1;
 
+  /// Whether we are emulating Windows-style pathnames (-fwindows-filesystem)
+  unsigned WindowsPathnameEmulation : 1;
+
   HeaderSearchOptions(StringRef _Sysroot = "/")
       : Sysroot(_Sysroot), ModuleFormat("raw"), DisableModuleHash(false),
         ImplicitModuleMaps(false), ModuleMapFileHomeIsCwd(false),
         UseBuiltinIncludes(true), UseStandardSystemIncludes(true),
         UseStandardCXXIncludes(true), UseLibcxx(false), Verbose(false),
         ModulesValidateOncePerBuildSession(false),
         ModulesValidateSystemHeaders(false), UseDebugInfo(false),
         ModulesValidateDiagnosticOptions(true), ModulesHashContent(false) {}

lib/Basic/VirtualFileSystem.cpp

recursive_directory_iterator::increment(std::error_code &EC) {
   while (!State->empty() && State->top().increment(EC) == End)
     State->pop();
 
   if (State->empty())
     State.reset(); // end iterator
 
   return *this;
 }
+
+//===-----------------------------------------------------------------------===/
+// WindowsFileSystemWrapper implementation
+//===-----------------------------------------------------------------------===/
+
+class WindowsFileSystemWrapper : public vfs::FileSystem {
+  using String = SmallString<256>;
+
+  /// Path name style, used to identify path separators.
+  sys::path::Style PathStyle = sys::path::Style::windows;
+
+  /// The file system we are wrapping.
+  IntrusiveRefCntPtr<FileSystem> BaseFS;
+
+  /// Data about the contents of a single directory.
+  struct DirectoryCaseMap {
+    /// Map whose keys are the lowercase versions of filenames in that
+    /// directory, and values are the version of the same name that
+    /// actually exists.
+    std::map<String, String> Filenames;
+
+    /// Flag indicating whether we've done a complete trawl of the
+    /// directory yet, or whether we've only filled in a subset of
+    /// filenames.
+    bool Complete { false };
+
+    /// If we couldn't even scan this directory due to a filesystem
+    /// error, retain that error so we can return it again on the next
+    /// attempt.
+    std::error_code EC;
+  };
+
+  static String Lowercase(StringRef &Str);
+
+  /// Directories we've explored. Two sets of them, one relative to
+  /// the cwd and one to the filesystem root.
+  using CaseMapCollection = std::map<String, DirectoryCaseMap>;
+  CaseMapCollection AbsCaseMaps, RelCaseMaps;
+
+  /// Translates a path into one acceptable to the base file system.
+  ErrorOr<String> TranslatePath(const Twine &Path);
+
+  ErrorOr<Status> status(const Twine &Path) override {
+    auto TranslatedPath = TranslatePath(Path);
+    if (!TranslatedPath)
+      return TranslatedPath.getError();
+    return BaseFS->status(TranslatedPath.get());
+  }
+
+  ErrorOr<std::unique_ptr<File>> openFileForRead(const Twine &Path) override {
+    auto TranslatedPath = TranslatePath(Path);
+    if (!TranslatedPath)
+      return TranslatedPath.getError();
+    return BaseFS->openFileForRead(TranslatedPath.get());
+  }
+
+  std::error_code setCurrentWorkingDirectory(const Twine &Path) override {
+    auto TranslatedPath = TranslatePath(Path);
+    if (!TranslatedPath)
+      return TranslatedPath.getError();
+    auto ToReturn = BaseFS->setCurrentWorkingDirectory(TranslatedPath.get());
+    if (!ToReturn)
+      RelCaseMaps.clear(); // cwd changed, invalidating the cwd-relative cache
+    return ToReturn;
+  }
+
+  ErrorOr<std::string> getCurrentWorkingDirectory() const override {
+    return BaseFS->getCurrentWorkingDirectory();
+  }
+
+  directory_iterator dir_begin(const Twine &Dir, std::error_code &EC) override {
+    auto TranslatedPath = TranslatePath(Dir);
+    if (!TranslatedPath) {
+      EC = TranslatedPath.getError();
+      return directory_iterator();
+    }
+    return BaseFS->dir_begin(TranslatedPath.get(), EC);
+  }
+
+public:
+  WindowsFileSystemWrapper(IntrusiveRefCntPtr<FileSystem> Base)
+      : BaseFS(std::move(Base)) {}
+};
+
+auto WindowsFileSystemWrapper::TranslatePath(const Twine &InPathTw)
+  -> ErrorOr<String>
+{
+  String InPath;
+  InPathTw.toVector(InPath);
+
+  if (sys::path::root_name(InPath, PathStyle) != "")
+    return llvm::errc::no_such_file_or_directory;
+  String OutPath = sys::path::root_path(InPath, PathStyle);
+
+  CaseMapCollection *CMCP;
+  if (OutPath != "") {
+    InPath = relative_path(InPath, PathStyle);
+    CMCP = &AbsCaseMaps;
+  } else {
+    OutPath = ".";
+    CMCP = &RelCaseMaps;
+  }
+  CaseMapCollection &CMC = *CMCP;
+
+  for (sys::path::const_iterator InIter = sys::path::begin(InPath, PathStyle),
+         InEnd = sys::path::end(InPath); InIter != InEnd; ++InIter) {
+    StringRef Component = *InIter;
+    String TranslatedComponent;
+
+    auto &ThisCaseMap = CMC[OutPath];
+    if (ThisCaseMap.EC)
+      return ThisCaseMap.EC;
+
+    String Key = StringRef(StringRef(Component).lower());
+    auto Found = ThisCaseMap.Filenames.find(Key);
+
+    if (Found != ThisCaseMap.Filenames.end()) {
+      TranslatedComponent = Found->second;
+    } else {
+      String TempPath = OutPath;
+      sys::path::append(TempPath, Component);
+      auto Status = BaseFS->status(TempPath);
+      if (Status && Status.get().getType() != file_type::file_not_found) {
+        // This component in its original case refers to a directory
+        // entry that exists, so we don't need to look further.
+        ThisCaseMap.Filenames[Key] = TranslatedComponent = Component;
+      } else {
+        if (!ThisCaseMap.Complete) {
+          // We have to scan the whole containing directory to see if
+          // anything is a case-transformed version of this name.
+
+          ThisCaseMap.Complete = true; // even if we fail, we'll never retry
+
+          auto dir_iter = BaseFS->dir_begin(OutPath, ThisCaseMap.EC);
+          if (ThisCaseMap.EC)
+            return ThisCaseMap.EC;
+
+          std::error_code IncrementEC;
+          for (; dir_iter != directory_iterator();
+               dir_iter.increment(IncrementEC)) {
+            String Name = sys::path::filename(dir_iter->getName());
+            String Key = StringRef(StringRef(Name).lower());
+            ThisCaseMap.Filenames[Key] = Name;
+          }
+
+          // Retry the lookup, now we've fully populated the map.
+          Found = ThisCaseMap.Filenames.find(Key);
+        }
+
+        if (Found == ThisCaseMap.Filenames.end()) {
+          return llvm::errc::no_such_file_or_directory;
+        }
+
+        TranslatedComponent = Found->second;
+      }
+    }
+
+    sys::path::append(OutPath, TranslatedComponent);
+  }
+  return OutPath;
+}
+
+IntrusiveRefCntPtr<vfs::FileSystem>
+vfs::createWindowsFileSystemWrapper(IntrusiveRefCntPtr<vfs::FileSystem> BaseFS)
+{
+  return new WindowsFileSystemWrapper(std::move(BaseFS));
+}

lib/Frontend/CompilerInvocation.cpp

static void ParseHeaderSearchArgs(HeaderSearchOptions &Opts, ArgList &Args,
   // Add the path prefixes which are implicitly treated as being system headers.
   for (const auto *A :
        Args.filtered(OPT_system_header_prefix, OPT_no_system_header_prefix))
     Opts.AddSystemHeaderPrefix(
         A->getValue(), A->getOption().matches(OPT_system_header_prefix));
 
   for (const auto *A : Args.filtered(OPT_ivfsoverlay))
     Opts.AddVFSOverlayFile(A->getValue());
+
+  if (Args.hasArg(OPT_fwindows_filesystem))
+    Opts.WindowsPathnameEmulation = true;
 }
 
 void CompilerInvocation::setLangDefaults(LangOptions &Opts, InputKind IK,
                                          const llvm::Triple &T,
                                          PreprocessorOptions &PPOpts,
                                          LangStandard::Kind LangStd) {
   // Set some properties which depend solely on the input kind; it would be nice
   // to move these to the language standard, and have the driver resolve the
createVFSFromCompilerInvocation(const CompilerInvocation &CI,
                                 DiagnosticsEngine &Diags) {
   return createVFSFromCompilerInvocation(CI, Diags, vfs::getRealFileSystem());
 }
 
 IntrusiveRefCntPtr<vfs::FileSystem>
 createVFSFromCompilerInvocation(const CompilerInvocation &CI,
                                 DiagnosticsEngine &Diags,
                                 IntrusiveRefCntPtr<vfs::FileSystem> BaseFS) {
-  if (CI.getHeaderSearchOpts().VFSOverlayFiles.empty())
-    return BaseFS;
+  IntrusiveRefCntPtr<vfs::FileSystem> FS = BaseFS;
 
+  if (!CI.getHeaderSearchOpts().VFSOverlayFiles.empty()) {
     IntrusiveRefCntPtr<vfs::OverlayFileSystem> Overlay(
-      new vfs::OverlayFileSystem(BaseFS));
+      new vfs::OverlayFileSystem(FS));
     // earlier vfs files are on the bottom
     for (const auto &File : CI.getHeaderSearchOpts().VFSOverlayFiles) {
       llvm::ErrorOr<std::unique_ptr<llvm::MemoryBuffer>> Buffer =
         BaseFS->getBufferForFile(File);
       if (!Buffer) {
         Diags.Report(diag::err_missing_vfs_overlay_file) << File;
         continue;
       }
 
       IntrusiveRefCntPtr<vfs::FileSystem> FS = vfs::getVFSFromYAML(
         std::move(Buffer.get()), /*DiagHandler*/ nullptr, File);
       if (FS)
         Overlay->pushOverlay(FS);
       else
         Diags.Report(diag::err_invalid_vfs_overlay) << File;
     }
-  return Overlay;
+
+    FS = std::move(Overlay);
+  }
+
+  if (CI.getHeaderSearchOpts().WindowsPathnameEmulation) {
+    FS = createWindowsFileSystemWrapper(std::move(FS));
+  }
+
+  return FS;
 }
 
 } // namespace clang