Lex: Migrate HeaderSearch::LoadedModuleMaps to FileEntryRef

[dexonsmith]

Summary

Migrate HeaderSearch::LoadedModuleMaps and a number of APIs over to
FileEntryRef. This should have no functionality change. Note that two
FileEntryRefs hash the same if they point at the same FileEntry.

[dexonsmith]

I must have just been building clangLex before, since I missed (until now) updating the other API users.

[jansvoboda11]

LGTM.

[thakis]

One of the changes in https://github.com/llvm/llvm-project/compare/6af2c4ca9bd...2733a5a5b44 broke check-clang on windows. This change looks the most likely to me. Can you take a look?

http://45.33.8.238/win/29852/step_7.txt

[thakis]

Reverted in 7799ef7121aa7d59f4bd95cdf70035de724ead6f for now.

[thakis]

The revert helped, check-clang is no longer failing on Windows.

[dexonsmith]

Thanks for reverting; sorry I missed your email.

[dexonsmith]

There were also two Windows bots that emailed me yesterday afternoon, I just missed them somehow :/.

• http://lab.llvm.org:8011/#/builders/119/builds/1381 (llvm-clang-win-x-aarch64)
• http://lab.llvm.org:8011/#/builders/60/builds/1175 (llvm-clang-win-x-armv7l)

FTR, the test that fails is clang/test/VFS/real-path-found-first.m, which was added in 5de00f3b56805c7e980f049ceb3f166f8c69cec0 (Stopgap fix for finding module for a file mapped in the VFS). It fails on final -cc1 call at line 64, which is the reentry for "within a module build":

# command stderr:
<stdin>:1:2: fatal error: module 'import_some_frame' was built in directory 'C:/src\llvm-project\clang\test\VFS\Inputs' but now resides in directory 'C:\src\llvm-project\out\gn\obj\clang\test\VFS\Output\real-path-found-first.m.tmp'
@import import_some_frame;
 ^

[dexonsmith]

Reopening for now, although I haven't updated the patch yet, or quite tracked down the issue.

[dexonsmith]

Given that 5de00f3b56805c7e980f049ceb3f166f8c69cec0 updates the return value of FileEntry::getDir as a (self-described) hack, and FileEntryRef::getDir won't be similarly updated, it's somewhat reasonable that this would break... but I still need to understand the exact scenario here before I can fix it.

clang-tools-extra/modularize/ModularizeUtilities.cpp

std::error_code ModularizeUtilities::loadProblemHeaderList(
   }
   return std::error_code();
 }
 
 // Load single module map and extract header file list.
 std::error_code ModularizeUtilities::loadModuleMap(
     llvm::StringRef InputPath) {
   // Get file entry for module.modulemap file.
-  auto ModuleMapEntryOrErr =
-    SourceMgr->getFileManager().getFile(InputPath);
+  auto ExpectedModuleMapEntry =
+    SourceMgr->getFileManager().getFileRef(InputPath);
 
   // return error if not found.
-  if (!ModuleMapEntryOrErr) {
+  if (!ExpectedModuleMapEntry) {
     llvm::errs() << "error: File \"" << InputPath << "\" not found.\n";
-    return ModuleMapEntryOrErr.getError();
+    return errorToErrorCode(ExpectedModuleMapEntry.takeError());
   }
-  const FileEntry *ModuleMapEntry = *ModuleMapEntryOrErr;
+  FileEntryRef ModuleMapEntry = *ExpectedModuleMapEntry;
 
   // Because the module map parser uses a ForwardingDiagnosticConsumer,
   // which doesn't forward the BeginSourceFile call, we do it explicitly here.
   DC.BeginSourceFile(*LangOpts, nullptr);
 
   // Figure out the home directory for the module map file.
-  const DirectoryEntry *Dir = ModuleMapEntry->getDir();
+  const DirectoryEntry *Dir = ModuleMapEntry.getDir();
   StringRef DirName(Dir->getName());
   if (llvm::sys::path::filename(DirName) == "Modules") {
     DirName = llvm::sys::path::parent_path(DirName);
     if (DirName.endswith(".framework")) {
       if (auto DirEntry = FileMgr->getDirectory(DirName))
         Dir = *DirEntry;
       else
         Dir = nullptr;

clang/include/clang/Lex/HeaderSearch.h

class HeaderSearch {
   /// The mapping between modules and headers.
   mutable ModuleMap ModMap;
 
   /// Describes whether a given directory has a module map in it.
   llvm::DenseMap<const DirectoryEntry *, bool> DirectoryHasModuleMap;
 
   /// Set of module map files we've already loaded, and a flag indicating
   /// whether they were valid or not.
-  llvm::DenseMap<const FileEntry *, bool> LoadedModuleMaps;
+  llvm::DenseMap<FileEntryRef, bool> LoadedModuleMaps;
 
   /// Uniqued set of framework names, which is used to track which
   /// headers were included as framework headers.
   llvm::StringSet<llvm::BumpPtrAllocator> FrameworkNames;
 
   /// Entity used to resolve the identifier IDs of controlling
   /// macros into IdentifierInfo pointers, and keep the identifire up to date,
   /// as needed.
public:
   /// in subdirectories.
   ///
   /// \returns The module with the given name.
   Module *lookupModule(StringRef ModuleName, bool AllowSearch = true,
                        bool AllowExtraModuleMapSearch = false);
 
   /// Try to find a module map file in the given directory, returning
   /// \c nullptr if none is found.
-  const FileEntry *lookupModuleMapFile(const DirectoryEntry *Dir,
+  Optional<FileEntryRef> lookupModuleMapFile(const DirectoryEntry *Dir,
                                              bool IsFramework);
 
   /// Determine whether there is a module map that may map the header
   /// with the given file name to a (sub)module.
   /// Always returns false if modules are disabled.
   ///
   /// \param Filename The name of the file.
   ///
   /// \param Root The "root" directory, at which we should stop looking for
public:
   ///        processing a preprocessed module), the ID of the file.
   /// \param Offset [inout] An offset within ID to start parsing. On exit,
   ///        filled by the end of the parsed contents (either EOF or the
   ///        location of an end-of-module-map pragma).
   /// \param OriginalModuleMapFile The original path to the module map file,
   ///        used to resolve paths within the module (this is required when
   ///        building the module from preprocessed source).
   /// \returns true if an error occurred, false otherwise.
-  bool loadModuleMapFile(const FileEntry *File, bool IsSystem,
-                         FileID ID = FileID(), unsigned *Offset = nullptr,
+  bool loadModuleMapFile(FileEntryRef File, bool IsSystem, FileID ID = FileID(),
+                         unsigned *Offset = nullptr,
                          StringRef OriginalModuleMapFile = StringRef());
 
   /// Collect the set of all known, top-level modules.
   ///
   /// \param Modules Will be filled with the set of known, top-level modules.
   void collectAllModules(SmallVectorImpl<Module *> &Modules);
 
   /// Load all known, top-level system modules.
enum LoadModuleMapResult {
     /// There is was directory with the given name.
     LMM_NoDirectory,
 
     /// There was either no module map file or the module map file was
     /// invalid.
     LMM_InvalidModuleMap
   };
 
-  LoadModuleMapResult loadModuleMapFileImpl(const FileEntry *File,
-                                            bool IsSystem,
+  LoadModuleMapResult loadModuleMapFileImpl(FileEntryRef File, bool IsSystem,
                                             const DirectoryEntry *Dir,
                                             FileID ID = FileID(),
                                             unsigned *Offset = nullptr);
 
   /// Try to load the module map file in the given directory.
   ///
   /// \param DirName The name of the directory where we will look for a module
   /// map file.

clang/include/clang/Lex/ModuleMap.h

public:
   ///
   /// \param Offset [inout] On input the offset at which to start parsing. On
   ///        output, the offset at which the module map terminated.
   ///
   /// \param ExternModuleLoc The location of the "extern module" declaration
   ///        that caused us to load this module map file, if any.
   ///
   /// \returns true if an error occurred, false otherwise.
-  bool parseModuleMapFile(const FileEntry *File, bool IsSystem,
-                          const DirectoryEntry *HomeDir,
-                          FileID ID = FileID(), unsigned *Offset = nullptr,
+  bool parseModuleMapFile(FileEntryRef File, bool IsSystem,
+                          const DirectoryEntry *HomeDir, FileID ID = FileID(),
+                          unsigned *Offset = nullptr,
                           SourceLocation ExternModuleLoc = SourceLocation());
 
   /// Dump the contents of the module map, for debugging purposes.
   void dump();
 
   using module_iterator = llvm::StringMap<Module *>::const_iterator;
 
   module_iterator module_begin() const { return Modules.begin(); }

clang/lib/Frontend/FrontendAction.cpp

static bool loadModuleMapForModuleBuild(CompilerInstance &CI, bool IsSystem,
                                         bool IsPreprocessed,
                                         std::string &PresumedModuleMapFile,
                                         unsigned &Offset) {
   auto &SrcMgr = CI.getSourceManager();
   HeaderSearch &HS = CI.getPreprocessor().getHeaderSearchInfo();
 
   // Map the current input to a file.
   FileID ModuleMapID = SrcMgr.getMainFileID();
-  const FileEntry *ModuleMap = SrcMgr.getFileEntryForID(ModuleMapID);
+  Optional<FileEntryRef> ModuleMap = SrcMgr.getFileEntryRefForID(ModuleMapID);
 
   // If the module map is preprocessed, handle the initial line marker;
   // line directives are not part of the module map syntax in general.
   Offset = 0;
   if (IsPreprocessed) {
     SourceLocation EndOfLineMarker =
         ReadOriginalFileName(CI, PresumedModuleMapFile, /*IsModuleMap*/ true);
     if (EndOfLineMarker.isValid())
       Offset = CI.getSourceManager().getDecomposedLoc(EndOfLineMarker).second;
   }
 
   // Load the module map file.
-  if (HS.loadModuleMapFile(ModuleMap, IsSystem, ModuleMapID, &Offset,
+  if (HS.loadModuleMapFile(*ModuleMap, IsSystem, ModuleMapID, &Offset,
                            PresumedModuleMapFile))
     return true;
 
   if (SrcMgr.getBufferOrFake(ModuleMapID).getBufferSize() == Offset)
     Offset = 0;
 
   return false;
 }
bool FrontendAction::BeginSourceFile(CompilerInstance &CI,
   }
 
   // Initialize the action.
   if (!BeginSourceFileAction(CI))
     goto failure;
 
   // If we were asked to load any module map files, do so now.
   for (const auto &Filename : CI.getFrontendOpts().ModuleMapFiles) {
-    if (auto File = CI.getFileManager().getFile(Filename))
+    if (auto File = CI.getFileManager().getOptionalFileRef(Filename))
       CI.getPreprocessor().getHeaderSearchInfo().loadModuleMapFile(
           *File, /*IsSystem*/false);
     else
       CI.getDiagnostics().Report(diag::err_module_map_not_found) << Filename;
   }
 
   // Add a module declaration scope so that modules from -fmodule-map-file
   // arguments may shadow modules found implicitly in search paths.

clang/lib/Lex/HeaderSearch.cpp

if (needModuleLookup(RequestingModule, SuggestedModule)) {
     // important so that we're consistent about whether this header
     // corresponds to a module. Possibly we should lock down framework modules
     // so that this is not possible.
     return suggestModule(*this, File, RequestingModule, SuggestedModule);
   }
   return true;
 }
 
-static const FileEntry *getPrivateModuleMap(const FileEntry *File,
+static Optional<FileEntryRef> getPrivateModuleMap(FileEntryRef File,
                                                   FileManager &FileMgr) {
-  StringRef Filename = llvm::sys::path::filename(File->getName());
-  SmallString<128>  PrivateFilename(File->getDir()->getName());
+  StringRef Filename = llvm::sys::path::filename(File.getName());
+  SmallString<128> PrivateFilename(File.getDir().getName());
   if (Filename == "module.map")
     llvm::sys::path::append(PrivateFilename, "module_private.map");
   else if (Filename == "module.modulemap")
     llvm::sys::path::append(PrivateFilename, "module.private.modulemap");
   else
-    return nullptr;
-  if (auto File = FileMgr.getFile(PrivateFilename))
-    return *File;
-  return nullptr;
+    return None;
+  return FileMgr.getOptionalFileRef(PrivateFilename);
 }
 
-bool HeaderSearch::loadModuleMapFile(const FileEntry *File, bool IsSystem,
+bool HeaderSearch::loadModuleMapFile(FileEntryRef File, bool IsSystem,
                                      FileID ID, unsigned *Offset,
                                      StringRef OriginalModuleMapFile) {
   // Find the directory for the module. For frameworks, that may require going
   // up from the 'Modules' directory.
   const DirectoryEntry *Dir = nullptr;
   if (getHeaderSearchOpts().ModuleMapFileHomeIsCwd) {
     if (auto DirOrErr = FileMgr.getDirectory("."))
       Dir = *DirOrErr;
   } else {
     if (!OriginalModuleMapFile.empty()) {
       // We're building a preprocessed module map. Find or invent the directory
       // that it originally occupied.
       auto DirOrErr = FileMgr.getDirectory(
           llvm::sys::path::parent_path(OriginalModuleMapFile));
       if (DirOrErr) {
         Dir = *DirOrErr;
       } else {
         auto *FakeFile = FileMgr.getVirtualFile(OriginalModuleMapFile, 0, 0);
         Dir = FakeFile->getDir();
       }
     } else {
-      Dir = File->getDir();
+      Dir = File.getDir();
     }
 
     StringRef DirName(Dir->getName());
     if (llvm::sys::path::filename(DirName) == "Modules") {
       DirName = llvm::sys::path::parent_path(DirName);
       if (DirName.endswith(".framework"))
         if (auto DirOrErr = FileMgr.getDirectory(DirName))
           Dir = *DirOrErr;
bool HeaderSearch::loadModuleMapFile(FileEntryRef File, bool IsSystem,
   case LMM_NoDirectory:
   case LMM_InvalidModuleMap:
     return true;
   }
   llvm_unreachable("Unknown load module map result");
 }
 
 HeaderSearch::LoadModuleMapResult
-HeaderSearch::loadModuleMapFileImpl(const FileEntry *File, bool IsSystem,
+HeaderSearch::loadModuleMapFileImpl(FileEntryRef File, bool IsSystem,
                                     const DirectoryEntry *Dir, FileID ID,
                                     unsigned *Offset) {
-  assert(File && "expected FileEntry");
-
   // Check whether we've already loaded this module map, and mark it as being
   // loaded in case we recursively try to load it from itself.
   auto AddResult = LoadedModuleMaps.insert(std::make_pair(File, true));
   if (!AddResult.second)
     return AddResult.first->second ? LMM_AlreadyLoaded : LMM_InvalidModuleMap;
 
   if (ModMap.parseModuleMapFile(File, IsSystem, Dir, ID, Offset)) {
     LoadedModuleMaps[File] = false;
     return LMM_InvalidModuleMap;
   }
 
   // Try to load a corresponding private module map.
-  if (const FileEntry *PMMFile = getPrivateModuleMap(File, FileMgr)) {
-    if (ModMap.parseModuleMapFile(PMMFile, IsSystem, Dir)) {
+  if (Optional<FileEntryRef> PMMFile = getPrivateModuleMap(File, FileMgr)) {
+    if (ModMap.parseModuleMapFile(*PMMFile, IsSystem, Dir)) {
       LoadedModuleMaps[File] = false;
       return LMM_InvalidModuleMap;
     }
   }
 
   // This directory has a module map.
   return LMM_NewlyLoaded;
 }
 
-const FileEntry *
+Optional<FileEntryRef>
 HeaderSearch::lookupModuleMapFile(const DirectoryEntry *Dir, bool IsFramework) {
   if (!HSOpts->ImplicitModuleMaps)
-    return nullptr;
+    return None;
   // For frameworks, the preferred spelling is Modules/module.modulemap, but
   // module.map at the framework root is also accepted.
   SmallString<128> ModuleMapFileName(Dir->getName());
   if (IsFramework)
     llvm::sys::path::append(ModuleMapFileName, "Modules");
   llvm::sys::path::append(ModuleMapFileName, "module.modulemap");
-  if (auto F = FileMgr.getFile(ModuleMapFileName))
-    return *F;
+  if (auto F = FileMgr.getOptionalFileRef(ModuleMapFileName))
+    return F;
 
   // Continue to allow module.map
   ModuleMapFileName = Dir->getName();
   llvm::sys::path::append(ModuleMapFileName, "module.map");
-  if (auto F = FileMgr.getFile(ModuleMapFileName))
-    return *F;
+  if (auto F = FileMgr.getOptionalFileRef(ModuleMapFileName))
+    return F;
 
   // For frameworks, allow to have a private module map with a preferred
   // spelling when a public module map is absent.
   if (IsFramework) {
     ModuleMapFileName = Dir->getName();
     llvm::sys::path::append(ModuleMapFileName, "Modules",
                             "module.private.modulemap");
-    if (auto F = FileMgr.getFile(ModuleMapFileName))
-      return *F;
+    if (auto F = FileMgr.getOptionalFileRef(ModuleMapFileName))
+      return F;
   }
-  return nullptr;
+  return None;
 }
 
 Module *HeaderSearch::loadFrameworkModule(StringRef Name,
                                           const DirectoryEntry *Dir,
                                           bool IsSystem) {
   if (Module *Module = ModMap.findModule(Name))
     return Module;
 

 
 HeaderSearch::LoadModuleMapResult
 HeaderSearch::loadModuleMapFile(const DirectoryEntry *Dir, bool IsSystem,
                                 bool IsFramework) {
   auto KnownDir = DirectoryHasModuleMap.find(Dir);
   if (KnownDir != DirectoryHasModuleMap.end())
     return KnownDir->second ? LMM_AlreadyLoaded : LMM_InvalidModuleMap;
 
-  if (const FileEntry *ModuleMapFile = lookupModuleMapFile(Dir, IsFramework)) {
+  if (Optional<FileEntryRef> ModuleMapFile =
+          lookupModuleMapFile(Dir, IsFramework)) {
     LoadModuleMapResult Result =
-        loadModuleMapFileImpl(ModuleMapFile, IsSystem, Dir);
+        loadModuleMapFileImpl(*ModuleMapFile, IsSystem, Dir);
     // Add Dir explicitly in case ModuleMapFile is in a subdirectory.
     // E.g. Foo.framework/Modules/module.modulemap
     //      ^Dir                  ^ModuleMapFile
     if (Result == LMM_NewlyLoaded)
       DirectoryHasModuleMap[Dir] = true;
     else if (Result == LMM_InvalidModuleMap)
       DirectoryHasModuleMap[Dir] = false;
     return Result;

clang/lib/Lex/ModuleMap.cpp

if (llvm::sys::path::has_parent_path(FrameworkDirName)) {
         // Check whether we have already looked into the parent directory
         // for a module map.
         llvm::DenseMap<const DirectoryEntry *, InferredDirectory>::const_iterator
           inferred = InferredDirectories.find(*ParentDir);
         if (inferred == InferredDirectories.end()) {
           // We haven't looked here before. Load a module map, if there is
           // one.
           bool IsFrameworkDir = Parent.endswith(".framework");
-          if (const FileEntry *ModMapFile =
+          if (Optional<FileEntryRef> ModMapFile =
                   HeaderInfo.lookupModuleMapFile(*ParentDir, IsFrameworkDir)) {
-            parseModuleMapFile(ModMapFile, Attrs.IsSystem, *ParentDir);
+            parseModuleMapFile(*ModMapFile, Attrs.IsSystem, *ParentDir);
             inferred = InferredDirectories.find(*ParentDir);
           }
 
           if (inferred == InferredDirectories.end())
             inferred = InferredDirectories.insert(
                          std::make_pair(*ParentDir, InferredDirectory())).first;
         }
 
void ModuleMapParser::parseExternModuleDecl() {
 
   StringRef FileNameRef = FileName;
   SmallString<128> ModuleMapFileName;
   if (llvm::sys::path::is_relative(FileNameRef)) {
     ModuleMapFileName += Directory->getName();
     llvm::sys::path::append(ModuleMapFileName, FileName);
     FileNameRef = ModuleMapFileName;
   }
-  if (auto File = SourceMgr.getFileManager().getFile(FileNameRef))
+  if (auto File = SourceMgr.getFileManager().getOptionalFileRef(FileNameRef))
     Map.parseModuleMapFile(
         *File, /*IsSystem=*/false,
         Map.HeaderInfo.getHeaderSearchOpts().ModuleMapFileHomeIsCwd
             ? Directory
-            : (*File)->getDir(),
+            : File->getDir(),
         FileID(), nullptr, ExternLoc);
 }
 
 /// Whether to add the requirement \p Feature to the module \p M.
 ///
 /// This preserves backwards compatibility for two hacks in the Darwin system
 /// module map files:
 ///
case MMToken::UseKeyword:
       Diags.Report(Tok.getLocation(), diag::err_mmap_expected_module);
       HadError = true;
       consumeToken();
       break;
     }
   } while (true);
 }
 
-bool ModuleMap::parseModuleMapFile(const FileEntry *File, bool IsSystem,
+bool ModuleMap::parseModuleMapFile(FileEntryRef File, bool IsSystem,
                                    const DirectoryEntry *Dir, FileID ID,
                                    unsigned *Offset,
                                    SourceLocation ExternModuleLoc) {
   assert(Target && "Missing target information");
   llvm::DenseMap<const FileEntry *, bool>::iterator Known
     = ParsedModuleMap.find(File);
   if (Known != ParsedModuleMap.end())
     return Known->second;