[clang][lex] Refactor check for the first file include

[jansvoboda11]

Summary

This patch refactors the code that checks whether a file has just been included for the first time.

The HeaderSearch::FirstTimeLexingFile function is removed and the information is threaded to the original call site from HeaderSearch::ShouldEnterIncludeFile. This will make it possible to avoid tracking the number of includes in a follow up patch.

Depends on D114092.

[dexonsmith]

LGTM with one nit inline.

clang/include/clang/Lex/Lexer.h

143–144

Please move up next to the other bool declarations so that it uses the same word for storage.

[vsapsai]

Have just a few little tweaks, otherwise looks good. And we've discussed that it's possible we don't really need the first time lexing check anymore. But that's a separate issue and I support your decision to preserve the existing behavior and not to increase the scope of work.

clang/include/clang/Lex/HeaderSearch.h

446–448

As we've discussed earlier, the order of parameters is slightly weird. Usually, out-parameters are at the end of the list, though I'm not sure there is a rule about it. Personally, I would move File to the front because the method is ShouldEnterIncludeFile and keep the order as File, isImport, PP, ModulesEnabled, M, IsFirstIncludeOfFile but that's more my personal opinion and I'll leave the final decision to you.

clang/include/clang/Lex/Preprocessor.h

1370–1371

Not sure we need a default value for IsFirstIncludeOfFile. Grepping shows the only other place the method is called is IncrementalParser::Parse, so we can provide an explicit value there.

[jansvoboda11]

clang/include/clang/Lex/HeaderSearch.h

446–448

I agree the order is sometimes a bit weird. It seems like the convention is to pass "bigger" objects (like Preprocessor, HeaderSearch, CompilerInstance etc.) first. I'll move the new out param to the back, but probably won't mess with the existing parameters.

clang/include/clang/Lex/Preprocessor.h

1370–1371

It's also called twice in Preprocessor::EnterMainSourceFile where specifying true would seem redundant. I think I'll keep it as is for now.

clang/include/clang/Lex/HeaderSearch.h

public:
   }
 
   /// Mark the specified file as a target of a \#include,
   /// \#include_next, or \#import directive.
   ///
   /// \return false if \#including the file will have no effect or true
   /// if we should include it.
   bool ShouldEnterIncludeFile(Preprocessor &PP, const FileEntry *File,
-                              bool isImport, bool ModulesEnabled,
-                              Module *M);
+                              bool isImport, bool ModulesEnabled, Module *M,
+                              bool &IsFirstIncludeOfFile);
 
   /// Return whether the specified file is a normal header,
   /// a system header, or a C++ friendly system header.
   SrcMgr::CharacteristicKind getFileDirFlavor(const FileEntry *File) {
     return (SrcMgr::CharacteristicKind)getFileInfo(File).DirInfo;
   }
 
   /// Mark the specified file as a "once only" file due to
   /// \#pragma once.
public:
   ///
   /// This is used by the multiple-include optimization to eliminate
   /// no-op \#includes.
   void SetFileControllingMacro(const FileEntry *File,
                                const IdentifierInfo *ControllingMacro) {
     getFileInfo(File).ControllingMacro = ControllingMacro;
   }
 
-  /// Return true if this is the first time encountering this header.
-  bool FirstTimeLexingFile(const FileEntry *File) {
-    return getFileInfo(File).NumIncludes == 1;
-  }
-
   /// Determine whether this file is intended to be safe from
   /// multiple inclusions, e.g., it has \#pragma once or a controlling
   /// macro.
   ///
   /// This routine does not consider the effect of \#import
   bool isFileMultipleIncludeGuarded(const FileEntry *File);
 
   /// Determine whether the given file is known to have ever been \#imported.

clang/include/clang/Lex/Lexer.h

class Lexer : public PreprocessorLexer {
   bool IsAtStartOfLine;
 
   bool IsAtPhysicalStartOfLine;
 
   bool HasLeadingSpace;
 
   bool HasLeadingEmptyMacro;
 
+  /// True if this is the first time we're lexing the input file.
+  bool IsFirstTimeLexingFile;
+
   // NewLinePtr - A pointer to new line character '\n' being lexed. For '\r\n',
   // it also points to '\n.'
   const char *NewLinePtr;
 
   // CurrentConflictMarkerState - The kind of conflict marker we are handling.
   ConflictMarkerKind CurrentConflictMarkerState;
 
   void InitLexer(const char *BufStart, const char *BufPtr, const char *BufEnd);
 
 public:
   /// Lexer constructor - Create a new lexer object for the specified buffer
   /// with the specified preprocessor managing the lexing process.  This lexer
   /// assumes that the associated file buffer and Preprocessor objects will
   /// outlive it, so it doesn't take ownership of either of them.
-  Lexer(FileID FID, const llvm::MemoryBufferRef &InputFile, Preprocessor &PP);
+  Lexer(FileID FID, const llvm::MemoryBufferRef &InputFile, Preprocessor &PP,
+        bool IsFirstIncludeOfFile = true);
 
   /// Lexer constructor - Create a new raw lexer object.  This object is only
   /// suitable for calls to 'LexFromRawLexer'.  This lexer assumes that the
   /// text range will outlive it, so it doesn't take ownership of it.
   Lexer(SourceLocation FileLoc, const LangOptions &LangOpts,
-        const char *BufStart, const char *BufPtr, const char *BufEnd);
+        const char *BufStart, const char *BufPtr, const char *BufEnd,
+        bool IsFirstIncludeOfFile = true);
 
   /// Lexer constructor - Create a new raw lexer object.  This object is only
   /// suitable for calls to 'LexFromRawLexer'.  This lexer assumes that the
   /// text range will outlive it, so it doesn't take ownership of it.
   Lexer(FileID FID, const llvm::MemoryBufferRef &FromFile,
-        const SourceManager &SM, const LangOptions &LangOpts);
+        const SourceManager &SM, const LangOptions &LangOpts,
+        bool IsFirstIncludeOfFile = true);
 
   Lexer(const Lexer &) = delete;
   Lexer &operator=(const Lexer &) = delete;
 
   /// Create_PragmaLexer: Lexer constructor - Create a new lexer object for
   /// _Pragma expansion.  This has a variety of magic semantics that this method
   /// sets up.  It returns a new'd Lexer that must be delete'd when done.
   static Lexer *Create_PragmaLexer(SourceLocation SpellingLoc,
static inline char getCharAndSizeNoWarn(const char *Ptr, unsigned &Size,
     return getCharAndSizeSlowNoWarn(Ptr, Size, LangOpts);
   }
 
   /// Returns the leading whitespace for line that corresponds to the given
   /// location \p Loc.
   static StringRef getIndentationForLine(SourceLocation Loc,
                                          const SourceManager &SM);
 
+  /// Check if this is the first time we're lexing the input file.
+  bool isFirstTimeLexingFile() const { return IsFirstTimeLexingFile; }
+
 private:
   //===--------------------------------------------------------------------===//
   // Internal implementation interfaces.
 
   /// LexTokenInternal - Internal interface to lex a preprocessing token. Called
   /// by Lex.
   ///
   bool LexTokenInternal(Token &Result, bool TokAtPhysicalStartOfLine);

clang/include/clang/Lex/Preprocessor.h

public:
 
   /// Inform the preprocessor callbacks that processing is complete.
   void EndSourceFile();
 
   /// Add a source file to the top of the include stack and
   /// start lexing tokens from it instead of the current buffer.
   ///
   /// Emits a diagnostic, doesn't enter the file, and returns true on error.
   bool EnterSourceFile(FileID FID, const DirectoryLookup *Dir,
-                       SourceLocation Loc);
+                       SourceLocation Loc, bool IsFirstIncludeOfFile = true);
 
   /// Add a Macro to the top of the include stack and start lexing
   /// tokens from it instead of the current buffer.
   ///
   /// \param Args specifies the tokens input to a function-like macro.
   /// \param ILEnd specifies the location of the ')' for a function-like macro
   /// or the identifier for an object-like macro.
   void EnterMacro(Token &Tok, SourceLocation ILEnd, MacroInfo *Macro,

clang/lib/Lex/HeaderSearch.cpp

void HeaderSearch::MarkFileModuleHeader(const FileEntry *FE,
 
   auto &HFI = getFileInfo(FE);
   HFI.isModuleHeader |= isModularHeader;
   HFI.isCompilingModuleHeader |= isCompilingModuleHeader;
 }
 
 bool HeaderSearch::ShouldEnterIncludeFile(Preprocessor &PP,
                                           const FileEntry *File, bool isImport,
-                                          bool ModulesEnabled, Module *M) {
+                                          bool ModulesEnabled, Module *M,
+                                          bool &IsFirstIncludeOfFile) {
   ++NumIncluded; // Count # of attempted #includes.
 
+  IsFirstIncludeOfFile = false;
+
   // Get information about this file.
   HeaderFileInfo &FileInfo = getFileInfo(File);
 
   // FIXME: this is a workaround for the lack of proper modules-aware support
   // for #import / #pragma once
   auto TryEnterImported = [&]() -> bool {
     if (!ModulesEnabled)
       return false;
if (M ? PP.isMacroDefinedInLocalModule(ControllingMacro, M)
       ++NumMultiIncludeFileOptzn;
       return false;
     }
   }
 
   // Increment the number of times this file has been included.
   ++FileInfo.NumIncludes;
 
+  IsFirstIncludeOfFile = FileInfo.NumIncludes == 1;
+
   return true;
 }
 
 size_t HeaderSearch::getTotalMemory() const {
   return SearchDirs.capacity()
     + llvm::capacity_in_bytes(FileInfo)
     + llvm::capacity_in_bytes(HeaderMaps)
     + LookupFileCache.getAllocator().getTotalMemory()

clang/lib/Lex/Lexer.cpp

void Lexer::InitLexer(const char *BufStart, const char *BufPtr,
   NewLinePtr = nullptr;
 }
 
 /// Lexer constructor - Create a new lexer object for the specified buffer
 /// with the specified preprocessor managing the lexing process.  This lexer
 /// assumes that the associated file buffer and Preprocessor objects will
 /// outlive it, so it doesn't take ownership of either of them.
 Lexer::Lexer(FileID FID, const llvm::MemoryBufferRef &InputFile,
-             Preprocessor &PP)
+             Preprocessor &PP, bool IsFirstIncludeOfFile)
     : PreprocessorLexer(&PP, FID),
       FileLoc(PP.getSourceManager().getLocForStartOfFile(FID)),
-      LangOpts(PP.getLangOpts()) {
+      LangOpts(PP.getLangOpts()), IsFirstTimeLexingFile(IsFirstIncludeOfFile) {
   InitLexer(InputFile.getBufferStart(), InputFile.getBufferStart(),
             InputFile.getBufferEnd());
 
   resetExtendedTokenMode();
 }
 
 /// Lexer constructor - Create a new raw lexer object.  This object is only
 /// suitable for calls to 'LexFromRawLexer'.  This lexer assumes that the text
 /// range will outlive it, so it doesn't take ownership of it.
 Lexer::Lexer(SourceLocation fileloc, const LangOptions &langOpts,
-             const char *BufStart, const char *BufPtr, const char *BufEnd)
-    : FileLoc(fileloc), LangOpts(langOpts) {
+             const char *BufStart, const char *BufPtr, const char *BufEnd,
+             bool IsFirstIncludeOfFile)
+    : FileLoc(fileloc), LangOpts(langOpts),
+      IsFirstTimeLexingFile(IsFirstIncludeOfFile) {
   InitLexer(BufStart, BufPtr, BufEnd);
 
   // We *are* in raw mode.
   LexingRawMode = true;
 }
 
 /// Lexer constructor - Create a new raw lexer object.  This object is only
 /// suitable for calls to 'LexFromRawLexer'.  This lexer assumes that the text
 /// range will outlive it, so it doesn't take ownership of it.
 Lexer::Lexer(FileID FID, const llvm::MemoryBufferRef &FromFile,
-             const SourceManager &SM, const LangOptions &langOpts)
+             const SourceManager &SM, const LangOptions &langOpts,
+             bool IsFirstIncludeOfFile)
     : Lexer(SM.getLocForStartOfFile(FID), langOpts, FromFile.getBufferStart(),
-            FromFile.getBufferStart(), FromFile.getBufferEnd()) {}
+            FromFile.getBufferStart(), FromFile.getBufferEnd(),
+            IsFirstIncludeOfFile) {}
 
 void Lexer::resetExtendedTokenMode() {
   assert(PP && "Cannot reset token mode without a preprocessor");
   if (LangOpts.TraditionalCPP)
     SetKeepWhitespaceMode(true);
   else
     SetCommentRetentionState(PP->getCommentRetentionState());
 }

clang/lib/Lex/PPDirectives.cpp

Preprocessor::ImportAction Preprocessor::HandleHeaderIncludeOrImport(
   //
   // FIXME: If we have a suggested module for a '#include', and we've already
   // visited this file, don't bother entering it again. We know it has no
   // further effect.
   bool EnterOnce =
       IsImportDecl ||
       IncludeTok.getIdentifierInfo()->getPPKeywordID() == tok::pp_import;
 
+  bool IsFirstIncludeOfFile = false;
+
   // Ask HeaderInfo if we should enter this #include file.  If not, #including
   // this file will have no effect.
   if (Action == Enter && File &&
-      !HeaderInfo.ShouldEnterIncludeFile(*this, &File->getFileEntry(),
-                                         EnterOnce, getLangOpts().Modules,
-                                         SuggestedModule.getModule())) {
+      !HeaderInfo.ShouldEnterIncludeFile(
+          *this, &File->getFileEntry(), EnterOnce, getLangOpts().Modules,
+          SuggestedModule.getModule(), IsFirstIncludeOfFile)) {
     // Even if we've already preprocessed this header once and know that we
     // don't need to see its contents again, we still need to import it if it's
     // modular because we might not have imported it from this submodule before.
     //
     // FIXME: We don't do this when compiling a PCH because the AST
     // serialization layer can't cope with it. This means we get local
     // submodule visibility semantics wrong in that case.
     Action = (SuggestedModule && !getLangOpts().CompilingPCH) ? Import : Skip;
if (IncludePos.isMacroID())
     IncludePos = SourceMgr.getExpansionRange(IncludePos).getEnd();
   FileID FID = SourceMgr.createFileID(*File, IncludePos, FileCharacter);
   if (!FID.isValid()) {
     TheModuleLoader.HadFatalFailure = true;
     return ImportAction::Failure;
   }
 
   // If all is good, enter the new file!
-  if (EnterSourceFile(FID, CurDir, FilenameTok.getLocation()))
+  if (EnterSourceFile(FID, CurDir, FilenameTok.getLocation(),
+                      IsFirstIncludeOfFile))
     return {ImportAction::None};
 
   // Determine if we're switching to building a new submodule, and which one.
   if (auto *M = SuggestedModule.getModule()) {
     if (M->getTopLevelModule()->ShadowingModule) {
       // We are building a submodule that belongs to a shadowed module. This
       // means we find header files in the shadowed module.
       Diag(M->DefinitionLoc, diag::err_module_build_shadowed_submodule)

clang/lib/Lex/PPLexerChange.cpp

 
 //===----------------------------------------------------------------------===//
 // Methods for Entering and Callbacks for leaving various contexts
 //===----------------------------------------------------------------------===//
 
 /// EnterSourceFile - Add a source file to the top of the include stack and
 /// start lexing tokens from it instead of the current buffer.
 bool Preprocessor::EnterSourceFile(FileID FID, const DirectoryLookup *CurDir,
-                                   SourceLocation Loc) {
+                                   SourceLocation Loc,
+                                   bool IsFirstIncludeOfFile) {
   assert(!CurTokenLexer && "Cannot #include a file inside a macro!");
   ++NumEnteredSourceFiles;
 
   if (MaxIncludeStackDepth < IncludeMacroStack.size())
     MaxIncludeStackDepth = IncludeMacroStack.size();
 
   // Get the MemoryBuffer for this FID, if it fails, we fail.
   llvm::Optional<llvm::MemoryBufferRef> InputFile =
       getSourceManager().getBufferOrNone(FID, Loc);
   if (!InputFile) {
     SourceLocation FileStart = SourceMgr.getLocForStartOfFile(FID);
     Diag(Loc, diag::err_pp_error_opening_file)
         << std::string(SourceMgr.getBufferName(FileStart)) << "";
     return true;
   }
 
   if (isCodeCompletionEnabled() &&
       SourceMgr.getFileEntryForID(FID) == CodeCompletionFile) {
     CodeCompletionFileLoc = SourceMgr.getLocForStartOfFile(FID);
     CodeCompletionLoc =
         CodeCompletionFileLoc.getLocWithOffset(CodeCompletionOffset);
   }
 
-  EnterSourceFileWithLexer(new Lexer(FID, *InputFile, *this), CurDir);
+  EnterSourceFileWithLexer(
+      new Lexer(FID, *InputFile, *this, IsFirstIncludeOfFile), CurDir);
   return false;
 }
 
 /// EnterSourceFileWithLexer - Add a source file to the top of the include stack
 ///  and start lexing tokens from it instead of the current buffer.
 void Preprocessor::EnterSourceFileWithLexer(Lexer *TheLexer,
                                             const DirectoryLookup *CurDir) {
 
if (const IdentifierInfo *ControllingMacro =
         HeaderInfo.SetFileControllingMacro(FE, ControllingMacro);
         if (MacroInfo *MI =
               getMacroInfo(const_cast<IdentifierInfo*>(ControllingMacro)))
           MI->setUsedForHeaderGuard(true);
         if (const IdentifierInfo *DefinedMacro =
               CurPPLexer->MIOpt.GetDefinedMacro()) {
           if (!isMacroDefined(ControllingMacro) &&
               DefinedMacro != ControllingMacro &&
-              HeaderInfo.FirstTimeLexingFile(FE)) {
+              CurLexer->isFirstTimeLexingFile()) {
 
             // If the edit distance between the two macros is more than 50%,
             // DefinedMacro may not be header guard, or can be header guard of
             // another header file. Therefore, it maybe defining something
             // completely different. This can be observed in the wild when
             // handling feature macros or header guards in different files.
 
             const StringRef ControllingMacroName = ControllingMacro->getName();