[Lex] Make sure to notify `MultipleIncludeOpt` for "read tokens" during fast dependency directive lexing

[akyrtzi]

Summary

Otherwise a header may be erroneously marked as having a header macro guard and won't get re-included.

[benlangmuir]

clang/lib/Lex/Lexer.cpp

4251

Why do we need the >1 check? I'm not familiar with the details of MIO here.

[akyrtzi]

clang/lib/Lex/Lexer.cpp

4251

I'm making sure that MIOpt.ReadToken() is called for every token except the starting hash of a preprocessor directive, to match what regular lexing is doing.

clang/lib/Lex/Lexer.cpp

while (NextDepDirectiveTokenIndex == DepDirectives.front().Tokens.size()) {
     if (DepDirectives.front().Kind == pp_eof)
       return LexEndOfFile(Result, BufferEnd);
     NextDepDirectiveTokenIndex = 0;
     DepDirectives = DepDirectives.drop_front();
   }
 
   const dependency_directives_scan::Token &DDTok =
       DepDirectives.front().Tokens[NextDepDirectiveTokenIndex++];
+  if (NextDepDirectiveTokenIndex > 1 || DDTok.Kind != tok::hash) {
+    // Read something other than a preprocessor directive hash.
+    MIOpt.ReadToken();
+  }
 
   const char *TokPtr = convertDependencyDirectiveToken(DDTok, Result);
 
   if (Result.is(tok::hash) && Result.isAtStartOfLine()) {
     PP->HandleDirective(Result);
     return false;
   }
   if (Result.is(tok::raw_identifier)) {

clang/test/ClangScanDeps/more-content-after-headerguard.c

+// RUN: rm -rf %t
+// RUN: split-file %s %t
+// RUN: sed -e "s|DIR|%/t|g" %t/cdb.json.template > %t/cdb.json
+
+// RUN: clang-scan-deps -compilation-database %t/cdb.json | FileCheck %s
+
+// CHECK: t.c
+// CHECK: top.h
+// CHECK: n1.h
+// CHECK: n2.h
+// CHECK: n3.h
+
+//--- cdb.json.template
+[
+  {
+    "directory": "DIR",
+    "command": "clang -fsyntax-only DIR/t.c",
+    "file": "DIR/t.c"
+  }
+]
+
+//--- t.c
+
+#include "top.h"
+#define INCLUDE_N3
+#include "top.h"
+
+//--- top.h
+#ifndef _TOP_H_
+#define _TOP_H_
+
+#include "n1.h"
+
+#endif
+
+// More stuff after following '#endif', should invalidate the macro guard optimization,
+// and allow `top.h` to get re-included.
+#include "n2.h"
+
+//--- n1.h
+
+//--- n2.h
+#ifdef INCLUDE_N3
+#include "n3.h"
+#endif
+
+//--- n3.h