[Lexer] Speedup LexTokenInternal

[serge-sans-paille]

Summary

Only reset "NeedsCleaning" flag in case of re-entrant call.
Do not needlessly blank IdentifierInfo. This information will be set
once the token type is picked.

This yields a nice 1% speedup when pre-processing sqlite amalgamation
through:

valgrind --tool=callgrind ./bin/clang -E sqlite3.c -o/dev/null

[serge-sans-paille]

Compile time tracker record : https://llvm-compile-time-tracker.com/compare.php?from=f71d32a0eea47b3d2bb43d6be15cf09d47ef6971&to=b4df7d2a01aacd29ae539b04e72b3667e2362aa1&stat=instructions:u
The cost of preprocessing is hidden by the optimization time, but -O0 gives an hint of the expected speedup.

[lattner]

Nice! I'd love someone else's eyes on this, but that's a great speedup!

[aaron.ballman]

clang/lib/Lex/Lexer.cpp

3522

Was this setting the identifier info to nullptr as a shorthand to clear *any* token data that was previously built up (basically, a cheaper call to startToken())? Should we be asserting that Token::PtrData is nullptr?

4143–4145

Unrelated formatting changes?

4172–4174

Unrelated formatting changes?

[serge-sans-paille]

Add extra assert to ensure we start lexing in a clean state

[aaron.ballman]

LGTM! FYI: there's still some unrelated formatting changes ;-)

[foad]

clang/lib/Lex/Lexer.cpp

3520–3521

Messages are backwards. They should be like "Result needs cleaning!" and "Result has not been reset!".

[serge-sans-paille]

clang/lib/Lex/Lexer.cpp

3520–3521

Fixed as c8ecbaa2eba9de4338642bbfb6efd75b286878cb. Thanks for pointing that out.

clang/include/clang/Lex/Token.h

public:
   void startToken() {
     Kind = tok::unknown;
     Flags = 0;
     PtrData = nullptr;
     UintData = 0;
     Loc = SourceLocation().getRawEncoding();
   }
 
+  bool hasPtrData() const { return PtrData != nullptr; }
+
   IdentifierInfo *getIdentifierInfo() const {
     assert(isNot(tok::raw_identifier) &&
            "getIdentifierInfo() on a tok::raw_identifier token!");
     assert(!isAnnotation() &&
            "getIdentifierInfo() on an annotation token!");
     if (isLiteral()) return nullptr;
     if (is(tok::eof)) return nullptr;
     return (IdentifierInfo*) PtrData;

clang/lib/Lex/Lexer.cpp

 }
 
 /// LexTokenInternal - This implements a simple C family lexer.  It is an
 /// extremely performance critical piece of code.  This assumes that the buffer
 /// has a null character at the end of the file.  This returns a preprocessing
 /// token, not a normal token, as such, it is an internal interface.  It assumes
 /// that the Flags of result have been cleared before calling this.
 bool Lexer::LexTokenInternal(Token &Result, bool TokAtPhysicalStartOfLine) {
-LexNextToken:
-  // New token, can't need cleaning yet.
-  Result.clearFlag(Token::NeedsCleaning);
-  Result.setIdentifierInfo(nullptr);
+LexStart:
+  assert(!Result.needsCleaning() && "Result doesn't need cleaning");
+  assert(!Result.hasPtrData() && "Result has been reset");
 
   // CurPtr - Cache BufferPtr in an automatic variable.
   const char *CurPtr = BufferPtr;
 
   // Small amounts of horizontal whitespace is very common between tokens.
   if (isHorizontalWhitespace(*CurPtr)) {
     do {
       ++CurPtr;
case '^':
     break;
   case '|':
     Char = getCharAndSize(CurPtr, SizeTmp);
     if (Char == '=') {
       Kind = tok::pipeequal;
       CurPtr = ConsumeChar(CurPtr, SizeTmp, Result);
     } else if (Char == '|') {
       // If this is '|||||||' and we're in a conflict marker, ignore it.
       if (CurPtr[1] == '|' && HandleEndOfConflictMarker(CurPtr-1))
         goto LexNextToken;
       Kind = tok::pipepipe;
       CurPtr = ConsumeChar(CurPtr, SizeTmp, Result);
     } else {
       Kind = tok::pipe;
     }
     break;
   case ':':
     Char = getCharAndSize(CurPtr, SizeTmp);
     if (LangOpts.Digraphs && Char == '>') {
case ':':
     break;
   case ';':
     Kind = tok::semi;
     break;
   case '=':
     Char = getCharAndSize(CurPtr, SizeTmp);
     if (Char == '=') {
       // If this is '====' and we're in a conflict marker, ignore it.
       if (CurPtr[1] == '=' && HandleEndOfConflictMarker(CurPtr-1))
         goto LexNextToken;
 
       Kind = tok::equalequal;
       CurPtr = ConsumeChar(CurPtr, SizeTmp, Result);
     } else {
       Kind = tok::equal;
     }
     break;
   case ',':
     Kind = tok::comma;
HandleDirective:
   if (PP->hadModuleLoaderFatalFailure()) {
     // With a fatal failure in the module loader, we abort parsing.
     assert(Result.is(tok::eof) && "Preprocessor did not set tok:eof");
     return true;
   }
 
   // We parsed the directive; lex a token with the new state.
   return false;
+
+LexNextToken:
+  Result.clearFlag(Token::NeedsCleaning);
+  goto LexStart;
 }
 
 const char *Lexer::convertDependencyDirectiveToken(
     const dependency_directives_scan::Token &DDTok, Token &Result) {
   const char *TokPtr = BufferStart + DDTok.Offset;
   Result.startToken();
   Result.setLocation(getSourceLocation(TokPtr));
   Result.setKind(DDTok.Kind);