[clang][lexer] adds a way for tokens to track their physical spelling

[cjdb]

Summary

Tokens don't currently have a way to track their source-based spelling,
which makes implementing some warnings difficult.

This patch is the basis for two independent warnings, and should be
a non-functional change.

[cor3ntin]

A few questions, also for @aaron.ballman

Can we have a "isDigraph" flag instead of keeping a pointer for the spelling?

Do we maybe want to promote and etc to proper keywords and maybe add functions to check isAndOrAmpAmp ? so that we can in the future make void f(auto and) ill-formed in the committee?
This second solution is more long term and maybe the patch is good enough as an intermediate step

clang/lib/Lex/Lexer.cpp

3594

Did you intend to remove these?

[aaron.ballman]

You should always be able to go back to the token's spelling through the SourceManager object. Is there a reason why SourceManager::getCharacterData() is insufficient? I'm a bit worried about adding another pointer to Token because we sometimes store these in containers (like we do for cached tokens), so the overhead might be noticeable (or it might not, I don't know).

[cjdb]

In D107291#2920408, @aaron.ballman wrote:

You should always be able to go back to the token's spelling through the SourceManager object. Is there a reason why SourceManager::getCharacterData() is insufficient? I'm a bit worried about adding another pointer to Token because we sometimes store these in containers (like we do for cached tokens), so the overhead might be noticeable (or it might not, I don't know).

I found it very strange that there wasn't already a way for me to do this. Thanks, I've abandoned this patch and will adjust the dependents accordingly.

clang/include/clang/Lex/Token.h

class Token {
   void *PtrData;
 
   /// Kind - The actual flavor of token this is.
   tok::TokenKind Kind;
 
   /// Flags - Bits we track about this token, members of the TokenFlags enum.
   unsigned short Flags;
 
+  /// Spelling - How a token is spelt in the source.
+  const char *Spelling;
+
 public:
   // Various flags set per token:
   enum TokenFlags {
     StartOfLine = 0x01,   // At start of line or only after whitespace
                           // (considering the line after macro expansion).
     LeadingSpace = 0x02,  // Whitespace exists before this token (considering
                           // whitespace after macro expansion).
     DisableExpand = 0x04, // This identifier may never be macro expanded.
void setAnnotationValue(void *val) {
     PtrData = val;
   }
 
   /// Set the specified flag.
   void setFlag(TokenFlags Flag) {
     Flags |= Flag;
   }
 
+  StringRef getSpelling() const { return {Spelling, getLength()}; }
+
+  void setSpelling(const char *S) { Spelling = S; }
+
   /// Get the specified flag.
   bool getFlag(TokenFlags Flag) const {
     return (Flags & Flag) != 0;
   }
 
   /// Unset the specified flag.
   void clearFlag(TokenFlags Flag) {
     Flags &= ~Flag;

clang/lib/Lex/Lexer.cpp

if (isHorizontalWhitespace(*CurPtr)) {
     BufferPtr = CurPtr;
     Result.setFlag(Token::LeadingSpace);
   }
 
   unsigned SizeTmp, SizeTmp2;   // Temporaries for use in cases below.
 
   // Read a character, advancing over it.
   char Char = getAndAdvanceChar(CurPtr, Result);
+  Result.setSpelling(CurPtr - 1);
   tok::TokenKind Kind;
 
   if (!isVerticalWhitespace(Char))
     NewLinePtr = nullptr;
 
   switch (Char) {
   case 0:  // Null.
     // Found end of file?
case '.':
     } else {
       Kind = tok::period;
     }
     break;
   case '&':
     Char = getCharAndSize(CurPtr, SizeTmp);
     if (Char == '&') {
       Kind = tok::ampamp;
+      // Result.setSpelling(CurPtr - 1);
       CurPtr = ConsumeChar(CurPtr, SizeTmp, Result);
     } else if (Char == '=') {
       Kind = tok::ampequal;
       CurPtr = ConsumeChar(CurPtr, SizeTmp, Result);
     } else {
       Kind = tok::amp;
+      // Result.setSpelling(CurPtr - 1);
     }
     break;
   case '*':
     if (getCharAndSize(CurPtr, SizeTmp) == '=') {
       Kind = tok::starequal;
       CurPtr = ConsumeChar(CurPtr, SizeTmp, Result);
     } else {
       Kind = tok::star;
case '-':
     } else if (Char == '=') {   // -=
       CurPtr = ConsumeChar(CurPtr, SizeTmp, Result);
       Kind = tok::minusequal;
     } else {
       Kind = tok::minus;
     }
     break;
   case '~':
+    // Result.setSpelling(CurPtr - 1);
     Kind = tok::tilde;
     break;
   case '!':
     if (getCharAndSize(CurPtr, SizeTmp) == '=') {
       Kind = tok::exclaimequal;
       CurPtr = ConsumeChar(CurPtr, SizeTmp, Result);
     } else {
+      // Result.setSpelling(CurPtr - 1);
       Kind = tok::exclaim;
     }
     break;
   case '/':
     // 6.4.9: Comments
     Char = getCharAndSize(CurPtr, SizeTmp);
     if (Char == '/') {         // Line comment.
       // Even if Line comments are disabled (e.g. in C89 mode), we generally
case '^':
     Char = getCharAndSize(CurPtr, SizeTmp);
     if (Char == '=') {
       CurPtr = ConsumeChar(CurPtr, SizeTmp, Result);
       Kind = tok::caretequal;
     } else if (LangOpts.OpenCL && Char == '^') {
       CurPtr = ConsumeChar(CurPtr, SizeTmp, Result);
       Kind = tok::caretcaret;
     } else {
+      // Result.setSpelling(CurPtr - 1);
       Kind = tok::caret;
     }
     break;
   case '|':
     Char = getCharAndSize(CurPtr, SizeTmp);
     if (Char == '=') {
       Kind = tok::pipeequal;
       CurPtr = ConsumeChar(CurPtr, SizeTmp, Result);
     } else if (Char == '|') {
       // If this is '|||||||' and we're in a conflict marker, ignore it.
       if (CurPtr[1] == '|' && HandleEndOfConflictMarker(CurPtr-1))
         goto LexNextToken;
+      // Result.setSpelling(CurPtr - 1);
       Kind = tok::pipepipe;
       CurPtr = ConsumeChar(CurPtr, SizeTmp, Result);
     } else {
+      // Result.setSpelling(CurPtr - 1);
       Kind = tok::pipe;
     }
     break;
   case ':':
     Char = getCharAndSize(CurPtr, SizeTmp);
     if (LangOpts.Digraphs && Char == '>') {
       Kind = tok::r_square; // ':>' -> ']'
       CurPtr = ConsumeChar(CurPtr, SizeTmp, Result);