Preprocessor: preserve whitespace in -traditional-cpp mode.

[jordan_rose]

Summary

This allows Lexer's "keep whitespace" mode to work for -E -traditional-cpp, which is used by people (ab)using the preprocessor to preprocess things other than C.

The internal uses we know about don't rely on the GNU behavior that whitespace is also preserved within macros, so this patch doesn't attempt to do that.

[rsmith]

lib/Lex/Lexer.cpp

1879–1882

Could we handle all the changes to this function by just setting IsAtStartOfLine to Result.hasFlag(Token::StartOfLine) and clearing the Token::StartOfLine flag here?

I'm also not sure whether the current approach does the right thing if (for instance) a file starts with <space>#define <...>. Will we set the StartOfLine flag for the '#' token?

2661–2666

If the sequence of whitespace ends in a newline, we would previously not set the LeadingSpace flag, now we always set it. Is that what we want? (Likewise for all the similar changes below.)

2813–2814

This is no longer correct.

test/Preprocessor/traditional-cpp.c

16–17

What are the {{ }} for here? Is this some oddity of -strict-whitespace?

[gribozavr]

test/Preprocessor/traditional-cpp.c

16–17

It might have been cleaner to use the {{^}}...{{$}} pattern for this test.

[jordan_rose]

lib/Lex/Lexer.cpp

1879–1882

That almost works, but then we'd lose the validity of StartOfLine for whitespace that does start a line, but also starts the next line. We might not care, though.

As for the start-of-file case, we do get this correct because we don't set IsAtStartOfLine to false in this method.

2661–2666

Hm. I was probably overzealous in moving around the LeadingSpace flag—it's often paired with StartOfLine, but it doesn't have the same behavior when there are whitespace tokens involved. I'll go check back on all of these.

test/Preprocessor/traditional-cpp.c

16–17

Ah, yes. Richard: these are needed because otherwise the CHECK line matches itself. Dmitri's change would allow them to match the whole line instead.

[jordan_rose]

Looking at this again, it's going to be hard to get LeadingSpace to be correct in all cases in keep-whitespace mode. I'm hesitant to add another variable like IsAtStartOfLine that will just slow down every token lexed in non-traditional mode.

(To do -traditional-cpp properly, we'd actually need LeadingSpace to be correct, because # only starts a directive if it's actually in the first column. But this patch doesn't do that anyway.)

[jordan_rose]

Make sure LeadingSpace stays correct in regular mode, and let it be incorrect in keep-whitespace mode.

[jordan_rose]

Ping.

On IRC Richard suggested that perhaps it's not worth trying to implement -traditional-cpp in Clang, since the real -traditional-cpp is character-based rather than token-based. However, I consider this more of a patch fix to satisfy our internal developers, so that they don't need a separate binary (currently a separate GCC binary) to process their whitespace-sensitive configuration files. The whitespace preservation in this patch is good enough for that, and hopefully we won't need to continue piling on hacks.

include/clang/Lex/Lexer.h

public:
   /// deal with the excess tokens.
   bool isKeepWhitespaceMode() const {
     return ExtendedTokenMode > 1;
   }
 
   /// SetKeepWhitespaceMode - This method lets clients enable or disable
   /// whitespace retention mode.
   void SetKeepWhitespaceMode(bool Val) {
-    assert((!Val || LexingRawMode) &&
-           "Can only enable whitespace retention in raw mode");
+    assert((!Val || LexingRawMode || LangOpts.TraditionalCPP) &&
+           "Can only retain whitespace in raw mode or -traditional-cpp");
     ExtendedTokenMode = Val ? 2 : 0;
   }
 
   /// inKeepCommentMode - Return true if the lexer should return comments as
   /// tokens.
   bool inKeepCommentMode() const {
     return ExtendedTokenMode > 0;
   }
 
   /// SetCommentRetentionMode - Change the comment retention mode of the lexer
   /// to the specified mode.  This is really only useful when lexing in raw
   /// mode, because otherwise the lexer needs to manage this.
   void SetCommentRetentionState(bool Mode) {
     assert(!isKeepWhitespaceMode() &&
            "Can't play with comment retention state when retaining whitespace");
     ExtendedTokenMode = Mode ? 1 : 0;
   }
 
+  /// Sets the extended token mode back to its initial value, according to the
+  /// language options and preprocessor. This controls whether the lexer
+  /// produces comment and whitespace tokens.
+  ///
+  /// This requires the lexer to have an associated preprocessor. A standalone
+  /// lexer has nothing to reset to.
+  void resetExtendedTokenMode();
+
   const char *getBufferStart() const { return BufferStart; }
 
   /// ReadToEndOfLine - Read the rest of the current preprocessor line as an
   /// uninterpreted string.  This switches the lexer out of directive mode.
   void ReadToEndOfLine(SmallVectorImpl<char> *Result = 0);
 
 
   /// Diag - Forwarding function for diagnostics.  This translate a source

lib/Lex/Lexer.cpp

 Lexer::Lexer(FileID FID, const llvm::MemoryBuffer *InputFile, Preprocessor &PP)
   : PreprocessorLexer(&PP, FID),
     FileLoc(PP.getSourceManager().getLocForStartOfFile(FID)),
     LangOpts(PP.getLangOpts()) {
 
   InitLexer(InputFile->getBufferStart(), InputFile->getBufferStart(),
             InputFile->getBufferEnd());
 
-  // Default to keeping comments if the preprocessor wants them.
-  SetCommentRetentionState(PP.getCommentRetentionState());
+  resetExtendedTokenMode();
+}
+
+void Lexer::resetExtendedTokenMode() {
+  assert(PP && "Cannot reset token mode without a preprocessor");
+  if (LangOpts.TraditionalCPP)
+    SetKeepWhitespaceMode(true);
+  else
+    SetCommentRetentionState(PP->getCommentRetentionState());
 }
 
 /// Lexer constructor - Create a new raw lexer object.  This object is only
 /// suitable for calls to 'LexFromRawLexer'.  This lexer assumes that the text
 /// range will outlive it, so it doesn't take ownership of it.
 Lexer::Lexer(SourceLocation fileloc, const LangOptions &langOpts,
              const char *BufStart, const char *BufPtr, const char *BufEnd)
   : FileLoc(fileloc), LangOpts(langOpts) {

 
 /// SkipWhitespace - Efficiently skip over a series of whitespace characters.
 /// Update BufferPtr to point to the next non-whitespace character and return.
 ///
 /// This method forms a token and returns true if KeepWhitespaceMode is enabled.
 ///
 bool Lexer::SkipWhitespace(Token &Result, const char *CurPtr) {
   // Whitespace - Skip it, then return the token after the whitespace.
+  bool SawNewline = isVerticalWhitespace(CurPtr[-1]);
+
   unsigned char Char = *CurPtr;  // Skip consequtive spaces efficiently.
   while (1) {
     // Skip horizontal whitespace very aggressively.
     while (isHorizontalWhitespace(Char))
       Char = *++CurPtr;
 
     // Otherwise if we have something other than whitespace, we're done.
     if (Char != '\n' && Char != '\r')
       break;
 
     if (ParsingPreprocessorDirective) {
       // End of preprocessor directive line, let LexTokenInternal handle this.
       BufferPtr = CurPtr;
       return false;
     }
 
     // ok, but handle newline.
-    // The returned token is at the start of the line.
-    Result.setFlag(Token::StartOfLine);
-    // No leading whitespace seen so far.
-    Result.clearFlag(Token::LeadingSpace);
+    SawNewline = true;
     Char = *++CurPtr;
   }
 
-  // If this isn't immediately after a newline, there is leading space.
-  char PrevChar = CurPtr[-1];
-  if (PrevChar != '\n' && PrevChar != '\r')
-    Result.setFlag(Token::LeadingSpace);
-
   // If the client wants us to return whitespace, return it now.
   if (isKeepWhitespaceMode()) {
     FormTokenWithChars(Result, CurPtr, tok::unknown);
+    if (SawNewline)
+      IsAtStartOfLine = true;
     return true;
   }
 
+  // If this isn't immediately after a newline, there is leading space.
+  char PrevChar = CurPtr[-1];
+  bool HasLeadingSpace = PrevChar != '\n' && PrevChar != '\r';
+
+  Result.setFlagValue(Token::LeadingSpace, HasLeadingSpace);
+  if (SawNewline)
+    Result.setFlag(Token::StartOfLine);
+
   BufferPtr = CurPtr;
   return false;
 }
 
 /// We have just read the // characters from input.  Skip until we find the
 /// newline character thats terminate the comment.  Then update BufferPtr and
 /// return.
 ///
bool Lexer::LexEndOfFile(Token &Result, const char *CurPtr) {
   // then be the end of file.
   if (ParsingPreprocessorDirective) {
     // Done parsing the "line".
     ParsingPreprocessorDirective = false;
     // Update the location of token as well as BufferPtr.
     FormTokenWithChars(Result, CurPtr, tok::eod);
 
     // Restore comment saving mode, in case it was disabled for directive.
-    SetCommentRetentionState(PP->getCommentRetentionState());
+    resetExtendedTokenMode();
     return true;  // Have a token.
   }
  
   // If we are in raw mode, return this event as an EOF token.  Let the caller
   // that put us in raw mode handle the event.
   if (isLexingRawMode()) {
     Result.startToken();
     BufferPtr = BufferEnd;
uint32_t Lexer::tryReadUCN(const char *&StartPtr, const char *SlashLoc,
   return CodePoint;
 }
 
 void Lexer::LexUnicode(Token &Result, uint32_t C, const char *CurPtr) {
   if (!isLexingRawMode() && !PP->isPreprocessedOutput() &&
       isCharInSet(C, UnicodeWhitespaceChars)) {
     Diag(BufferPtr, diag::ext_unicode_whitespace)
       << makeCharRange(*this, BufferPtr, CurPtr);
 
-    Result.setFlag(Token::LeadingSpace);
     if (SkipWhitespace(Result, CurPtr))
       return; // KeepWhitespaceMode
 
+    Result.setFlag(Token::LeadingSpace);
     return LexTokenInternal(Result);
   }
 
   if (isAllowedIDChar(C, LangOpts) && isAllowedInitiallyIDChar(C, LangOpts)) {
     if (!isLexingRawMode() && !ParsingPreprocessorDirective &&
         !PP->isPreprocessedOutput()) {
       maybeDiagnoseIDCharCompat(PP->getDiagnostics(), C,
                                 makeCharRange(*this, BufferPtr, CurPtr),
                                 /*IsFirst=*/true);
if (isCodeCompletionPoint(CurPtr-1)) {
       // Return the code-completion token.
       Result.startToken();
       FormTokenWithChars(Result, CurPtr, tok::code_completion);
       return;
     }
 
     if (!isLexingRawMode())
       Diag(CurPtr-1, diag::null_in_file);
-    Result.setFlag(Token::LeadingSpace);
     if (SkipWhitespace(Result, CurPtr))
       return; // KeepWhitespaceMode
 
+    Result.setFlag(Token::LeadingSpace);
     goto LexNextToken;   // GCC isn't tail call eliminating.
       
   case 26:  // DOS & CP/M EOF: "^Z".
     // If we're in Microsoft extensions mode, treat this as end of file.
     if (LangOpts.MicrosoftExt) {
       // Read the PP instance variable into an automatic variable, because
       // LexEndOfFile will often delete 'this'.
       Preprocessor *PPCache = PP;
case '\r':
     // If we are inside a preprocessor directive and we see the end of line,
     // we know we are done with the directive, so return an EOD token.
     if (ParsingPreprocessorDirective) {
       // Done parsing the "line".
       ParsingPreprocessorDirective = false;
 
       // Restore comment saving mode, in case it was disabled for directive.
       if (PP)
-        SetCommentRetentionState(PP->getCommentRetentionState());
+        resetExtendedTokenMode();
 
       // Since we consumed a newline, we are back at the start of a line.
       IsAtStartOfLine = true;
 
       Kind = tok::eod;
       break;
     }
+      
+    if (SkipWhitespace(Result, CurPtr))
+      return; // KeepWhitespaceMode
+
     // The returned token is at the start of the line.
     Result.setFlag(Token::StartOfLine);
     // No leading whitespace seen so far.
     Result.clearFlag(Token::LeadingSpace);
       
-    if (SkipWhitespace(Result, CurPtr))
-      return; // KeepWhitespaceMode
     goto LexNextToken;   // GCC isn't tail call eliminating.
   case ' ':
   case '\t':
   case '\f':
   case '\v':
   SkipHorizontalWhitespace:
-    Result.setFlag(Token::LeadingSpace);
     if (SkipWhitespace(Result, CurPtr))
       return; // KeepWhitespaceMode
 
+    Result.setFlag(Token::LeadingSpace);
+
   SkipIgnoredUnits:
     CurPtr = BufferPtr;
 
     // If the next token is obviously a // or /* */ comment, skip it efficiently
     // too (without going through the big switch stmt).
     if (CurPtr[0] == '/' && CurPtr[1] == '/' && !inKeepCommentMode() &&
         LangOpts.LineComment && !LangOpts.TraditionalCPP) {
       if (SkipLineComment(Result, CurPtr+2))

lib/Lex/PPDirectives.cpp

while (1) {
     // Read the next token, the directive flavor.
     LexUnexpandedToken(Tok);
 
     // If this isn't an identifier directive (e.g. is "# 1\n" or "#\n", or
     // something bogus), skip it.
     if (Tok.isNot(tok::raw_identifier)) {
       CurPPLexer->ParsingPreprocessorDirective = false;
       // Restore comment saving mode.
-      if (CurLexer) CurLexer->SetCommentRetentionState(KeepComments);
+      if (CurLexer) CurLexer->resetExtendedTokenMode();
       continue;
     }
 
     // If the first letter isn't i or e, it isn't intesting to us.  We know that
     // this is safe in the face of spelling differences, because there is no way
     // to spell an i/e in a strange way that is another letter.  Skipping this
     // allows us to avoid looking up the identifier info for #define/#undef and
     // other common directives.
     const char *RawCharData = Tok.getRawIdentifierData();
 
     char FirstChar = RawCharData[0];
     if (FirstChar >= 'a' && FirstChar <= 'z' &&
         FirstChar != 'i' && FirstChar != 'e') {
       CurPPLexer->ParsingPreprocessorDirective = false;
       // Restore comment saving mode.
-      if (CurLexer) CurLexer->SetCommentRetentionState(KeepComments);
+      if (CurLexer) CurLexer->resetExtendedTokenMode();
       continue;
     }
 
     // Get the identifier name without trigraphs or embedded newlines.  Note
     // that we can't use Tok.getIdentifierInfo() because its lookup is disabled
     // when skipping.
     char DirectiveBuf[20];
     StringRef Directive;
     if (!Tok.needsCleaning() && Tok.getLength() < 20) {
       Directive = StringRef(RawCharData, Tok.getLength());
     } else {
       std::string DirectiveStr = getSpelling(Tok);
       unsigned IdLen = DirectiveStr.size();
       if (IdLen >= 20) {
         CurPPLexer->ParsingPreprocessorDirective = false;
         // Restore comment saving mode.
-        if (CurLexer) CurLexer->SetCommentRetentionState(KeepComments);
+        if (CurLexer) CurLexer->resetExtendedTokenMode();
         continue;
       }
       memcpy(DirectiveBuf, &DirectiveStr[0], IdLen);
       Directive = StringRef(DirectiveBuf, IdLen);
     }
 
     if (Directive.startswith("if")) {
       StringRef Sub = Directive.substr(2);
if (Directive.startswith("if")) {
                             CondInfo.IfLoc);
           break;
         }
       }
     }
 
     CurPPLexer->ParsingPreprocessorDirective = false;
     // Restore comment saving mode.
-    if (CurLexer) CurLexer->SetCommentRetentionState(KeepComments);
+    if (CurLexer) CurLexer->resetExtendedTokenMode();
   }
 
   // Finally, if we are out of the conditional (saw an #endif or ran off the end
   // of the file, just stop skipping and return to lexing whatever came after
   // the #if block.
   CurPPLexer->LexingRawMode = false;
 
   if (Callbacks) {

 /// read is the correct one.
 void Preprocessor::HandleDirective(Token &Result) {
   // FIXME: Traditional: # with whitespace before it not recognized by K&R?
 
   // We just parsed a # character at the start of a line, so we're in directive
   // mode.  Tell the lexer this so any newlines we see will be converted into an
   // EOD token (which terminates the directive).
   CurPPLexer->ParsingPreprocessorDirective = true;
+  if (CurLexer) CurLexer->SetKeepWhitespaceMode(false);
 
   ++NumDirectives;
 
   // We are about to read a token.  For the multiple-include optimization FA to
   // work, we have to remember if we had read any tokens *before* this
   // pp-directive.
   bool ReadAnyTokensBeforeDirective =CurPPLexer->MIOpt.getHasReadAnyTokensVal();
 
if (InMacroArgs) {
     }
     Diag(Result, diag::ext_embedded_directive);
   }
 
   // Temporarily enable macro expansion if set so
   // and reset to previous state when returning from this function.
   ResetMacroExpansionHelper helper(this);
 
-TryAgain:
   switch (Result.getKind()) {
   case tok::eod:
     return;   // null directive.
-  case tok::comment:
-    // Handle stuff like "# /*foo*/ define X" in -E -C mode.
-    LexUnexpandedToken(Result);
-    goto TryAgain;
   case tok::code_completion:
     if (CodeComplete)
       CodeComplete->CodeCompleteDirective(
                                     CurPPLexer->getConditionalStackDepth() > 0);
     setCodeCompletionReached();
     return;
   case tok::numeric_constant:  // # 7  GNU line marker directive.
     if (getLangOpts().AsmPreprocessor)

test/Preprocessor/traditional-cpp.c

 /* Clang supports a very limited subset of -traditional-cpp, basically we only
  * intend to add support for things that people actually rely on when doing
  * things like using /usr/bin/cpp to preprocess non-source files. */
 
 /*
  RUN: %clang_cc1 -traditional-cpp %s -E -o %t
- RUN: FileCheck < %t %s
+ RUN: FileCheck -strict-whitespace < %t %s
 */
 
-/* CHECK: foo // bar
+/* CHECK: foo // {{bar}}
  */
 foo // bar
+
+
+/* The following lines contain tab characters; do not change them! */
+/* CHECK: {{i}}ndented!
+ * CHECK: {{tab}} {{separated}} {{values}}
+ */
+  indented!
+tab separated values
+
+#define bracket(x) >>>x<<<
+bracket(|  spaces  |)
+/* CHECK: >>>|  {{spaces}}  |<<<
+ */
+
+/* This is still a preprocessing directive. */
+# define foo bar
+foo!
+/* CHECK: {{bar}}!
+ */
+
+/* FIXME: -traditional-cpp should not consider this a preprocessing directive
+ * because the # isn't in the first column.
+ */
+ #define foo2 bar
+foo2!
+/* If this were working, both of these checks would be on.
+ * CHECK-NOT:  #{{define}} foo2 bar
+ * CHECK-NOT: foo2!
+ */
+
+/* FIXME: -traditional-cpp should not homogenize whitespace in macros.
+ */
+#define bracket2(x) >>>  x  <<<
+bracket2(spaces)
+/* If this were working, the second check here would be on.
+ * CHECK:  #{{define}} foo2 bar
+ * CHECK-NOT: >>>  spaces  <<<
+ */