Stringizing raw string literals containing newline

[twoh]

Summary

This patch implements 4.3 of http://open-std.org/jtc1/sc22/wg21/docs/papers/2014/n4220.pdf. If a raw string contains a newline character, replace each newline character with the \n escape code. Without this patch, included test case (macro_raw_string.cpp) results compilation failure.

[twoh]

clang-format

[twoh]

Ping. Thanks!

[vsapsai]

On the first glance string erase and insert seem to be done correctly. But we can still improve the tests. Can you please add tests for tricky cases where new line or backslash appear in unexpected places? Some of the examples are

R"ab\
cd"

R"ab

c"

R"ab
\"

R"ab
\\"

I didn't check if all of them are valid but I hope you got the idea.

lib/Lex/Lexer.cpp

246

getEscapedNewLineSize mentions

P[-1] is known to be a "\" or a trigraph equivalent on entry to this function.

Is this precondition correct in this case?

And std::string::substr creates a copy of a substring. It is inefficient in the loop and looks like you don't really need std::string here anyway.

[twoh]

Addressing @vsapsai's comments. Thank you for the suggestion! Added test case actually finds an off-by-one error in the original patch. I improved the comments as well.

[jkorous-apple]

lib/Lex/Lexer.cpp

217

Wouldn't auto or typename T::size_type instead of unsigned be more appropriate here?

Both of your supported use cases have this member type.
http://llvm.org/doxygen/classllvm_1_1StringRef.html#a54e59e2d53e5ee736ee060be7c457508
http://llvm.org/doxygen/classllvm_1_1SmallVectorImpl.html#acc72e8846802a1e703501219cf19458e

231

I am just wondering if potential performance benefit of counting all the extra space in advance and resizing the string just once might be interesting here.
Basically with current approach characters at the end of the string are moved as many times as there are endlines in the string.

[twoh]

Thanks @jkorous-apple for your comments. I modified the type for the variables and replaced unnecessary inserts and erases with updates.

[jkorous-apple]

I am sorry I wasn't really clear. What I meant was to do something like this (pseudo-code, dealing only with newlines):

if( Str.size() == 0)
  return;

// Calculate all the extra space needed first.

typename T::size_type extra_space = 0;
bool previous_char_was_endline = false;

for(const auto ch : Str) {
  if( ch == '\n' || ch == '\r' ) {
    if( !previous_char_was_endline )
      ++extra_space;
    previous_char_was_endline = true;
  } else {
    previous_char_was_endline = false;
  }
}

if (extra_space == 0)
  return; 

// Resize the string.

const typename T::size_type original_size = Str.size();
Str.resize(original_size + extra_space);

// Iterate backwards and move characters as needed.

bool is_in_block_of_endlines = false;

for(typename T::size_type i = original_size - 1; i > 0; --i) {
  if( Str[i] == '\n' || Str[i] == '\r' ) {
    if (!is_in_block_of_endlines) {
      Str[i + extra_space - 1] = '\\';
      Str[i + extra_space] = 'n';
      --extra_space;
      if(extra_space == 0)
        return; // early exit - no more characters need to be moved
    }
    is_in_block_of_endlines = true;
  } else {
    Str[i + extra_space] = Str[i];
    is_in_block_of_endlines = false;
  }
}

This is just a suggestion, it is a bit more complicated but should be O(lenght_of_string) whereas your solution is a bit more straightforward but is more like O(length_of_string * number_of_endlines_in_string). I leave it up to you what is better here. If you decide to go this way, please assume my pseudocode is buggy and don't rely on it.

[twoh]

@jkorous-apple Thanks for the comments! Yeah, I was thinking of O(lenght_of_string) approach, but considering the complicatedness of the implementation (I guess the real implementation would be a bit more complex than your pseudo implementation to handle quote and '\n\r' '\r\n' cases) I decided to stay with O(length_of_string * number_of_endlines_in_string) but optimizing the number of move operations.

@vsapsai @jkorous-apple, I wonder if you can actually approve the patch or suggest other reviewers? Thanks!

[jkorous-apple]

Thank you for your patience @twoh and sorry for the delay.
I have few suggestions about doxygen annotations but otherwise LGTM.

include/clang/Lex/Lexer.h

247

Shouldn't we put all the details from implementation annotation here as well (since this is the public interface that people will actually use)?

251

Shouldn't we put all the details from implementation annotation here as well (since this is the public interface that people will actually use)?

lib/Lex/Lexer.cpp

214

I am not sure I understand this correctly but wouldn't it be more precise if these literals are escaped?
... escaping '\' ... -> ...escaping '\\' ...
... with "\n" ... -> ... with "\\n"

Alternatively we could use R"(\)" and R"(\n)".

[twoh]

Thanks @jkorous-apple for the comment. I think your suggestion is a more
precise description for the implementation, and adjusted the comments
accordingly.

I intentionally didn't add the details to the comments for the header,
as LLVM coding standard "Don’t duplicate the documentation comment in
the header file and in the implementation file. ... implementation files
can include additional comments (not necessarily in Doxygen markup) to
explain implementation details as needed." I'd like to leave it as of
now if you're not strongly against it.

Thank you again!

[jkorous-apple]

I see your concern. Would you just consider moving the annotation to header file then? It looks more like documentation of public interface than implementation details to me. I imagine any programmer interested in these methods needs to know these details so they would probably appreciate not having to search for it.

Anyway, I don't want to dwell on minor details and it's probably a matter of opinion as well. If you prefer to leave it as it is feel free to commit your patch.

Good job!

LGTM.

[twoh]

@jkorous-apple Got it. I agree that it would be better to move the comments to the header. Will land it soon. Thanks!

[twoh]

clang-format

include/clang/Lex/Lexer.h

class Lexer : public PreprocessorLexer {
 
   //===--------------------------------------------------------------------===//
   // Constant configuration values for this lexer.
   const char *BufferStart;       // Start of the buffer.
   const char *BufferEnd;         // End of the buffer.
   SourceLocation FileLoc;        // Location for start of file.
   LangOptions LangOpts;          // LangOpts enabled by this language (cache).
   bool Is_PragmaLexer;           // True if lexer for _Pragma handling.
-  
   //===--------------------------------------------------------------------===//
   // Context-specific lexing flags set by the preprocessor.
   //
 
   /// ExtendedTokenMode - The lexer can optionally keep comments and whitespace
   /// and return them as tokens.  This is used for -C and -CC modes, and
   /// whitespace preservation can be useful for some clients that want to lex
   /// the file in raw mode and get every character from the file.
public:
   /// getSourceLocation - Return a source location for the next character in
   /// the current file.
   SourceLocation getSourceLocation() override {
     return getSourceLocation(BufferPtr);
   }
 
   /// \brief Return the current location in the buffer.
   const char *getBufferLocation() const { return BufferPtr; }
-  
-  /// Stringify - Convert the specified string into a C string by escaping '\'
-  /// and " characters.  This does not add surrounding ""'s to the string.
-  /// If Charify is true, this escapes the ' character instead of ".
+  /// Stringify - Convert the specified string into a C string. This does not
+  /// add surrounding ""'s to the string. If Charify is true, this escapes the
+  /// ' character instead of ".
   static std::string Stringify(StringRef Str, bool Charify = false);
 
-  /// Stringify - Convert the specified string into a C string by escaping '\'
-  /// and " characters.  This does not add surrounding ""'s to the string.
+  /// Stringify - Convert the specified string into a C string. This does not
+  /// add surrounding ""'s to the string.
   static void Stringify(SmallVectorImpl<char> &Str);
 
-  
   /// getSpelling - This method is used to get the spelling of a token into a
   /// preallocated buffer, instead of as an std::string.  The caller is required
   /// to allocate enough space for the token, which is guaranteed to be at least
   /// Tok.getLength() bytes long.  The length of the actual result is returned.
   ///
   /// Note that this method may do two possible things: it may either fill in
   /// the buffer specified with characters, or it may *change the input pointer*
   /// to point to a constant buffer with the data already in it (avoiding a
   /// copy).  The caller is not allowed to modify the returned buffer pointer
   /// if an internal buffer is returned.
-  static unsigned getSpelling(const Token &Tok, const char *&Buffer, 
+  static unsigned getSpelling(const Token &Tok, const char *&Buffer,
                               const SourceManager &SourceMgr,
                               const LangOptions &LangOpts,
                               bool *Invalid = nullptr);
-  
   /// getSpelling() - Return the 'spelling' of the Tok token.  The spelling of a
   /// token is the characters used to represent the token in the source file
   /// after trigraph expansion and escaped-newline folding.  In particular, this
   /// wants to get the true, uncanonicalized, spelling of things like digraphs
   /// UCNs, etc.
   static std::string getSpelling(const Token &Tok,
                                  const SourceManager &SourceMgr,
-                                 const LangOptions &LangOpts, 
+                                 const LangOptions &LangOpts,
                                  bool *Invalid = nullptr);
 
   /// getSpelling - This method is used to get the spelling of the
   /// token at the given source location.  If, as is usually true, it
   /// is not necessary to copy any data, then the returned string may
   /// not point into the provided buffer.
   ///
   /// This method lexes at the expansion depth of the given
   /// location and does not jump to the expansion or spelling
   /// location.
   static StringRef getSpelling(SourceLocation loc,
                                SmallVectorImpl<char> &buffer,
                                const SourceManager &SourceMgr,
                                const LangOptions &LangOpts,
                                bool *invalid = nullptr);
-  
   /// MeasureTokenLength - Relex the token at the specified location and return
   /// its length in bytes in the input file.  If the token needs cleaning (e.g.
   /// includes a trigraph or an escaped newline) then this count includes bytes
   /// that are part of that.
   static unsigned MeasureTokenLength(SourceLocation Loc,
                                      const SourceManager &SM,
                                      const LangOptions &LangOpts);
 
   /// \brief Relex the token at the specified location.
   /// \returns true if there was a failure, false on success.
   static bool getRawToken(SourceLocation Loc, Token &Result,
                           const SourceManager &SM,
                           const LangOptions &LangOpts,
                           bool IgnoreWhiteSpace = false);
 
   /// \brief Given a location any where in a source buffer, find the location
   /// that corresponds to the beginning of the token in which the original
   /// source location lands.
   static SourceLocation GetBeginningOfToken(SourceLocation Loc,
                                             const SourceManager &SM,
                                             const LangOptions &LangOpts);
-  
   /// AdvanceToTokenCharacter - If the current SourceLocation specifies a
   /// location at the start of a token, return a new location that specifies a
   /// character within the token.  This handles trigraphs and escaped newlines.
   static SourceLocation AdvanceToTokenCharacter(SourceLocation TokStart,
                                                 unsigned Character,
                                                 const SourceManager &SM,
                                                 const LangOptions &LangOpts);
-  
   /// \brief Computes the source location just past the end of the
   /// token at this source location.
   ///
   /// This routine can be used to produce a source location that
   /// points just past the end of the token referenced by \p Loc, and
   /// is generally used when a diagnostic needs to point just after a
   /// token where it expected something different that it received. If
   /// the returned source location would not be meaningful (e.g., if
private:
   bool LexEndOfFile          (Token &Result, const char *CurPtr);
   bool SkipWhitespace        (Token &Result, const char *CurPtr,
                               bool &TokAtPhysicalStartOfLine);
   bool SkipLineComment       (Token &Result, const char *CurPtr,
                               bool &TokAtPhysicalStartOfLine);
   bool SkipBlockComment      (Token &Result, const char *CurPtr,
                               bool &TokAtPhysicalStartOfLine);
   bool SaveLineComment       (Token &Result, const char *CurPtr);
-  
   bool IsStartOfConflictMarker(const char *CurPtr);
   bool HandleEndOfConflictMarker(const char *CurPtr);
 
   bool lexEditorPlaceholder(Token &Result, const char *CurPtr);
 
   bool isCodeCompletionPoint(const char *CurPtr) const;
   void cutOffLexing() { BufferPtr = BufferEnd; }
 

lib/Lex/Lexer.cpp

Lexer *Lexer::Create_PragmaLexer(SourceLocation SpellingLoc,
   // return an EOD token.
   L->ParsingPreprocessorDirective = true;
 
   // This lexer really is for _Pragma.
   L->Is_PragmaLexer = true;
   return L;
 }
 
-/// Stringify - Convert the specified string into a C string, with surrounding
-/// ""'s, and with escaped \ and " characters.
+/// StringifyImpl - Implementation of Stringify functions. Convert the
+/// specified string into a C string by i) escaping '\' and " characters and
+/// ii) replacing newline character(s) with "\n".
+template <typename T>
+void StringifyImpl(T& Str, char Quote) {
+  typename T::size_type i = 0, e = Str.size();
+  while (i < e) {
+    if (Str[i] == '\\' || Str[i] == Quote) {
+      Str.insert(Str.begin() + i, '\\');
+      i += 2;
+      ++e;
+    } else if (Str[i] == '\n' || Str[i] == '\r') {
+      // Replace '\r\n' and '\n\r' to '\\' followed by 'n'.
+      if ((i < e - 1) && (Str[i + 1] == '\n' || Str[i + 1] == '\r') &&
+          Str[i] != Str[i + 1]) {
+        Str[i] = '\\';
+        Str[i+1] = 'n';
+      } else {
+        // Replace '\n' and '\r' to '\\' followed by 'n'.
+        Str[i] = '\\';
+        Str.insert(Str.begin() + i + 1, 'n');
+        ++e;
+      }
+      i += 2;
+    } else
+      ++i;
+  }
+}
+
 std::string Lexer::Stringify(StringRef Str, bool Charify) {
   std::string Result = Str;
   char Quote = Charify ? '\'' : '"';
-  for (unsigned i = 0, e = Result.size(); i != e; ++i) {
-    if (Result[i] == '\\' || Result[i] == Quote) {
-      Result.insert(Result.begin()+i, '\\');
-      ++i; ++e;
-    }
-  }
+  StringifyImpl(Result, Quote);
   return Result;
 }
 
-/// Stringify - Convert the specified string into a C string by escaping '\'
-/// and " characters.  This does not add surrounding ""'s to the string.
 void Lexer::Stringify(SmallVectorImpl<char> &Str) {
-  for (unsigned i = 0, e = Str.size(); i != e; ++i) {
-    if (Str[i] == '\\' || Str[i] == '"') {
-      Str.insert(Str.begin()+i, '\\');
-      ++i; ++e;
-    }
-  }
+  StringifyImpl(Str, '"');
 }
 
 //===----------------------------------------------------------------------===//
 // Token Spelling
 //===----------------------------------------------------------------------===//
 
 /// \brief Slow case of getSpelling. Extract the characters comprising the
 /// spelling of this token from the provided input buffer.

 /// to allocate enough space for the token, which is guaranteed to be at least
 /// Tok.getLength() bytes long.  The actual length of the token is returned.
 ///
 /// Note that this method may do two possible things: it may either fill in
 /// the buffer specified with characters, or it may *change the input pointer*
 /// to point to a constant buffer with the data already in it (avoiding a
 /// copy).  The caller is not allowed to modify the returned buffer pointer
 /// if an internal buffer is returned.
-unsigned Lexer::getSpelling(const Token &Tok, const char *&Buffer, 
+unsigned Lexer::getSpelling(const Token &Tok, const char *&Buffer,
                             const SourceManager &SourceMgr,
                             const LangOptions &LangOpts, bool *Invalid) {
   assert((int)Tok.getLength() >= 0 && "Token character range is bogus!");
 
   const char *TokStart = nullptr;
   // NOTE: this has to be checked *before* testing for an IdentifierInfo.
   if (Tok.is(tok::raw_identifier))
     TokStart = Tok.getRawIdentifier().data();
PreambleBounds Lexer::ComputePreamble(StringRef Buffer,
   do {
     TheLexer.LexFromRawLexer(TheTok);
 
     if (InPreprocessorDirective) {
       // If we've hit the end of the file, we're done.
       if (TheTok.getKind() == tok::eof) {
         break;
       }
-      
       // If we haven't hit the end of the preprocessor directive, skip this
       // token.
       if (!TheTok.isAtStartOfLine())
         continue;
-        
       // We've passed the end of the preprocessor directive, and will look
       // at this token again below.
       InPreprocessorDirective = false;
     }
-    
     // Keep track of the # of lines in the preamble.
     if (TheTok.isAtStartOfLine()) {
       unsigned TokOffset = TheTok.getLocation().getRawEncoding() - StartOffset;
 
       // If we were asked to limit the number of lines in the preamble,
       // and we're about to exceed that limit, we're done.
       if (MaxLineOffset && TokOffset >= MaxLineOffset)
         break;
     }
 
     // Comments are okay; skip over them.
     if (TheTok.getKind() == tok::comment) {
       if (ActiveCommentLoc.isInvalid())
         ActiveCommentLoc = TheTok.getLocation();
       continue;
     }
-    
     if (TheTok.isAtStartOfLine() && TheTok.getKind() == tok::hash) {
-      // This is the start of a preprocessor directive. 
+      // This is the start of a preprocessor directive.
       Token HashTok = TheTok;
       InPreprocessorDirective = true;
       ActiveCommentLoc = SourceLocation();
-      
       // Figure out which directive this is. Since we're lexing raw tokens,
       // we don't have an identifier table available. Instead, just look at
       // the raw identifier to recognize and categorize preprocessor directives.
       TheLexer.LexFromRawLexer(TheTok);
       if (TheTok.getKind() == tok::raw_identifier && !TheTok.needsCleaning()) {
         StringRef Keyword = TheTok.getRawIdentifier();
         PreambleDirectiveKind PDK
           = llvm::StringSwitch<PreambleDirectiveKind>(Keyword)
if (TheTok.isAtStartOfLine() && TheTok.getKind() == tok::hash) {
         case PDK_Skipped:
           continue;
 
         case PDK_Unknown:
           // We don't know what this directive is; stop at the '#'.
           break;
         }
       }
-      
       // We only end up here if we didn't recognize the preprocessor
       // directive or it was one that can't occur in the preamble at this
       // point. Roll back the current token to the location of the '#'.
       InPreprocessorDirective = false;
       TheTok = HashTok;
     }
 
     // We hit a token that we don't recognize as being in the
     // "preprocessing only" part of the file, so we're no longer in
     // the preamble.
     break;
   } while (true);
-  
   SourceLocation End;
   if (ActiveCommentLoc.isValid())
     End = ActiveCommentLoc; // don't truncate a decl comment.
   else
     End = TheTok.getLocation();
 
   return PreambleBounds(End.getRawEncoding() - FileLoc.getRawEncoding(),
                         TheTok.isAtStartOfLine());
 }
 
 /// AdvanceToTokenCharacter - Given a location that specifies the start of a
 /// token, return a new location that specifies a character within the token.
 SourceLocation Lexer::AdvanceToTokenCharacter(SourceLocation TokStart,
                                               unsigned CharNo,
                                               const SourceManager &SM,
                                               const LangOptions &LangOpts) {
   // Figure out how many physical characters away the specified expansion
   // character is.  This needs to take into consideration newlines and
   // trigraphs.
   bool Invalid = false;
   const char *TokPtr = SM.getCharacterData(TokStart, &Invalid);
-  
   // If they request the first char of the token, we're trivially done.
   if (Invalid || (CharNo == 0 && Lexer::isObviouslySimpleCharacter(*TokPtr)))
     return TokStart;
-  
   unsigned PhysOffset = 0;
-  
   // The usual case is that tokens don't contain anything interesting.  Skip
   // over the uninteresting characters.  If a token only consists of simple
   // chars, this method is extremely fast.
   while (Lexer::isObviouslySimpleCharacter(*TokPtr)) {
     if (CharNo == 0)
       return TokStart.getLocWithOffset(PhysOffset);
     ++TokPtr;
     --CharNo;
     ++PhysOffset;
   }
-  
   // If we have a character that may be a trigraph or escaped newline, use a
   // lexer to parse it correctly.
   for (; CharNo; --CharNo) {
     unsigned Size;
     Lexer::getCharAndSizeNoWarn(TokPtr, Size, LangOpts);
     TokPtr += Size;
     PhysOffset += Size;
   }
-  
   // Final detail: if we end up on an escaped newline, we want to return the
   // location of the actual byte of the token.  For example foo\<newline>bar
   // advanced by 3 should return the location of b, not of \\.  One compounding
   // detail of this is that the escape may be made by a trigraph.
   if (!Lexer::isObviouslySimpleCharacter(*TokPtr))
     PhysOffset += Lexer::SkipEscapedNewLines(TokPtr)-TokPtr;
-  
   return TokStart.getLocWithOffset(PhysOffset);
 }
 
 /// \brief Computes the source location just past the end of the
 /// token at this source location.
 ///
 /// This routine can be used to produce a source location that
 /// points just past the end of the token referenced by \p Loc, and
if (Offset > 0 || !isAtEndOfMacroExpansion(Loc, SM, LangOpts, &Loc))
       return SourceLocation(); // Points inside the macro expansion.
   }
 
   unsigned Len = Lexer::MeasureTokenLength(Loc, SM, LangOpts);
   if (Len > Offset)
     Len = Len - Offset;
   else
     return Loc;
-  
   return Loc.getLocWithOffset(Len);
 }
 
 /// \brief Returns true if the given MacroID location points at the first
 /// token of the macro expansion.
 bool Lexer::isAtStartOfMacroExpansion(SourceLocation loc,
                                       const SourceManager &SM,
                                       const LangOptions &LangOpts,
while (true) {
     const SrcMgr::SLocEntry *E = &SM.getSLocEntry(FID);
     const SrcMgr::ExpansionInfo &Expansion = E->getExpansion();
     Loc = Expansion.getExpansionLocStart();
     if (!Expansion.isMacroArgExpansion())
       break;
 
     // For macro arguments we need to check that the argument did not come
     // from an inner macro, e.g: "MAC1( MAC2(foo) )"
-    
     // Loc points to the argument id of the macro definition, move to the
     // macro expansion.
     Loc = SM.getImmediateExpansionRange(Loc).first;
     SourceLocation SpellLoc = Expansion.getSpellingLoc();
     if (SpellLoc.isFileID())
       break; // No inner macro.
 
     // If spelling location resides in the same FileID as macro expansion
Diag(BufferPtr, getLangOpts().CPlusPlus
            : diag::warn_c99_compat_unicode_literal);
 
   char C = getAndAdvanceChar(CurPtr, Result);
   while (C != '"') {
     // Skip escaped characters.  Escaped newlines will already be processed by
     // getAndAdvanceChar.
     if (C == '\\')
       C = getAndAdvanceChar(CurPtr, Result);
-    
     if (C == '\n' || C == '\r' ||             // Newline.
         (C == 0 && CurPtr-1 == BufferEnd)) {  // End of file.
       if (!isLexingRawMode() && !LangOpts.AsmPreprocessor)
         Diag(BufferPtr, diag::ext_unterminated_char_or_string) << 1;
       FormTokenWithChars(Result, CurPtr-1, tok::unknown);
       return true;
     }
-    
     if (C == 0) {
       if (isCodeCompletionPoint(CurPtr-1)) {
         PP->CodeCompleteNaturalLanguage();
         FormTokenWithChars(Result, CurPtr-1, tok::unknown);
         cutOffLexing();
         return true;
       }
 
if (!ParsingPreprocessorDirective || LexingRawMode)
     return true;
 
   // If this Line-style comment is in a macro definition, transmogrify it into
   // a C-style block comment.
   bool Invalid = false;
   std::string Spelling = PP->getSpelling(Result, &Invalid);
   if (Invalid)
     return true;
-  
   assert(Spelling[0] == '/' && Spelling[1] == '/' && "Not line comment?");
   Spelling[1] = '*';   // Change prefix to "/*".
   Spelling += "*/";    // add suffix.
 
   Result.setKind(tok::comment);
   PP->CreateString(Spelling, Result,
                    Result.getLocation(), Result.getLocation());
   return true;
if (ParsingPreprocessorDirective) {
     // Update the location of token as well as BufferPtr.
     FormTokenWithChars(Result, CurPtr, tok::eod);
 
     // Restore comment saving mode, in case it was disabled for directive.
     if (PP)
       resetExtendedTokenMode();
     return true;  // Have a token.
   }
- 
   // If we are in raw mode, return this event as an EOF token.  Let the caller
   // that put us in raw mode handle the event.
   if (isLexingRawMode()) {
     Result.startToken();
     BufferPtr = BufferEnd;
     FormTokenWithChars(Result, BufferEnd, tok::eof);
     return true;
   }
-  
   if (PP->isRecordingPreamble() && PP->isInPrimaryFile()) {
     PP->setRecordedPreambleConditionalStack(ConditionalStack);
     ConditionalStack.clear();
   }
 
   // Issue diagnostics for unterminated #if and missing newline.
 
   // If we are in a #if directive, emit an error.

 /// control conflict marker like '<<<<<<<', recognize it as such, emit an error
 /// and recover nicely.  This returns true if it is a conflict marker and false
 /// if not.
 bool Lexer::IsStartOfConflictMarker(const char *CurPtr) {
   // Only a conflict marker if it starts at the beginning of a line.
   if (CurPtr != BufferStart &&
       CurPtr[-1] != '\n' && CurPtr[-1] != '\r')
     return false;
-  
   // Check to see if we have <<<<<<< or >>>>.
   if (!StringRef(CurPtr, BufferEnd - CurPtr).startswith("<<<<<<<") &&
       !StringRef(CurPtr, BufferEnd - CurPtr).startswith(">>>> "))
     return false;
 
   // If we have a situation where we don't care about conflict markers, ignore
   // it.
   if (CurrentConflictMarkerState || isLexingRawMode())
     return false;
-  
   ConflictMarkerKind Kind = *CurPtr == '<' ? CMK_Normal : CMK_Perforce;
 
   // Check to see if there is an ending marker somewhere in the buffer at the
   // start of a line to terminate this conflict marker.
   if (FindConflictEnd(CurPtr, BufferEnd, Kind)) {
     // We found a match.  We are really in a conflict marker.
     // Diagnose this, and ignore to the end of line.
     Diag(CurPtr, diag::err_conflict_marker);
     CurrentConflictMarkerState = Kind;
-    
     // Skip ahead to the end of line.  We know this exists because the
     // end-of-conflict marker starts with \r or \n.
     while (*CurPtr != '\r' && *CurPtr != '\n') {
       assert(CurPtr != BufferEnd && "Didn't find end of line");
       ++CurPtr;
     }
     BufferPtr = CurPtr;
     return true;
   }
-  
   // No end of conflict marker found.
   return false;
 }
 
 /// HandleEndOfConflictMarker - If this is a '====' or '||||' or '>>>>', or if
 /// it is '<<<<' and the conflict marker started with a '>>>>' marker, then it
 /// is the end of a conflict marker.  Handle it by ignoring up until the end of
 /// the line.  This returns true if it is a conflict marker and false if not.
 bool Lexer::HandleEndOfConflictMarker(const char *CurPtr) {
   // Only a conflict marker if it starts at the beginning of a line.
   if (CurPtr != BufferStart &&
       CurPtr[-1] != '\n' && CurPtr[-1] != '\r')
     return false;
-  
   // If we have a situation where we don't care about conflict markers, ignore
   // it.
   if (!CurrentConflictMarkerState || isLexingRawMode())
     return false;
-  
   // Check to see if we have the marker (4 characters in a row).
   for (unsigned i = 1; i != 4; ++i)
     if (CurPtr[i] != CurPtr[0])
       return false;
-  
   // If we do have it, search for the end of the conflict marker.  This could
   // fail if it got skipped with a '#if 0' or something.  Note that CurPtr might
   // be the end of conflict marker.
   if (const char *End = FindConflictEnd(CurPtr, BufferEnd,
                                         CurrentConflictMarkerState)) {
     CurPtr = End;
-    
     // Skip ahead to the end of line.
     while (CurPtr != BufferEnd && *CurPtr != '\r' && *CurPtr != '\n')
       ++CurPtr;
-    
     BufferPtr = CurPtr;
-    
     // No longer in the conflict marker.
     CurrentConflictMarkerState = CMK_None;
     return true;
   }
-  
   return false;
 }
 
 static const char *findPlaceholderEnd(const char *CurPtr,
                                       const char *BufferEnd) {
   if (CurPtr == BufferEnd)
     return nullptr;
   BufferEnd -= 1; // Scan until the second last character.
if (!isLexingRawMode())
       Diag(CurPtr-1, diag::null_in_file);
     Result.setFlag(Token::LeadingSpace);
     if (SkipWhitespace(Result, CurPtr, TokAtPhysicalStartOfLine))
       return true; // KeepWhitespaceMode
 
     // We know the lexer hasn't changed, so just try again with this lexer.
     // (We manually eliminate the tail call to avoid recursion.)
     goto LexNextToken;
-      
   case 26:  // DOS & CP/M EOF: "^Z".
     // If we're in Microsoft extensions mode, treat this as end of file.
     if (LangOpts.MicrosoftExt) {
       if (!isLexingRawMode())
         Diag(CurPtr-1, diag::ext_ctrl_z_eof_microsoft);
       return LexEndOfFile(Result, CurPtr-1);
     }
 
     // If Microsoft extensions are disabled, this is just random garbage.
     Kind = tok::unknown;
     break;
-      
   case '\r':
     if (CurPtr[0] == '\n')
       Char = getAndAdvanceChar(CurPtr, Result);
     LLVM_FALLTHROUGH;
   case '\n':
     // If we are inside a preprocessor directive and we see the end of line,
     // we know we are done with the directive, so return an EOD token.
     if (ParsingPreprocessorDirective) {
if (CurPtr[0] == '/' && CurPtr[1] == '/' && !inKeepCommentMode() &&
         return true; // There is a token to return.
       goto SkipIgnoredUnits;
     } else if (isHorizontalWhitespace(*CurPtr)) {
       goto SkipHorizontalWhitespace;
     }
     // We only saw whitespace, so just try again with this lexer.
     // (We manually eliminate the tail call to avoid recursion.)
     goto LexNextToken;
-      
   // C99 6.4.4.1: Integer Constants.
   // C99 6.4.4.2: Floating Constants.
   case '0': case '1': case '2': case '3': case '4':
   case '5': case '6': case '7': case '8': case '9':
     // Notify MIOpt that we read a non-whitespace/non-comment token.
     MIOpt.ReadToken();
     return LexNumericConstant(Result, CurPtr);
 
case ';':
     Kind = tok::semi;
     break;
   case '=':
     Char = getCharAndSize(CurPtr, SizeTmp);
     if (Char == '=') {
       // If this is '====' and we're in a conflict marker, ignore it.
       if (CurPtr[1] == '=' && HandleEndOfConflictMarker(CurPtr-1))
         goto LexNextToken;
-      
       Kind = tok::equalequal;
       CurPtr = ConsumeChar(CurPtr, SizeTmp, Result);
     } else {
       Kind = tok::equal;
     }
     break;
   case ',':
     Kind = tok::comma;
if (Status == llvm::conversionOK) {
           return true; // KeepWhitespaceMode
 
         // We only saw whitespace, so just try again with this lexer.
         // (We manually eliminate the tail call to avoid recursion.)
         goto LexNextToken;
       }
       return LexUnicode(Result, CodePoint, CurPtr);
     }
-    
     if (isLexingRawMode() || ParsingPreprocessorDirective ||
         PP->isPreprocessedOutput()) {
       ++CurPtr;
       Kind = tok::unknown;
       break;
     }
 
     // Non-ASCII characters tend to creep into source code unintentionally.

test/Preprocessor/macro_raw_string.cpp

+// RUN: %clang_cc1 -E -std=c++11 %s -o %t
+// RUN: %clang_cc1 %t
+
+#define FOO(str) foo(#str)
+
+extern void foo(const char *str);
+
+void bar() {
+  FOO(R"(foo
+    bar)");
+}

unittests/Lex/LexerTest.cpp

 // The test fixture.
 class LexerTest : public ::testing::Test {
 protected:
   LexerTest()
     : FileMgr(FileMgrOpts),
       DiagID(new DiagnosticIDs()),
       Diags(DiagID, new DiagnosticOptions, new IgnoringDiagConsumer()),
       SourceMgr(Diags, FileMgr),
-      TargetOpts(new TargetOptions) 
+      TargetOpts(new TargetOptions)
   {
     TargetOpts->Triple = "x86_64-apple-darwin11.1.0";
     Target = TargetInfo::CreateTargetInfo(Diags, TargetOpts);
   }
 
   std::unique_ptr<Preprocessor> CreatePP(StringRef Source,
                                          TrivialModuleLoader &ModLoader) {
     std::unique_ptr<llvm::MemoryBuffer> Buf =
TEST_F(LexerTest, GetBeginningOfTokenWithEscapedNewLine) {
   }
 }
 
 TEST_F(LexerTest, AvoidPastEndOfStringDereference) {
   std::vector<Token> LexedTokens = Lex("  //  \\\n");
   EXPECT_TRUE(LexedTokens.empty());
 }
 
+TEST_F(LexerTest, StringizingRasString) {
+  // For "std::string Lexer::Stringify(StringRef Str, bool Charify)".
+  std::string String1 = R"(foo
+    {"bar":[]}
+    baz)";
+  // For "void Lexer::Stringify(SmallVectorImpl<char> &Str)".
+  SmallString<128> String2;
+  String2 += String1.c_str();
+
+  // Corner cases.
+  std::string String3 = R"(\
+    \n
+    \\n
+    \\)";
+  SmallString<128> String4;
+  String4 += String3.c_str();
+  std::string String5 = R"(a\
+
+
+    \\b)";
+  SmallString<128> String6;
+  String6 += String5.c_str();
+
+  String1 = Lexer::Stringify(StringRef(String1));
+  Lexer::Stringify(String2);
+  String3 = Lexer::Stringify(StringRef(String3));
+  Lexer::Stringify(String4);
+  String5 = Lexer::Stringify(StringRef(String5));
+  Lexer::Stringify(String6);
+
+  EXPECT_EQ(String1, R"(foo\n    {\"bar\":[]}\n    baz)");
+  EXPECT_EQ(String2, R"(foo\n    {\"bar\":[]}\n    baz)");
+  EXPECT_EQ(String3, R"(\\\n    \\n\n    \\\\n\n    \\\\)");
+  EXPECT_EQ(String4, R"(\\\n    \\n\n    \\\\n\n    \\\\)");
+  EXPECT_EQ(String5, R"(a\\\n\n\n    \\\\b)");
+  EXPECT_EQ(String6, R"(a\\\n\n\n    \\\\b)");
+}
+
 } // anonymous namespace