[AST] Added a helper to extract a user-friendly text of a comment.

[ilya-biryukov]

Summary

The helper is used in clangd for documentation shown in code completion
and storing the docs in the symbols. See D45999.

This patch reuses the code of the Doxygen comment lexer, disabling the
bits that do command and html tag parsing.
The new helper works on all comments, including non-doxygen comments
and is faster. However, it does not understand or transform any
doxygen directives, i.e. cannot extract brief text, etc.

[ilya-biryukov]

Added forgotten bits of the change

[ioeric]

Overall looks good. Could you add tests for the new methods?

lib/AST/CommentLexer.cpp

294

micro-nit: I'd probably

return ParseCommands ? lexWithCommands(T) : lexWithoutCommands(T);

485

Can we share code with lexCommentTextWithCommands for these two common cases?

lib/AST/RawCommentList.cpp

353

nit: SkipWhitespaces for readability?

380

Explain when this would be invalid and why TokColumn = 0 is used?

383

nit: unsigned MaxSkip = IsFirstLine ? ... : ...;

392

I'd probably make SkipWs return the number of white spaces skipped and do the drop-front here, so that you could simplify the awkward calculation of IndentColumn below.

[ilya-biryukov]

In D46000#1077926, @ioeric wrote:

Overall looks good. Could you add tests for the new methods?

Sure. There are a few tests in D46002, but I haven't (yet) moved them to clang.

[ilya-biryukov]

• Attempt to reuse lexing code with/without command parsing.
• Get rid of SkipWs.

lib/AST/CommentLexer.cpp

485

I couldn't come up with a way to do that previsouly.
Made another attempt which seems to work.
Please take a look, the change is somewhat non-trivial (includes removing the loop that seems redundant)

lib/AST/RawCommentList.cpp

380

I don't know whether this can be even be invalid, but I'm not confident enough to add an assert there.
TokColumn = 0 seems like a reasonable way to recover if we can't compute the column number, i.e. assume the line starts at the first column if SourceLocation of the line was invalid for any reason.

This whole column thing looks weird to me, maybe I should just remove it altogether and just remove the same amount of whitespace in all the lines. WDYT?

383

That would force to get rid of the comments in the if branches, but they seem to be useful.
Am I missing an obvious style that would preserve the comments?

392

Got rid of it altogether.
The code seems is clearer now, thanks for the suggestion!

[ilya-biryukov]

• Update a comment after latest changes

[ilya-biryukov]

• Fix indentation

[ilya-biryukov]

lib/AST/RawCommentList.cpp

380

On a second thought, now I remember why I added this in the first place.
To support the following example we want to take column numbers into account:

class Foo {
    /* A block comment
       spanning multiple lines
       has too many spaces on the all lines except the first one.
    */
    int func();
};

[ioeric]

include/clang/AST/RawCommentList.h

118

I'm trying to understand how these cases and RawComment work.

For this case, are the // ... block and /* ... */ merged in one RawComment by default?

126

Are the *s in each lines automatically consumed by the lexer?

lib/AST/CommentLexer.cpp

301

I think we could avoid the "somewhat non-trivial" control flow by merging command and command-less cases in one function:

Something like:

const char *TokenPtr = ...;
auto HandleNonCommandToken = ...;
if (!ParseComand) {
   HandleNonCommandToken(...);
   return;
}
... 
switch (...) {
 // after all command cases 
default:
   HandleNonCommandToken(...);
 }

331–352

We should be extra careful about removing the loop... (It does seem to be redundant though)

lib/AST/RawCommentList.cpp

343

nit: We don't really know if there was a failure or the comment is simply empty. So I'd probably leave out the comment here to avoid confusion.

349

s/ParseCommentText/ParseCommands/?

352

This variable could use a comment.

380

So I think you would want to force MaxSkip to 0 if token loc is invalid to make sure no comment is accidentally eaten?

383

You could simply merge the comments, which doesn't seem to compromise readability.

393

I think MaxSkip could be removed with:

llvm::StringRef Trimmed = TokText.drop_front(IsFirstLine ? WhitespaceLen : 
                                                 std::max((int)IndentColumn - (int)TokColumn, 0)));

[ilya-biryukov]

include/clang/AST/RawCommentList.h

118

Yes, RawComment can represent multiple merged comments of different styles.

126

Yes, see comments::Lexer::skipLineStartingDecorations()

lib/AST/RawCommentList.cpp

343

getRawTextSlow returns empty string on error, so this comment has some ground.
Removed it anyway

349

Thanks for catching this!

380

Makes sense, thanks. Unfortunately I don't have any ideas on how we can test this case :-(

393

I've removed MaxSkip, but introduced SkipLen instead that captures an actual numbers of chars we want to skip.
I would still keep as a separate variable, as the expression seems somewhat complex and giving a name to it, arguably, makes the code more readable.

[ilya-biryukov]

• Remove tryLexCommands(), call into helper that parses commands directly
• Addressed other review comments

[ioeric]

Looks good. We still need tests though :)

lib/AST/RawCommentList.cpp

376

This is a bit confusing... Could you please add comments about the behavior here (as chatted offline)?

394

use static_cast instead of conversions.

405

I think it's end of file?

[ilya-biryukov]

• Move unit tests from clangd code to AST tests
• Assert locations are valid
• Address review other comments

[ilya-biryukov]

lib/AST/RawCommentList.cpp

376

After thinking about it for a while, decided to add an assert that location was valid instead.
Invalid locations don't make any sense there, since we won't be able to get the comment text in case of invalid locs.

394

Done. Rewrote the code to keep the reduce the number of casts too. It was unreadable with 3 static casts and default formatting.

[ilya-biryukov]

• Fixed infinite loop with comments that contain doxygen commands

[ioeric]

Thanks for adding the tests!

include/clang/AST/RawCommentList.h

138

I think we can get rid of the interface that takes ASTContext? If SourceManager and Diags are sufficient, I don't see why we would want another interface for ASTContext.

lib/AST/RawCommentList.cpp

352

I'm not quite sure about this. Could we just require a CommandTraits in the interface? And only make this assumption in tests?

unittests/AST/CommentTextTest.cpp

32 ↗

(On Diff #145691)

SourceManagerForFile added in D46176 should save you a few lines here. (I'm landing it right now...)

[ilya-biryukov]

• Simplify test code with SourceManagerForFile.

[ilya-biryukov]

include/clang/AST/RawCommentList.h

138

Two reasons that come to mind: it's simpler to use and follows the API of getBriefText. If not for mocking the tests, I would totally only keep the ASTContext-based one, since it does not really make any sense to create RawComment without ASTContext for any reason other than testing.

lib/AST/RawCommentList.cpp

352

I think we shouldn't add this to params, the whole point of this function is to do parsing that ignores the commands and the CommandTraits. The fact that lexer still needs them is because we haven't extracted a simpler interface from Lexer that doesn't rely on unused params, i.e. CommandTraits and Allocator.

unittests/AST/CommentTextTest.cpp

32 ↗

(On Diff #145691)

Thanks!

[ilya-biryukov]

• Removed the overload that accepts ASTContext

[ioeric]

lgtm

include/clang/AST/RawCommentList.h

138

As discussed offline, two interfaces that do the same thing are a bit confusing. I would still suggest favoring minimal API and test-ability over simplified usage - two parameters aren't that better than one after all :)

lib/AST/RawCommentList.cpp

352

Makes sense.

include/clang/AST/CommentLexer.h

private:
 
   /// Current lexing mode.
   LexerState State;
 
   /// If State is LS_VerbatimBlock, contains the name of verbatim end
   /// command, including command marker.
   SmallString<16> VerbatimBlockEndCommandName;
 
+  /// If true, the commands, html tags, etc will be parsed and reported as
+  /// separate tokens inside the comment body. If false, the comment text will
+  /// be parsed into text and newline tokens.
+  bool ParseCommands;
+
   /// Given a character reference name (e.g., "lt"), return the character that
   /// it stands for (e.g., "<").
   StringRef resolveHTMLNamedCharacterReference(StringRef Name) const;
 
   /// Given a Unicode codepoint as base-10 integer, return the character.
   StringRef resolveHTMLDecimalCharacterReference(StringRef Name) const;
 
   /// Given a Unicode codepoint as base-16 integer, return the character.
private:
 
   DiagnosticBuilder Diag(SourceLocation Loc, unsigned DiagID) {
     return Diags.Report(Loc, DiagID);
   }
 
   /// Eat string matching regexp \code \s*\* \endcode.
   void skipLineStartingDecorations();
 
-  /// Lex stuff inside comments.  CommentEnd should be set correctly.
+  /// Calls lexCommentText(With|Without)Commands, depending on value of
+  /// ParseCommands.
   void lexCommentText(Token &T);
 
-  void setupAndLexVerbatimBlock(Token &T,
-                                const char *TextBegin,
-                                char Marker, const CommandInfo *Info);
+  /// Try to lex commands, html tags and verbatim blocks inside comment text.
+  /// CommentEnd should be set correctly.
+  /// Returns true if command was lexed and no further processing is required.
+  /// Returns false if no special tokens were found and lexing of ordinary comment text need to be performed instead.
+  bool tryLexCommands(Token &T);
+
+  /// Lex only newlines and text inside comments. CommentEnd should be set
+  /// correctly.
+  void lexCommentTextWithoutCommands(Token &T);
+
+  void setupAndLexVerbatimBlock(Token &T, const char *TextBegin, char Marker,
+                                const CommandInfo *Info);
 
   void lexVerbatimBlockFirstLine(Token &T);
 
   void lexVerbatimBlockBody(Token &T);
 
   void setupAndLexVerbatimLine(Token &T, const char *TextBegin,
                                const CommandInfo *Info);
 
   void lexVerbatimLineText(Token &T);
 
   void lexHTMLCharacterReference(Token &T);
 
   void setupAndLexHTMLStartTag(Token &T);
 
   void lexHTMLStartTag(Token &T);
 
   void setupAndLexHTMLEndTag(Token &T);
 
   void lexHTMLEndTag(Token &T);
 
 public:
   Lexer(llvm::BumpPtrAllocator &Allocator, DiagnosticsEngine &Diags,
-        const CommandTraits &Traits,
-        SourceLocation FileLoc,
-        const char *BufferStart, const char *BufferEnd);
+        const CommandTraits &Traits, SourceLocation FileLoc,
+        const char *BufferStart, const char *BufferEnd,
+        bool ParseCommands = true);
 
   void lex(Token &T);
 
-  StringRef getSpelling(const Token &Tok,
-                        const SourceManager &SourceMgr,
+  StringRef getSpelling(const Token &Tok, const SourceManager &SourceMgr,
                         bool *Invalid = nullptr) const;
 };
 
 } // end namespace comments
 } // end namespace clang
 
 #endif
 

include/clang/AST/RawCommentList.h

public:
 
   const char *getBriefText(const ASTContext &Context) const {
     if (BriefTextValid)
       return BriefText;
 
     return extractBriefText(Context);
   }
 
+  /// Returns sanitized comment text, suitable for presentation in editor UIs.
+  /// E.g. will transform:
+  ///     // This is a long multiline comment.
+  ///     //   Parts of it  might be indented.
+  ///     /* The comments styles might be mixed. */
+  ///  into
+  ///     "This is a long multiline comment.\n"
+  ///     "  Parts of it  might be indented.\n"
+  ///     "The comments styles might be mixed."
+  /// Also removes leading indentation and sanitizes some common cases:
+  ///     /* This is a first line.
+  ///      *   This is a second line. It is indented.
+  ///      * This is a third line. */
+  /// and
+  ///     /* This is a first line.
+  ///          This is a second line. It is indented.
+  ///     This is a third line. */
+  /// will both turn into:
+  ///     "This is a first line.\n"
+  ///     "  This is a second line. It is indented.\n"
+  ///     "This is a third line."
+  std::string getFormattedText(const ASTContext &Context) const;
+
   /// Parse the comment, assuming it is attached to decl \c D.
   comments::FullComment *parse(const ASTContext &Context,
                                const Preprocessor *PP, const Decl *D) const;
 
 private:
   SourceRange Range;
 
   mutable StringRef RawText;
   mutable const char *BriefText;
 

lib/AST/CommentLexer.cpp

 #ifndef NDEBUG
   Result.TextPtr = "<UNSET>";
   Result.IntVal = 7;
 #endif
   BufferPtr = TokEnd;
 }
 
 void Lexer::lexCommentText(Token &T) {
+  if (ParseCommands) {
+    if (tryLexCommands(T))
+      return;
+  }
+  lexCommentTextWithoutCommands(T);
+}
+
+bool Lexer::tryLexCommands(Token &T) {
   assert(CommentState == LCS_InsideBCPLComment ||
          CommentState == LCS_InsideCComment);
+  if (!ParseCommands)
+    return false;
 
   switch (State) {
   case LS_Normal:
     break;
   case LS_VerbatimBlockFirstLine:
     lexVerbatimBlockFirstLine(T);
-    return;
+    return true;
   case LS_VerbatimBlockBody:
     lexVerbatimBlockBody(T);
-    return;
+    return true;
   case LS_VerbatimLineText:
     lexVerbatimLineText(T);
-    return;
+    return true;
   case LS_HTMLStartTag:
     lexHTMLStartTag(T);
-    return;
+    return true;
   case LS_HTMLEndTag:
     lexHTMLEndTag(T);
-    return;
+    return true;
   }
 
   assert(State == LS_Normal);
 
   const char *TokenPtr = BufferPtr;
   assert(TokenPtr < CommentEnd);
-  while (TokenPtr != CommentEnd) {
-    switch(*TokenPtr) {
-      case '\\':
-      case '@': {
-        // Commands that start with a backslash and commands that start with
-        // 'at' have equivalent semantics.  But we keep information about the
-        // exact syntax in AST for comments.
-        tok::TokenKind CommandKind =
-            (*TokenPtr == '@') ? tok::at_command : tok::backslash_command;
-        TokenPtr++;
-        if (TokenPtr == CommentEnd) {
-          formTextToken(T, TokenPtr);
-          return;
-        }
-        char C = *TokenPtr;
-        switch (C) {
-        default:
-          break;
+        return true;
 
-        case '\\': case '@': case '&': case '$':
-        case '#':  case '<': case '>': case '%':
-        case '\"': case '.': case ':':
-          // This is one of \\ \@ \& \$ etc escape sequences.
-          TokenPtr++;
-          if (C == ':' && TokenPtr != CommentEnd && *TokenPtr == ':') {
-            // This is the \:: escape sequence.
-            TokenPtr++;
-          }
-          StringRef UnescapedText(BufferPtr + 1, TokenPtr - (BufferPtr + 1));
-          formTokenWithChars(T, TokenPtr, tok::text);
-          T.setText(UnescapedText);
-          return;
-        }
+        return true;
 
-        // Don't make zero-length commands.
-        if (!isCommandNameStartCharacter(*TokenPtr)) {
-          formTextToken(T, TokenPtr);
-          return;
-        }
+        return true;
 
-        TokenPtr = skipCommandName(TokenPtr, CommentEnd);
-        unsigned Length = TokenPtr - (BufferPtr + 1);
 
-        // Hardcoded support for lexing LaTeX formula commands
-        // \f$ \f[ \f] \f{ \f} as a single command.
-        if (Length == 1 && TokenPtr[-1] == 'f' && TokenPtr != CommentEnd) {
-          C = *TokenPtr;
-          if (C == '$' || C == '[' || C == ']' || C == '{' || C == '}') {
-            TokenPtr++;
-            Length++;
-          }
-        }
 
-        StringRef CommandName(BufferPtr + 1, Length);
 
-        const CommandInfo *Info = Traits.getCommandInfoOrNULL(CommandName);
-        if (!Info) {
-          if ((Info = Traits.getTypoCorrectCommandInfo(CommandName))) {
-            StringRef CorrectedName = Info->Name;
-            SourceLocation Loc = getSourceLocation(BufferPtr);
-            SourceLocation EndLoc = getSourceLocation(TokenPtr);
-            SourceRange FullRange = SourceRange(Loc, EndLoc);
-            SourceRange CommandRange(Loc.getLocWithOffset(1), EndLoc);
-            Diag(Loc, diag::warn_correct_comment_command_name)
-              << FullRange << CommandName << CorrectedName
-              << FixItHint::CreateReplacement(CommandRange, CorrectedName);
-          } else {
-            formTokenWithChars(T, TokenPtr, tok::unknown_command);
-            T.setUnknownCommandName(CommandName);
-            Diag(T.getLocation(), diag::warn_unknown_comment_command_name)
-                << SourceRange(T.getLocation(), T.getEndLocation());
-            return;
-          }
-        }
-        if (Info->IsVerbatimBlockCommand) {
-          setupAndLexVerbatimBlock(T, TokenPtr, *BufferPtr, Info);
-          return;
-        }
-        if (Info->IsVerbatimLineCommand) {
-          setupAndLexVerbatimLine(T, TokenPtr, Info);
-          return;
-        }
-        formTokenWithChars(T, TokenPtr, CommandKind);
-        T.setCommandID(Info->getID());
-        return;
-      }
+          return true;
+        return true;
+        return true;
+      return true;
 
-      case '&':
-        lexHTMLCharacterReference(T);
-        return;
+      return true;
 
-      case '<': {
-        TokenPtr++;
-        if (TokenPtr == CommentEnd) {
-          formTextToken(T, TokenPtr);
-          return;
-        }
-        const char C = *TokenPtr;
-        if (isHTMLIdentifierStartingCharacter(C))
-          setupAndLexHTMLStartTag(T);
-        else if (C == '/')
-          setupAndLexHTMLEndTag(T);
-        else
-          formTextToken(T, TokenPtr);
-        return;
-      }
+        return true;
+      return true;
+    }
+
+    default:
+      // Signal lexing code for an ordinary comment should be run instead.
+      return false;
+}
+
+void Lexer::lexCommentTextWithoutCommands(Token &T) {
+  assert(CommentState == LCS_InsideBCPLComment ||
+         CommentState == LCS_InsideCComment);
+  assert(State == LS_Normal);
 
-      case '\n':
-      case '\r':
+  const char *TokenPtr = BufferPtr;
+  assert(TokenPtr < CommentEnd);
+  switch (*TokenPtr) {
         TokenPtr = skipNewline(TokenPtr, CommentEnd);
         formTokenWithChars(T, TokenPtr, tok::newline);
 
         if (CommentState == LCS_InsideCComment)
-          skipLineStartingDecorations();
         return;
 
-      default: {
-        size_t End = StringRef(TokenPtr, CommentEnd - TokenPtr).
-                         find_first_of("\n\r\\@&<");
+        size_t End =
+            StringRef(TokenPtr, CommentEnd - TokenPtr).find_first_of("\n\r\\@&<");
         if (End != StringRef::npos)
-          TokenPtr += End;
         else
-          TokenPtr = CommentEnd;
         formTextToken(T, TokenPtr);
         return;
-      }
-    }
-  }
-}
 
 void Lexer::setupAndLexVerbatimBlock(Token &T,
                                      const char *TextBegin,
                                      char Marker, const CommandInfo *Info) {
   assert(Info->IsVerbatimBlockCommand);
 
   VerbatimBlockEndCommandName.clear();
   VerbatimBlockEndCommandName.append(Marker == '\\' ? "\\" : "@");
   VerbatimBlockEndCommandName.append(Info->EndCommandName);
 
   formTokenWithChars(T, TextBegin, tok::verbatim_block_begin);
   T.setVerbatimBlockID(Info->getID());
 
   // If there is a newline following the verbatim opening command, skip the
   // newline so that we don't create an tok::verbatim_block_line with empty
   // text content.
   if (BufferPtr != CommentEnd &&
       isVerticalWhitespace(*BufferPtr)) {
     BufferPtr = skipNewline(BufferPtr, CommentEnd);
     State = LS_VerbatimBlockBody;
     return;
   }

 void Lexer::lexHTMLEndTag(Token &T) {
   assert(BufferPtr != CommentEnd && *BufferPtr == '>');
 
   formTokenWithChars(T, BufferPtr + 1, tok::html_greater);
   State = LS_Normal;
 }
 
 Lexer::Lexer(llvm::BumpPtrAllocator &Allocator, DiagnosticsEngine &Diags,
-             const CommandTraits &Traits,
-             SourceLocation FileLoc,
-             const char *BufferStart, const char *BufferEnd):
-    Allocator(Allocator), Diags(Diags), Traits(Traits),
-    BufferStart(BufferStart), BufferEnd(BufferEnd),
-    FileLoc(FileLoc), BufferPtr(BufferStart),
-    CommentState(LCS_BeforeComment), State(LS_Normal) {
-}
+             const CommandTraits &Traits, SourceLocation FileLoc,
+             const char *BufferStart, const char *BufferEnd,
+             bool ParseCommands)
+    : Allocator(Allocator), Diags(Diags), Traits(Traits),
+      BufferStart(BufferStart), BufferEnd(BufferEnd), FileLoc(FileLoc),
+      BufferPtr(BufferStart), CommentState(LCS_BeforeComment), State(LS_Normal),
+      ParseCommands(ParseCommands) {}
 
 void Lexer::lex(Token &T) {
 again:
   switch (CommentState) {
   case LCS_BeforeComment:
     if (BufferPtr == BufferEnd) {
       formTokenWithChars(T, BufferPtr, tok::eof);
       return;

lib/AST/RawCommentList.cpp

void RawCommentList::addDeserializedComments(ArrayRef<RawComment *> DeserializedComments) {
   MergedComments.reserve(Comments.size() + DeserializedComments.size());
 
   std::merge(Comments.begin(), Comments.end(),
              DeserializedComments.begin(), DeserializedComments.end(),
              std::back_inserter(MergedComments),
              BeforeThanCompare<RawComment>(SourceMgr));
   std::swap(Comments, MergedComments);
 }
+
+std::string RawComment::getFormattedText(const ASTContext &Ctx) const {
+  auto &SourceMgr = Ctx.getSourceManager();
+  llvm::StringRef CommentText = getRawText(SourceMgr);
+  if (CommentText.empty())
+    return ""; // we couldn't retreive the comment.
+
+  llvm::BumpPtrAllocator Allocator;
+  comments::Lexer L(Allocator, Ctx.getDiagnostics(),
+                    Ctx.getCommentCommandTraits(), getSourceRange().getBegin(),
+                    CommentText.begin(), CommentText.end(),
+                    /*ParseCommentText=*/false);
+
+  std::string Result;
+  unsigned IndentColumn = 0;
+
+  // Processes one line of the comment and adds it to the result.
+  // Handles skipping the indent at the start of the line.
+  // Returns false when eof is reached and true otherwise.
+  auto LexLine = [&](bool IsFirstLine) -> bool {
+    comments::Token Tok;
+    // Lex the first token on the line. We handle it separately, because we to
+    // fix up its indentation.
+    L.lex(Tok);
+    if (Tok.is(comments::tok::eof))
+      return false;
+    if (Tok.is(comments::tok::newline)) {
+      Result += "\n";
+      return true;
+    }
+    llvm::StringRef TokText = L.getSpelling(Tok, SourceMgr);
+    bool LocInvalid = false;
+    unsigned TokColumn =
+        SourceMgr.getSpellingColumnNumber(Tok.getLocation(), &LocInvalid);
+    if (LocInvalid)
+      TokColumn = 0;
+    // Compute the length of whitespace we're allowed to skip.
+    size_t MaxSkip;
+    if (IsFirstLine) {
+      // For the first line we skip all leading ws.
+      MaxSkip = std::numeric_limits<size_t>::max();
+    } else {
+      // For the rest, we skip up to the column of first non-ws symbol on the
+      // first line..
+      MaxSkip = std::max((int)IndentColumn - (int)TokColumn, 0);
+    }
+    // Amount of leading whitespace in TokText.
+    size_t WhitespaceLen = TokText.find_first_not_of(" \t");
+    if (WhitespaceLen == StringRef::npos)
+      WhitespaceLen = TokText.size();
+    // Remember the amount of whitespace we skipped in the first line to remove
+    // indent up to that column in the following lines.
+    if (IsFirstLine)
+      IndentColumn = TokColumn + WhitespaceLen;
+
+    llvm::StringRef Trimmed = TokText.drop_front(std::min(MaxSkip, WhitespaceLen));
+    Result += Trimmed;
+    // Lex all tokens in the rest of the line.
+    for (L.lex(Tok); Tok.isNot(comments::tok::eof); L.lex(Tok)) {
+      if (Tok.is(comments::tok::newline)) {
+        Result += "\n";
+        return true;
+      }
+      Result += L.getSpelling(Tok, SourceMgr);
+    }
+    // We've reached the end of the line.
+    return false;
+  };
+
+  auto DropTrailingNewLines = [](std::string &Str) {
+    while (Str.back() == '\n')
+      Str.pop_back();
+  };
+
+  // Proces first line separately to remember indent for the following lines.
+  if (!LexLine(/*IsFirstLine=*/true)) {
+    DropTrailingNewLines(Result);
+    return Result;
+  }
+  // Process the rest of the lines.
+  while (LexLine(/*IsFirstLine=*/false))
+    ;
+  DropTrailingNewLines(Result);
+  return Result;
+}