[Format] Fix crash when hitting eof while lexing JS template string

[sammccall]

Summary

Different loop termination conditions resulted in confusion of whether
*Offset was intended to be inside or outside the token.
This ultimately led to constructing an out-of-range SourceLocation.

Fix by making Offset consistently point *after* the token.

[kadircet]

thanks!

clang/lib/Format/FormatTokenLexer.cpp

767

it's a separate concern, but feels like this could also trigger some crashes (e.g. an input like "`\")

772

nit: Offset += 2; instead?

[sammccall]

clang/lib/Format/FormatTokenLexer.cpp

767

Good catch. I think you're right, but:

• I'm not sure
• that example doesn't crash under asan, and I didn't find a crashing one
• that pattern occurs a bunch more times in this file
• I really want to have an isolated minimal fix + test as this bug is causing an outage
• I don't really have spare time now to dive deeper into this

[thakis]

This breaks tests: http://45.33.8.238/linux/88341/step_7.txt

Please take a look and revert for now if it takes a while to fix.

[gulfem]

We are seeing the same error reported above:

FormatTests: clang/lib/Lex/Lexer.cpp:1151: SourceLocation clang::Lexer::getSourceLocation(const char *, unsigned int) const: Assertion `Loc >= BufferStart && Loc <= BufferEnd && "Location out of range for this buffer!"' failed.

https://luci-milo.appspot.com/ui/p/fuchsia/builders/toolchain.ci/clang-linux-x64/b8801040768684259873/overview

[sammccall]

Fixed by 4b53c00173163774d32125fbcae283a46a9a4b19 I think.

(@kadircet suggested a possible second crash, I couldn't get it to crash so included the testcase with this patch. Turns out it does crash...)

[gulfem]

In D135356#3840428, @sammccall wrote:

Fixed by 4b53c00173163774d32125fbcae283a46a9a4b19 I think.

It fixed the test error that we are seeing, thanks!

clang/lib/Format/FormatTokenLexer.cpp

void FormatTokenLexer::handleTemplateStrings() {
   }
 
   // 'Manually' lex ahead in the current file buffer.
   const char *Offset = Lex->getBufferLocation();
   const char *TmplBegin = Offset - BacktickToken->TokenText.size(); // at "`"
   for (; Offset != Lex->getBuffer().end(); ++Offset) {
     if (Offset[0] == '`') {
       StateStack.pop();
+      ++Offset;
       break;
     }
     if (Offset[0] == '\\') {
       ++Offset; // Skip the escaped character.
     } else if (Offset + 1 < Lex->getBuffer().end() && Offset[0] == '$' &&
                Offset[1] == '{') {
       // '${' introduces an expression interpolation in the template string.
       StateStack.push(LexerState::NORMAL);
       ++Offset;
+      ++Offset;
       break;
     }
   }
 
-  StringRef LiteralText(TmplBegin, Offset - TmplBegin + 1);
+  StringRef LiteralText(TmplBegin, Offset - TmplBegin);
   BacktickToken->setType(TT_TemplateString);
   BacktickToken->Tok.setKind(tok::string_literal);
   BacktickToken->TokenText = LiteralText;
 
   // Adjust width for potentially multiline string literals.
   size_t FirstBreak = LiteralText.find('\n');
   StringRef FirstLineText = FirstBreak == StringRef::npos
                                 ? LiteralText
                                 : LiteralText.substr(0, FirstBreak);
   BacktickToken->ColumnWidth = encoding::columnWidthWithTabs(
       FirstLineText, BacktickToken->OriginalColumn, Style.TabWidth, Encoding);
   size_t LastBreak = LiteralText.rfind('\n');
   if (LastBreak != StringRef::npos) {
     BacktickToken->IsMultiline = true;
     unsigned StartColumn = 0; // The template tail spans the entire line.
     BacktickToken->LastLineColumnWidth =
         encoding::columnWidthWithTabs(LiteralText.substr(LastBreak + 1),
                                       StartColumn, Style.TabWidth, Encoding);
   }
 
-  SourceLocation loc = Offset < Lex->getBuffer().end()
-                           ? Lex->getSourceLocation(Offset + 1)
-                           : SourceMgr.getLocForEndOfFile(ID);
+  SourceLocation loc = Lex->getSourceLocation(Offset);
   resetLexer(SourceMgr.getFileOffset(loc));
 }
 
 void FormatTokenLexer::tryParsePythonComment() {
   FormatToken *HashToken = Tokens.back();
   if (!HashToken->isOneOf(tok::hash, tok::hashhash))
     return;
   // Turn the remainder of this line into a comment.

clang/unittests/Format/FormatTestJS.cpp

 
 TEST_F(FormatTestJS, NestedTemplateStrings) {
   verifyFormat(
       "var x = `<ul>${xs.map(x => `<li>${x}</li>`).join('\\n')}</ul>`;");
   verifyFormat("var x = `he${({text: 'll'}.text)}o`;");
 
   // Crashed at some point.
   verifyFormat("}");
+  verifyFormat("`");
 }
 
 TEST_F(FormatTestJS, TaggedTemplateStrings) {
   verifyFormat("var x = html`<ul>`;");
   verifyFormat("yield `hello`;");
   verifyFormat("var f = {\n"
                "  param: longTagName`This is a ${\n"
                "                    'really'} long line`\n"