clang-format: [JS] Support shebang lines on the very first line.

[mprobst]

Summary

Shebang lines (#!/bin/blah) can be used in JavaScript scripts to indicate
they should be run using e.g. node. This change treats # lines on the first line
as line comments.

[djasper]

Thinking some more, I think this is actually very close to what we do for #include(-like) statements. That is done in TokenAnnotator::parseLine() and TokenAnnotator::parseIncludeDirective(). Could you move the logic there?

[mprobst]

How would I disable the formatting using that way?

When I special case parsePreprocessorDirective(), consume the line, and return LT_ImportStatement, clang-format still tries to format tokens within the shebang. E.g. I get #!/usr/bin / env node, note the whitespace after bin.

Turning the shebang into a comment seems like it has the advantage of being safe, its contents should simply never be touched.

[djasper]

That's the same for #include directives (with <>). Just turn the tokens into TT_ImplicitStringLiteral, same as is done for #includes. I am not saying it's better, but I don't think we should have to different approaches..

[mprobst]

• use #include style shebang parsing.

[• alexeagle]

lib/Format/FormatTokenLexer.cpp

23 ↗

(On Diff #58529)

revert this file

lib/Format/TokenAnnotator.cpp

698 ↗

(On Diff #58529)

should we still restrict it to be on the first line?

I suppose if you start some other line with #, that's not valid JS or TS so it doesn't matter what clang-format does to it? Maybe it's fine like this.

[mprobst]

lib/Format/TokenAnnotator.cpp

698 ↗

(On Diff #58529)

Done. Doesn't make a big difference I guess, but it's better to be conservative.

[mprobst]

revert FormatTokenLexer, restrict to first token

[• alexeagle]

ping @djasper we need this fix in order to format angular 2 repo again

[mprobst]

Alex, do you accept this revision?

[djasper]

Looks good.

cfe/trunk/lib/Format/FormatToken.h

struct FormatToken {
 
   /// \brief Contains the width in columns of the last line of a multi-line
   /// token.
   unsigned LastLineColumnWidth = 0;
 
   /// \brief Whether the token text contains newlines (escaped or not).
   bool IsMultiline = false;
 
-  /// \brief Indicates that this is the first token.
+  /// \brief Indicates that this is the first token of the file.
   bool IsFirst = false;
 
   /// \brief Whether there must be a line break before this token.
   ///
   /// This happens for example when a preprocessor directive ended directly
   /// before the token.
   bool MustBreakBefore = false;
 

cfe/trunk/lib/Format/TokenAnnotator.cpp

if (CurrentToken &&
         if (IsMark || CurrentToken->Previous->is(TT_BinaryOperator))
           CurrentToken->Type = TT_ImplicitStringLiteral;
         next();
       }
     }
   }
 
   LineType parsePreprocessorDirective() {
+    bool IsFirstToken = CurrentToken->IsFirst;
     LineType Type = LT_PreprocessorDirective;
     next();
     if (!CurrentToken)
       return Type;
+
+    if (Style.Language == FormatStyle::LK_JavaScript && IsFirstToken) {
+      // JavaScript files can contain shebang lines of the form:
+      // #!/usr/bin/env node
+      // Treat these like C++ #include directives.
+      while (CurrentToken) {
+        // Tokens cannot be comments here.
+        CurrentToken->Type = TT_ImplicitStringLiteral;
+        next();
+      }
+      return LT_ImportStatement;
+    }
+
     if (CurrentToken->Tok.is(tok::numeric_constant)) {
       CurrentToken->SpacesRequiredBefore = 1;
       return Type;
     }
     // Hashes in the middle of a line can lead to any strange token
     // sequence.
     if (!CurrentToken->Tok.getIdentifierInfo())
       return Type;

cfe/trunk/unittests/Format/FormatTestJS.cpp

 
 TEST_F(FormatTestJS, RequoteStringsLeave) {
   FormatStyle LeaveQuotes = getGoogleStyle(FormatStyle::LK_JavaScript);
   LeaveQuotes.JavaScriptQuotes = FormatStyle::JSQS_Leave;
   verifyFormat("var x = \"foo\";", LeaveQuotes);
   verifyFormat("var x = 'foo';", LeaveQuotes);
 }
 
+TEST_F(FormatTestJS, SupportShebangLines) {
+  verifyFormat("#!/usr/bin/env node\n"
+               "var x = hello();",
+               "#!/usr/bin/env node\n"
+               "var x   =  hello();");
+}
+
 } // end namespace tooling
 } // end namespace clang