clang-format: Space and triple angle braces

[jpienaar]

Summary

clang-format changes "<<<" to "<< <" which is problematic for CUDA code. Change the behavior of clang-format to operate correctly with CUDA kernel launches without changing language options.

• No space required between "<<" and "<" in "<<<".
• No space required between identifier and "<<<". This avoids changing "f<<<" to "f <<<"
• No space required between ">" and "<<<". Avoids changing "f<t><<<" to "f<t> <<<".
• No space required between ">>>" and "("

Such that "f<param><<<1, 1>>>();" remains unchanged instead of being changed to "f<param> << <1, 1>>> ();"

[djasper]

We don't generally want to use the spacing in the input to matter for the output. Is this really important? What would be the downside of using language options?

Also, just controlling the space might not be sufficient as clang-format also might go ahead break between the tokens. Generally, it might be better to actually merge the tokens (as is done for the === operation in JavaScript), as these should be considered to be a single token. Alternatively, you need to at least control CanBreakBefore for these and add tests for BreakBeforeBinaryOperators set to All and None.

[jpienaar]

If CUDA language option is enabled then
A<A<A<int>>> a;
gets changed to
A < A < A < int >>> a;
as >>> at the end is no longer 3 tokens but 1. I could modify the
parseAngle (I believe) function to address this case though. The majority
of uses would be for regular C++, so enabling CUDA language option by
default would seem to benefit me but possibly causing others headaches, so
I went for a route where the opening triple angle bracket is retained as is
in as direct a manner as possible while affecting as few other code paths
as possible.

The importance is in the CUDA code generation side. foo<< <1,1>>>() would
not be recognized as a CUDAKernelCallExpr. So an input program that
compiled would not compile post formatting.

Would it be OK to merge the tokens in the following case?
friend std::ostream& operator << <T> (std::ostream&, const Self&);
That was pretty much the only other usage for three open angle brackets I
saw. Currently I think it would be merged, so that would not change but I
want to double check. If so, then I'll follow your suggestion and add those
tests thanks.

[jpienaar]

Version setting CUDA language option for comparison based on discussion earlier today.

The parseAngle function updated to treat >>> as 3 tokens. These are without the binary operator split tests as now both "<<<" and ">>>" are tokens. As the AST is not constructed, setting the CUDA language option should not have other side-effects here other than to recognize <<< and >>> as 1 token each.

[djasper]

lib/Format/TokenAnnotator.cpp

68

I really don't like such static variables. Also, now we have two such mechanisms (GreaterStashed and Count). Seems like the same mechanisms should be used.

69

In LLVM style, we don't allow one-liner ifs. Use clang-format ..

71–73

Do these lines actually change the behavior? Seems like the entire wrong place to do this.

1809

So, if we just set TT_TemplateOpener/Closer on these, is this really required?

unittests/Format/FormatTest.cpp

9616

nit: TripleAngleBrackets (one "p" and I don't think that these are "braces").

[jpienaar]

Removed the static variable and perform mergin in FormatTokenLexer. Also ran from clang-format -style llvm which changed some spacing.

[jpienaar]

The web ui is a bit confusing when I try to reply to comments on previous revision. So I'll reply here. Most of the previous comments probably don't apply since your pointer to GreaterStashed made me redo that part (thanks!). All the rules (except the last one about space between >>> and ( ) do effect the output, and the one about >>> and ( is probably fine. I didn't want to always merge greater, greater, greater into greatergreatergreater as then the template parsing logic needs to be modified, but if there is a lesslessless in the same instruction then that's a reasonably good signal that it is a kernel launch.

[jpienaar]

Redone following a simpler approach of splitting up << into < and <, and remerging only if not <<<. The formatting of kernel launch parameters then follows that of template instantiation, The previous approach could also have resulted in unmatched braces.

[djasper]

Could you generally upload diffs with the full file as context? That makes review here much easier.

lib/Format/Format.cpp

665

Wouldn't we want to fall back to merging in this case?

I think if the user formats the incomplete line "llvm::outs() <<", we should not rip apart the <<. Add a test for this.

674

Make this:

// Only merge if there currently is no whitespace between the two "<".

677

Indentation... Maybe use clang-format ;-).

679

I think this comment doesn't add information. Remove.

lib/Format/TokenAnnotator.cpp

2043

A few lines higher up, there is at test:

if (Left.is(tok::greater) && Right.is(tok::greater) &&
    Left.isNot(TT_TemplateCloser))
  return false;

Seems like we should combine those. Maybe simply remove the "Left.isNot(TT_TemplateCloser)"?

2044

Indentation.

[jpienaar]

Made the changes recommended. clang-format changed unrelated things in the file.

[djasper]

I think that this is basically good to go. Do you have commit access?

lib/Format/Format.cpp

665

Use early exit.

681

Maybe just use "auto" here.

[jpienaar]

Done. I've simplified around line 665 a bit too.

I have commit access and will commit in a couple of hours. Thanks.

lib/Format/Format.cpp

   FormatTokenLexer(SourceManager &SourceMgr, FileID ID, FormatStyle &Style,
                    encoding::Encoding Encoding)
       : FormatTok(nullptr), IsFirstToken(true), GreaterStashed(false),
-        Column(0), TrailingWhitespace(0), SourceMgr(SourceMgr), ID(ID),
-        Style(Style), IdentTable(getFormattingLangOpts(Style)),
-        Keywords(IdentTable), Encoding(Encoding), FirstInLineIndex(0),
-        FormattingDisabled(false) {
+        LessStashed(false), Column(0), TrailingWhitespace(0),
+        SourceMgr(SourceMgr), ID(ID), Style(Style),
+        IdentTable(getFormattingLangOpts(Style)), Keywords(IdentTable),
+        Encoding(Encoding), FirstInLineIndex(0), FormattingDisabled(false) {
     Lex.reset(new Lexer(ID, SourceMgr.getBuffer(ID), SourceMgr,
                         getFormattingLangOpts(Style)));
     Lex->SetKeepWhitespaceMode(true);

       return;
     if (tryMergeConflictMarkers())
       return;
+    if (tryMergeLessLess())
+      return;
 
     if (Style.Language == FormatStyle::LK_JavaScript) {
       if (tryMergeJSRegexLiteral())

     }
   }
 
+  bool tryMergeLessLess() {
+    // Merge X,less,less,Y into X,lessless,Y unless X or Y is less.
+    if (Tokens.size() < 4)
+        return false;
+    SmallVectorImpl<FormatToken *>::const_iterator First = Tokens.end() - 4;
+    if (First[3]->is(tok::less) ||
+        First[2]->isNot(tok::less) ||
+        First[1]->isNot(tok::less) ||
+        First[0]->is(tok::less))
+      return false;
+
+    // Check for whitespace between less and less
+    if (First[2]->WhitespaceRange.getBegin() !=
+        First[2]->WhitespaceRange.getEnd())
+        return false;
+
+    // Merge X,less,less,Y into X,lessless,Y
+    First[1]->Tok.setKind(tok::lessless);
+    First[1]->TokenText = "<<";
+    First[1]->ColumnWidth += 1;
+    Tokens.erase(Tokens.end() - 2);
+    return true;
+  }
+
   bool tryMergeTokens(ArrayRef<tok::TokenKind> Kinds) {
     if (Tokens.size() < Kinds.size())
       return false;

     return false;
   }
 
+  FormatToken *getStashedToken() {
+    // Create a synthesized second '>' or '<' token.
+    Token Tok = FormatTok->Tok;
+    StringRef TokenText = FormatTok->TokenText;
+
+    unsigned OriginalColumn = FormatTok->OriginalColumn;
+    FormatTok = new (Allocator.Allocate()) FormatToken;
+    FormatTok->Tok = Tok;
+    SourceLocation TokLocation =
+        FormatTok->Tok.getLocation().getLocWithOffset(1);
+    FormatTok->WhitespaceRange =
+        SourceRange(TokLocation, TokLocation);
+    FormatTok->TokenText = TokenText;
+    FormatTok->ColumnWidth = 1;
+    FormatTok->OriginalColumn = OriginalColumn;
+    return FormatTok;
+  }
+
   FormatToken *getNextToken() {
     if (GreaterStashed) {
-      // Create a synthesized second '>' token.
-      Token Greater = FormatTok->Tok;
-      unsigned OriginalColumn = FormatTok->OriginalColumn;
-      FormatTok = new (Allocator.Allocate()) FormatToken;
-      FormatTok->Tok = Greater;
-      SourceLocation GreaterLocation =
-          FormatTok->Tok.getLocation().getLocWithOffset(1);
-      FormatTok->WhitespaceRange =
-          SourceRange(GreaterLocation, GreaterLocation);
-      FormatTok->TokenText = ">";
-      FormatTok->ColumnWidth = 1;
-      FormatTok->OriginalColumn = OriginalColumn;
       GreaterStashed = false;
-      return FormatTok;
+      return getStashedToken();
+    }
+    if (LessStashed) {
+      LessStashed = false;
+      return getStashedToken();
     }
 
     FormatTok = new (Allocator.Allocate()) FormatToken;

       FormatTok->Tok.setKind(tok::greater);
       FormatTok->TokenText = FormatTok->TokenText.substr(0, 1);
       GreaterStashed = true;
+    } else if (FormatTok->Tok.is(tok::lessless)) {
+      FormatTok->Tok.setKind(tok::less);
+      FormatTok->TokenText = FormatTok->TokenText.substr(0, 1);
+      LessStashed = true;
     }
 
     // Now FormatTok is the next non-whitespace token.

 
   FormatToken *FormatTok;
   bool IsFirstToken;
-  bool GreaterStashed;
+  bool GreaterStashed, LessStashed;
   unsigned Column;
   unsigned TrailingWhitespace;
   std::unique_ptr<Lexer> Lex;

lib/Format/TokenAnnotator.cpp

     return true;
   if (Right.is(tok::kw_typename) && Left.isNot(tok::kw_const))
     return true;
+  if ((Left.is(TT_TemplateOpener) && Right.is(TT_TemplateOpener)) ||
+      (Left.is(TT_TemplateCloser) && Right.is(TT_TemplateCloser)))
+      return false;
   if (Left.isBinaryOperator() && !Left.isOneOf(tok::arrowstar, tok::lessless) &&
       Style.BreakBeforeBinaryOperators != FormatStyle::BOS_All &&
       (Style.BreakBeforeBinaryOperators == FormatStyle::BOS_None ||

unittests/Format/FormatTest.cpp

   verifyFormat("A<A<int>>();", Spaces);
 }
 
+TEST_F(FormatTest, TripleAngleBrackets) {
+  verifyFormat("f<<<1, 1>>>();");
+  verifyFormat("f<<<1, 1, 1, s>>>();");
+  verifyFormat("f<<<a, b, c, d>>>();");
+  EXPECT_EQ("f<<<1, 1>>>();",
+            format("f <<< 1, 1 >>> ();"));
+  verifyFormat("f<param><<<1, 1>>>();");
+  verifyFormat("f<1><<<1, 1>>>();");
+  EXPECT_EQ("f<param><<<1, 1>>>();",
+            format("f< param > <<< 1, 1 >>> ();"));
+  verifyFormat("aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+               "aaaaaaaaaaa<<<\n    1, 1>>>();");
+}
+
 TEST_F(FormatTest, HandleUnbalancedImplicitBracesAcrossPPBranches) {
   std::string code = "#if A\n"
                      "#if B\n"