[clang-format] Do not align assignments that aren't after the same number of commas. (Closes: 25329)

[berenm]

Summary

This fixes https://llvm.org/bugs/show_bug.cgi?id=25329, as well as misalignments like the following:

int a, b = 2;
int c    = 3;

[berenm]

[clang-format] Count the number of braces and parens on the line instead of remembering only one.

[berenm]

I've also added a fix for the other issue reported on the same bug. I could split the two reviews if necessary.

[djasper]

lib/Format/WhitespaceManager.cpp

163–171

Hm. So many lokal variables. I think it might make sense to wrap these in a class/struct? That could be a first step to re-using code in alignConsecutiveDeclarations.

Also, a comment would help here. Specifically, a comment explaining why you count commas, braces and parentheses (not explaining that this variables count them ;-) )

220

Use (here and everywhere else):

--LeftBracesOnLine;

[berenm]

[clang-format] Alignment code factorization.

[berenm]

[clang-format] Remove deleted methods declaration.

[djasper]

I like it :-)

lib/Format/WhitespaceManager.cpp

162

Why is this a class/struct? It doesn't seem to have any state.

[berenm]

[clang-format] Replace TokenSequenceAligner with simple static functions.

[berenm]

lib/Format/WhitespaceManager.cpp

150–267

Haha, good question. I initially moved the variable there, but then it didn't looked really better, so I moved the variables back into the function...

[berenm]

Ping?

[berenm]

[clang-format] alignConsecutiveXXX: replace the buggy paren / braces counter with a better scope tracker.

[berenm]

Updated the diff with a fix for the issue reported in http://lists.llvm.org/pipermail/cfe-dev/2015-November/046057.html

[djasper]

Sorry for being so slow to respond. Feel free to ping me more frequently ;-).

lib/Format/WhitespaceManager.cpp

315

Maybe be slightly more specific about what constitutes a "scope".

Also, FormatTokens already have this information stored in NestingLevel. Would it maybe make sense to copy that information into each Change? In the long run, we might actually wanna switch to storing a back-reference to the actual token in a change to have access to all the information. But that would probably better be done in a subsequent change.

326

FoundOnLine seems to be missing some crucial information. Would you might changing to FoundMathcingTokenOnLine or FoundMatchOnLine?

359

I think this would be slightly easier to read if you used "continue" (basically instead of each "else" here)
http://llvm.org/docs/CodingStandards.html#use-early-exits-and-continue-to-simplify-code

unittests/Format/FormatTest.cpp

8657

I am not sure that this is actually desirable, but then again, I also don't think it matters. If people don't like it, they can add a blank line or a comment in-between.

[berenm]

[clang-format] Code cleanup and variable naming in WhitespaceManager::AlignTokens

[berenm]

lib/Format/WhitespaceManager.cpp

315

I have added some details in the comments, but couldn't manage to use the FormatToken::NestingLevel member. I haven't figured out why exactly, but it doesn't increase the nesting level counter in the braced scope of examples like:

int l = []() {
  int i = 0;
  int h = 0;
}

i.e.: tokens i and h have NestingLevel == 0 as it is the case for token l.

I will try to figure out why exactly but it will require a bit more time.

359–360

I did that.

It makes me think that the code actually forbids the generic AlignToken function to be used with a matcher for aligning commas or braces, paren and brackets...

[djasper]

lib/Format/WhitespaceManager.cpp

315

Ah, right. It's fine to do this later. The reason is that the lines in the nested block are handled separately and scope counting starts from the beginning. You could probably use the line level somehow, but that's probably insufficient as it doesn't count the scopes in the outer statement.

359

Use:

if (!Matches(Changes[i]))
  continue;

359–360

I don't understand what you mean.

[berenm]

Fix continue statements.

[berenm]

lib/Format/WhitespaceManager.cpp

335–373

The code was only checking for a match if none of the previous conditions were true. I changed that and with your suggestion so I guess it's OK now.

[djasper]

Looks awesome :-)

lib/Format/WhitespaceManager.cpp

335–373

Ah, I see, I guess the initial if-else chains were too complicated for me ;-).

This looks good now, thanks.

[berenm]

I don't have commit access, if you don't mind landing it for me :)

[djasper]

Submitted as r254406.

Note that I had to make the struct "Change" public. How did this compile on your end?

[berenm]

I don't have any compilation error with GCC 5.2.1. I tried again with Clang 3.8 and indeed there is this error:

tools/clang/lib/Format/WhitespaceManager.cpp:155:51: error: 'Change' is a private member of 'clang::format::WhitespaceManager'
                   SmallVector<WhitespaceManager::Change, 16> &Changes) {
                                                  ^
tools/clang/lib/Format/WhitespaceManager.h:87:10: note: declared private here
  struct Change {
         ^

This can easily be reproduced: http://goo.gl/CUk665 vs http://goo.gl/6KcBFT

I don't know precisely which one is correct here, it looks like to be related to the access permission in the context of a template function instantiation.

lib/Format/WhitespaceManager.cpp

if (Change.Kind == tok::comment) {
         Change.IndentationOffset =
             Change.StartOfTokenColumn -
             Change.StartOfBlockComment->StartOfTokenColumn;
     } else {
       LastBlockComment = nullptr;
     }
   }
 }
 
 // Walk through all of the changes and find sequences of "=" to align.  To do
 // so, keep track of the lines and whether or not an "=" was found on align. If
 // a "=" is found on a line, extend the current sequence. If the current line
 // cannot be part of a sequence, e.g. because there is an empty line before it
 // or it contains non-assignments, finalize the previous sequence.
 //
 // FIXME: The code between assignment and declaration alignment is mostly
 // duplicated and would benefit from factorization.
 void WhitespaceManager::alignConsecutiveAssignments() {
   if (!Style.AlignConsecutiveAssignments)
     return;
 
   unsigned MinColumn = 0;
   unsigned MaxColumn = UINT_MAX;
   unsigned StartOfSequence = 0;
   unsigned EndOfSequence = 0;
   bool FoundAssignmentOnLine = false;
-  bool FoundLeftBraceOnLine = false;
-  bool FoundLeftParenOnLine = false;
+  unsigned LeftBracesOnLine = 0;
+  unsigned LeftParensOnLine = 0;
+  unsigned CommasOnPrevLine = 0;
+  unsigned CommasOnLine = 0;
 
   // Aligns a sequence of assignment tokens, on the MinColumn column.
   //
   // Sequences start from the first assignment token to align, and end at the
   // first token of the first line that doesn't need to be aligned.
   //
   // We need to adjust the StartOfTokenColumn of each Change that is on a line
   // containing any assignment to be aligned and located after such assignment
   auto AlignSequence = [&] {
     if (StartOfSequence > 0 && StartOfSequence < EndOfSequence)
       alignConsecutiveAssignments(StartOfSequence, EndOfSequence, MinColumn);
     MinColumn = 0;
     MaxColumn = UINT_MAX;
     StartOfSequence = 0;
     EndOfSequence = 0;
   };
 
   for (unsigned i = 0, e = Changes.size(); i != e; ++i) {
     if (Changes[i].NewlinesBefore != 0) {
+      CommasOnPrevLine = CommasOnLine;
+      CommasOnLine = 0;
       EndOfSequence = i;
       // If there is a blank line, if the last line didn't contain any
       // assignment, or if we found an open brace or paren, the sequence ends
       // here.
       if (Changes[i].NewlinesBefore > 1 || !FoundAssignmentOnLine ||
-          FoundLeftBraceOnLine || FoundLeftParenOnLine) {
+          LeftBracesOnLine > 0 || LeftParensOnLine > 0) {
         // NB: In the latter case, the sequence should end at the beggining of
         // the previous line, but it doesn't really matter as there is no
         // assignment on it
         AlignSequence();
       }
 
       FoundAssignmentOnLine = false;
-      FoundLeftBraceOnLine = false;
-      FoundLeftParenOnLine = false;
+      LeftBracesOnLine = 0;
+      LeftParensOnLine = 0;
     }
 
     // If there is more than one "=" per line, or if the "=" appears first on
     // the line of if it appears last, end the sequence
     if (Changes[i].Kind == tok::equal &&
         (FoundAssignmentOnLine || Changes[i].NewlinesBefore > 0 ||
          Changes[i + 1].NewlinesBefore > 0)) {
       AlignSequence();
     } else if (Changes[i].Kind == tok::r_brace) {
-      if (!FoundLeftBraceOnLine)
+      if (LeftBracesOnLine == 0)
         AlignSequence();
-      FoundLeftBraceOnLine = false;
+      else
+        LeftBracesOnLine--;
     } else if (Changes[i].Kind == tok::l_brace) {
-      FoundLeftBraceOnLine = true;
+      LeftBracesOnLine++;
       if (!FoundAssignmentOnLine)
         AlignSequence();
     } else if (Changes[i].Kind == tok::r_paren) {
-      if (!FoundLeftParenOnLine)
+      if (LeftParensOnLine == 0)
         AlignSequence();
-      FoundLeftParenOnLine = false;
+      else
+        LeftParensOnLine--;
     } else if (Changes[i].Kind == tok::l_paren) {
-      FoundLeftParenOnLine = true;
+      LeftParensOnLine++;
       if (!FoundAssignmentOnLine)
         AlignSequence();
-    } else if (!FoundAssignmentOnLine && !FoundLeftBraceOnLine &&
-               !FoundLeftParenOnLine && Changes[i].Kind == tok::equal) {
+    } else if (Changes[i].Kind == tok::comma) {
+      if (!FoundAssignmentOnLine)
+        CommasOnLine++;
+    } else if (!FoundAssignmentOnLine && LeftBracesOnLine == 0 &&
+               LeftParensOnLine == 0 && Changes[i].Kind == tok::equal) {
       FoundAssignmentOnLine = true;
       if (StartOfSequence == 0)
         StartOfSequence = i;
 
       unsigned ChangeMinColumn = Changes[i].StartOfTokenColumn;
       int LineLengthAfter = -Changes[i].Spaces;
       for (unsigned j = i; j != e && Changes[j].NewlinesBefore == 0; ++j)
         LineLengthAfter += Changes[j].Spaces + Changes[j].TokenLength;
       unsigned ChangeMaxColumn = Style.ColumnLimit - LineLengthAfter;
 
-      if (ChangeMinColumn > MaxColumn || ChangeMaxColumn < MinColumn) {
+      if (ChangeMinColumn > MaxColumn || ChangeMaxColumn < MinColumn ||
+          CommasOnPrevLine != CommasOnLine) {
         AlignSequence();
         StartOfSequence = i;
       }
 
       MinColumn = std::max(MinColumn, ChangeMinColumn);
       MaxColumn = std::min(MaxColumn, ChangeMaxColumn);
     }
   }
 
   EndOfSequence = Changes.size();
   AlignSequence();
 }
 
 void WhitespaceManager::alignConsecutiveAssignments(unsigned Start,
                                                     unsigned End,
                                                     unsigned Column) {
   bool FoundAssignmentOnLine = false;
   int Shift = 0;
   for (unsigned i = Start; i != End; ++i) {
     if (Changes[i].NewlinesBefore > 0) {
       FoundAssignmentOnLine = false;
       Shift = 0;
     }
 
     // If this is the first assignment to be aligned, remember by how many
void WhitespaceManager::alignConsecutiveDeclarations() {
   if (!Style.AlignConsecutiveDeclarations)
     return;
 
   unsigned MinColumn = 0;
   unsigned MaxColumn = UINT_MAX;
   unsigned StartOfSequence = 0;
   unsigned EndOfSequence = 0;
   bool FoundDeclarationOnLine = false;
-  bool FoundLeftBraceOnLine = false;
-  bool FoundLeftParenOnLine = false;
+  unsigned LeftBracesOnLine = 0;
+  unsigned LeftParensOnLine = 0;
+  unsigned CommasOnPrevLine = 0;
+  unsigned CommasOnLine = 0;
 
   auto AlignSequence = [&] {
     if (StartOfSequence > 0 && StartOfSequence < EndOfSequence)
       alignConsecutiveDeclarations(StartOfSequence, EndOfSequence, MinColumn);
     MinColumn = 0;
     MaxColumn = UINT_MAX;
     StartOfSequence = 0;
     EndOfSequence = 0;
   };
 
   for (unsigned i = 0, e = Changes.size(); i != e; ++i) {
     if (Changes[i].NewlinesBefore != 0) {
+      CommasOnPrevLine = CommasOnLine;
+      CommasOnLine = 0;
       EndOfSequence = i;
       if (Changes[i].NewlinesBefore > 1 || !FoundDeclarationOnLine ||
-          FoundLeftBraceOnLine || FoundLeftParenOnLine)
+          LeftBracesOnLine > 0 || LeftParensOnLine > 0)
         AlignSequence();
       FoundDeclarationOnLine = false;
-      FoundLeftBraceOnLine = false;
-      FoundLeftParenOnLine = false;
+      LeftBracesOnLine = 0;
+      LeftParensOnLine = 0;
     }
 
     if (Changes[i].Kind == tok::r_brace) {
-      if (!FoundLeftBraceOnLine)
+      if (!LeftBracesOnLine)
         AlignSequence();
-      FoundLeftBraceOnLine = false;
+      else
+        LeftBracesOnLine--;
     } else if (Changes[i].Kind == tok::l_brace) {
-      FoundLeftBraceOnLine = true;
+      LeftBracesOnLine++;
       if (!FoundDeclarationOnLine)
         AlignSequence();
     } else if (Changes[i].Kind == tok::r_paren) {
-      if (!FoundLeftParenOnLine)
+      if (!LeftParensOnLine)
         AlignSequence();
-      FoundLeftParenOnLine = false;
+      else
+        LeftParensOnLine--;
     } else if (Changes[i].Kind == tok::l_paren) {
-      FoundLeftParenOnLine = true;
+      LeftParensOnLine++;
       if (!FoundDeclarationOnLine)
         AlignSequence();
-    } else if (!FoundDeclarationOnLine && !FoundLeftBraceOnLine &&
-               !FoundLeftParenOnLine && Changes[i].IsStartOfDeclName) {
+    } else if (Changes[i].Kind == tok::comma) {
+      if (!FoundDeclarationOnLine)
+        CommasOnLine++;
+    } else if (!FoundDeclarationOnLine && LeftBracesOnLine == 0 &&
+               LeftParensOnLine == 0 && Changes[i].IsStartOfDeclName) {
       FoundDeclarationOnLine = true;
       if (StartOfSequence == 0)
         StartOfSequence = i;
 
       unsigned ChangeMinColumn = Changes[i].StartOfTokenColumn;
       int LineLengthAfter = -Changes[i].Spaces;
       for (unsigned j = i; j != e && Changes[j].NewlinesBefore == 0; ++j)
         LineLengthAfter += Changes[j].Spaces + Changes[j].TokenLength;
       unsigned ChangeMaxColumn = Style.ColumnLimit - LineLengthAfter;
 
-      if (ChangeMinColumn > MaxColumn || ChangeMaxColumn < MinColumn) {
+      if (ChangeMinColumn > MaxColumn || ChangeMaxColumn < MinColumn ||
+          CommasOnPrevLine != CommasOnLine) {
         AlignSequence();
         StartOfSequence = i;
       }
 
       MinColumn = std::max(MinColumn, ChangeMinColumn);
       MaxColumn = std::min(MaxColumn, ChangeMaxColumn);
     }
   }
 
   EndOfSequence = Changes.size();
   AlignSequence();

unittests/Format/FormatTest.cpp

TEST_F(FormatTest, AlignConsecutiveAssignments) {
   verifyFormat("void SomeFunction(int parameter = 0) {\n"
                "  int i   = 1;\n"
                "  int j   = 2;\n"
                "  int big = 10000;\n"
                "}",
                Alignment);
   verifyFormat("class C {\n"
                "public:\n"
                "  int i = 1;\n"
                "  virtual void f() = 0;\n"
                "};",
                Alignment);
   verifyFormat("int i = 1;\n"
                "if (SomeType t = getSomething()) {\n"
                "}\n"
                "int j   = 2;\n"
                "int big = 10000;",
TEST_F(FormatTest, AlignConsecutiveAssignments) {
 
   // FIXME: Should align all three assignments
   verifyFormat(
       "int i      = 1;\n"
       "SomeType a = SomeFunction(looooooooooooooooooooooongParameterA,\n"
       "                          loooooooooooooooooooooongParameterB);\n"
       "int j = 2;",
       Alignment);
+
+  verifyFormat("template <typename T, typename T_0 = very_long_type_name_0,\n"
+               "          typename B   = very_long_type_name_1,\n"
+               "          typename T_2 = very_long_type_name_2>\n"
+               "auto foo() {}\n",
+               Alignment);
+  verifyFormat("int a, b = 1;\n"
+               "int c  = 2;\n"
+               "int dd = 3;\n",
+               Alignment);
+  verifyFormat("int aa       = ((1 > 2) ? 3 : 4);\n"
+               "float b[1][] = {{3.f}};\n",
+               Alignment);
 }
 
 TEST_F(FormatTest, AlignConsecutiveDeclarations) {
   FormatStyle Alignment = getLLVMStyle();
   Alignment.AlignConsecutiveDeclarations = false;
   verifyFormat("float const a = 5;\n"
                "int oneTwoThree = 123;",
                Alignment);
TEST_F(FormatTest, AlignConsecutiveDeclarations) {
   verifyFormat("float    ii              = 1;\n"
                "unsigned j               = 2;\n"
                "int someVerylongVariable = 1;\n"
                "AnotherLongType  ll = 123456;\n"
                "VeryVeryLongType k  = 2;\n"
                "int              myvar = 1;",
                Alignment);
   Alignment.ColumnLimit = 80;
+  Alignment.AlignConsecutiveAssignments = false;
+
+  verifyFormat(
+      "template <typename LongTemplate, typename VeryLongTemplateTypeName,\n"
+      "          typename LongType, typename B>\n"
+      "auto foo() {}\n",
+      Alignment);
+  verifyFormat("float a, b = 1;\n"
+               "int   c = 2;\n"
+               "int   dd = 3;\n",
+               Alignment);
+  verifyFormat("int   aa = ((1 > 2) ? 3 : 4);\n"
+               "float b[1][] = {{3.f}};\n",
+               Alignment);
+  Alignment.AlignConsecutiveAssignments = true;
+  verifyFormat("float a, b = 1;\n"
+               "int   c  = 2;\n"
+               "int   dd = 3;\n",
+               Alignment);
+  verifyFormat("int   aa     = ((1 > 2) ? 3 : 4);\n"
+               "float b[1][] = {{3.f}};\n",
+               Alignment);
+  Alignment.AlignConsecutiveAssignments = false;
 }
 
 TEST_F(FormatTest, LinuxBraceBreaking) {
   FormatStyle LinuxBraceStyle = getLLVMStyle();
   LinuxBraceStyle.BreakBeforeBraces = FormatStyle::BS_Linux;
   verifyFormat("namespace a\n"
                "{\n"
                "class A\n"