[clang-format] ColumnLimit check for trailing comments alignment acts wrong for multi-byte UTF-8 #47624

[HazardyKnusperkeks]

Summary

Fix for
https://github.com/llvm/llvm-project/issues/47624
https://bugs.llvm.org/show_bug.cgi?id=48280

Used token for counting length instead of logical symbols.
That leads to wrong length count with utf8 symbols

[HazardyKnusperkeks]

Hi,

could you please reupload your patch with the full diff context? And please add a regression test.

[StailGot]

In D124260#3468773, @HazardyKnusperkeks wrote:

Hi,

could you please reupload your patch with the full diff context? And please add a regression test.

Done!

[HazardyKnusperkeks]

There seem to be failing tests. Did you run the FormatTests?

clang/unittests/Format/FormatTestComments.cpp

2861–2864

Here is something wrong.

[HazardyKnusperkeks]

Do you still intend to work on this? Otherwise I will commandeer and abandon.

[HazardyKnusperkeks]

You can reopen this, if you wish to continue working on it.

clang/lib/Format/WhitespaceManager.cpp

for (unsigned i = 0, e = Changes.size(); i != e; ++i) {
     if (!Changes[i].IsTrailingComment)
       continue;
 
     unsigned ChangeMinColumn = Changes[i].StartOfTokenColumn;
     unsigned ChangeMaxColumn;
 
     if (Style.ColumnLimit == 0)
       ChangeMaxColumn = UINT_MAX;
-    else if (Style.ColumnLimit >= Changes[i].TokenLength)
-      ChangeMaxColumn = Style.ColumnLimit - Changes[i].TokenLength;
+    else if (Style.ColumnLimit >= Changes[i].Tok->ColumnWidth)
+      ChangeMaxColumn = Style.ColumnLimit - Changes[i].Tok->ColumnWidth;
     else
       ChangeMaxColumn = ChangeMinColumn;
 
     // If we don't create a replacement for this change, we have to consider
     // it to be immovable.
     if (!Changes[i].CreateReplacement)
       ChangeMaxColumn = ChangeMinColumn;
 

clang/unittests/Format/FormatTestComments.cpp

EXPECT_EQ(
              "} // namespace m\n",
              getLLVMStyleWithColumns(80)));
 
   // https://llvm.org/PR53441
   verifyFormat("/* */  //\n"
                "int a; //\n");
   verifyFormat("/**/   //\n"
                "int a; //\n");
+
+  // https://llvm.org/PR48280
+  EXPECT_EQ("int ab; // utf8 🐉\n"
+            "int a;  // line\n",
+            format("int ab; // long 🐉\n"
+                   "int a; // line\n",
+                   getLLVMStyleWithColumns(17)));
 }
 
 TEST_F(FormatTestComments, AlignsBlockCommentDecorations) {
   EXPECT_EQ("/*\n"
             " */",
             format("/*\n"
                    "*/",
                    getLLVMStyle()));