[clang-format] Non-latin comment prefix whitespace

[ksyx]

Summary

This commit changes the condition of requiring comment to start with
alphabet characters to make no change only for a certain set of
characters, currently horizontal whitespace and punctuation characters.

[ksyx]

Related discussion: https://reviews.llvm.org/D92257#3252060

Proposed by @byronhe

cc @CoelacanthusHex

[krasimir]

It appears that this caused a regression by adding an additional space of indentation to line comments in some cases:

% cat test.cc  # (before)
// Comment
int i;
% clang-format -style=google test.cc
//  Comment
int i;

@ksyx could you please take a look?

[ksyx]

In D118869#3359439, @krasimir wrote:

It appears that this caused a regression by adding an additional space of indentation to line comments in some cases:

% cat test.cc  # (before)
// Comment
int i;
% clang-format -style=google test.cc
//  Comment
int i;

@ksyx could you please take a look?

I did not reproduce it in latest main branch. This problem may in relate to this issue which have recently been fixed: https://github.com/llvm/llvm-project/issues/53844

[krasimir]

Thank you!
Turns out the original source wasn't using an ascii space for the comment indentation but a unicode [c2 a0] no-break space:

2f 2f c2 a0 43 6f 6d 6d  65 6e 74 0a 69 6e 74 20  |//..Comment.int |

IMO we shouldn't aim to handle non-ascii spaces for indentation in clang-format; the new behavior on this example is fine.

clang/lib/Format/BreakableToken.cpp

for (const FormatToken *CurrentTok = &Tok;
     Prefix.resize(Lines.size());
     OriginalPrefix.resize(Lines.size());
     for (size_t i = FirstLineIndex, e = Lines.size(); i < e; ++i) {
       Lines[i] = Lines[i].ltrim(Blanks);
       StringRef IndentPrefix = getLineCommentIndentPrefix(Lines[i], Style);
       OriginalPrefix[i] = IndentPrefix;
       const unsigned SpacesInPrefix = llvm::count(IndentPrefix, ' ');
 
+      // This lambda also considers multibyte character that is not handled in
+      // functions like isPunctuation provided by CharInfo.
+      const auto NoSpaceBeforeFirstCommentChar = [&]() {
+        assert(Lines[i].size() > IndentPrefix.size());
+        const char FirstCommentChar = Lines[i][IndentPrefix.size()];
+        const unsigned FirstCharByteSize =
+            encoding::getCodePointNumBytes(FirstCommentChar, Encoding);
+        return encoding::columnWidth(
+                   Lines[i].substr(IndentPrefix.size(), FirstCharByteSize),
+                   Encoding) == 1 &&
+               (FirstCommentChar == '\\' || isPunctuation(FirstCommentChar) ||
+                isHorizontalWhitespace(FirstCommentChar));
+      };
+
       // On the first line of the comment section we calculate how many spaces
       // are to be added or removed, all lines after that just get only the
       // change and we will not look at the maximum anymore. Additionally to the
       // actual first line, we calculate that when the non space Prefix changes,
       // e.g. from "///" to "//".
       if (i == 0 || OriginalPrefix[i].rtrim(Blanks) !=
                         OriginalPrefix[i - 1].rtrim(Blanks)) {
         if (SpacesInPrefix < Style.SpacesInLineCommentPrefix.Minimum &&
             Lines[i].size() > IndentPrefix.size() &&
-            isAlphanumeric(Lines[i][IndentPrefix.size()])) {
+            !NoSpaceBeforeFirstCommentChar()) {
           FirstLineSpaceChange =
               Style.SpacesInLineCommentPrefix.Minimum - SpacesInPrefix;
         } else if (SpacesInPrefix > Style.SpacesInLineCommentPrefix.Maximum) {
           FirstLineSpaceChange =
               Style.SpacesInLineCommentPrefix.Maximum - SpacesInPrefix;
         } else {
           FirstLineSpaceChange = 0;
         }
       }
 
       if (Lines[i].size() != IndentPrefix.size()) {
         PrefixSpaceChange[i] = FirstLineSpaceChange;
 
         if (SpacesInPrefix + PrefixSpaceChange[i] <
             Style.SpacesInLineCommentPrefix.Minimum) {
           PrefixSpaceChange[i] += Style.SpacesInLineCommentPrefix.Minimum -
                                   (SpacesInPrefix + PrefixSpaceChange[i]);
         }
 
         assert(Lines[i].size() > IndentPrefix.size());
         const auto FirstNonSpace = Lines[i][IndentPrefix.size()];
         const auto AllowsSpaceChange =
             SpacesInPrefix != 0 ||
-            (isAlphanumeric(FirstNonSpace) ||
+            (!NoSpaceBeforeFirstCommentChar() ||
              (FirstNonSpace == '}' && FirstLineSpaceChange != 0));
 
         if (PrefixSpaceChange[i] > 0 && AllowsSpaceChange) {
           Prefix[i] = IndentPrefix.str();
           Prefix[i].append(PrefixSpaceChange[i], ' ');
         } else if (PrefixSpaceChange[i] < 0 && AllowsSpaceChange) {
           Prefix[i] = IndentPrefix
                           .drop_back(std::min<std::size_t>(

clang/unittests/Format/FormatTestComments.cpp

StringRef Code =
       "//Free comment without space\n"
       "\n"
       "//   Free comment with 3 spaces\n"
       "\n"
       "///Free Doxygen without space\n"
       "\n"
       "///   Free Doxygen with 3 spaces\n"
       "\n"
+      "//🐉 A nice dragon\n"
+      "\n"
+      "//\t abccba\n"
+      "\n"
+      "//\\t deffed\n"
+      "\n"
+      "//   🐉 Another nice dragon\n"
+      "\n"
+      "//   \t Three leading spaces following tab\n"
+      "\n"
+      "//   \\t Three leading spaces following backslash\n"
+      "\n"
       "/// A Doxygen Comment with a nested list:\n"
       "/// - Foo\n"
       "/// - Bar\n"
       "///   - Baz\n"
       "///   - End\n"
       "///     of the inner list\n"
       "///   .\n"
       "/// .\n"
TEST_F(FormatTestComments, SpaceAtLineCommentBegin) {
   EXPECT_EQ("// Free comment without space\n"
             "\n"
             "//   Free comment with 3 spaces\n"
             "\n"
             "/// Free Doxygen without space\n"
             "\n"
             "///   Free Doxygen with 3 spaces\n"
             "\n"
+            "// 🐉 A nice dragon\n"
+            "\n"
+            "//\t abccba\n"
+            "\n"
+            "//\\t deffed\n"
+            "\n"
+            "//   🐉 Another nice dragon\n"
+            "\n"
+            "//   \t Three leading spaces following tab\n"
+            "\n"
+            "//   \\t Three leading spaces following backslash\n"
+            "\n"
             "/// A Doxygen Comment with a nested list:\n"
             "/// - Foo\n"
             "/// - Bar\n"
             "///   - Baz\n"
             "///   - End\n"
             "///     of the inner list\n"
             "///   .\n"
             "/// .\n"
TEST_F(FormatTestComments, SpaceAtLineCommentBegin) {
   EXPECT_EQ("//Free comment without space\n"
             "\n"
             "//Free comment with 3 spaces\n"
             "\n"
             "///Free Doxygen without space\n"
             "\n"
             "///Free Doxygen with 3 spaces\n"
             "\n"
+            "//🐉 A nice dragon\n"
+            "\n"
+            "//\t abccba\n"
+            "\n"
+            "//\\t deffed\n"
+            "\n"
+            "//🐉 Another nice dragon\n"
+            "\n"
+            "//\t Three leading spaces following tab\n"
+            "\n"
+            "//\\t Three leading spaces following backslash\n"
+            "\n"
             "///A Doxygen Comment with a nested list:\n"
             "///- Foo\n"
             "///- Bar\n"
             "///  - Baz\n" // Here we keep the relative indentation
             "///  - End\n"
             "///    of the inner list\n"
             "///  .\n"
             "///.\n"
TEST_F(FormatTestComments, SpaceAtLineCommentBegin) {
   EXPECT_EQ("//  Free comment without space\n"
             "\n"
             "//   Free comment with 3 spaces\n"
             "\n"
             "///  Free Doxygen without space\n"
             "\n"
             "///   Free Doxygen with 3 spaces\n"
             "\n"
+            "//  🐉 A nice dragon\n"
+            "\n"
+            "//\t abccba\n"
+            "\n"
+            "//\\t deffed\n"
+            "\n"
+            "//   🐉 Another nice dragon\n"
+            "\n"
+            "//   \t Three leading spaces following tab\n"
+            "\n"
+            "//   \\t Three leading spaces following backslash\n"
+            "\n"
             "///  A Doxygen Comment with a nested list:\n"
             "///  - Foo\n"
             "///  - Bar\n"
             "///    - Baz\n"
             "///    - End\n"
             "///      of the inner list\n"
             "///    .\n"
             "///  .\n"
TEST_F(FormatTestComments, SpaceAtLineCommentBegin) {
   EXPECT_EQ("// Free comment without space\n"
             "\n"
             "//   Free comment with 3 spaces\n"
             "\n"
             "/// Free Doxygen without space\n"
             "\n"
             "///   Free Doxygen with 3 spaces\n"
             "\n"
+            "// 🐉 A nice dragon\n"
+            "\n"
+            "//\t abccba\n"
+            "\n"
+            "//\\t deffed\n"
+            "\n"
+            "//   🐉 Another nice dragon\n"
+            "\n"
+            "//   \t Three leading spaces following tab\n"
+            "\n"
+            "//   \\t Three leading spaces following backslash\n"
+            "\n"
             "/// A Doxygen Comment with a nested list:\n"
             "/// - Foo\n"
             "/// - Bar\n"
             "///   - Baz\n"
             "///   - End\n"
             "///     of the inner list\n"
             "///   .\n"
             "/// .\n"
TEST_F(FormatTestComments, SpaceAtLineCommentBegin) {
   EXPECT_EQ("//Free comment without space\n"
             "\n"
             "//Free comment with 3 spaces\n"
             "\n"
             "///Free Doxygen without space\n"
             "\n"
             "///Free Doxygen with 3 spaces\n"
             "\n"
+            "//🐉 A nice dragon\n"
+            "\n"
+            "//\t abccba\n"
+            "\n"
+            "//\\t deffed\n"
+            "\n"
+            "//🐉 Another nice dragon\n"
+            "\n"
+            "//\t Three leading spaces following tab\n"
+            "\n"
+            "//\\t Three leading spaces following backslash\n"
+            "\n"
             "///A Doxygen Comment with a nested list:\n"
             "///- Foo\n"
             "///- Bar\n"
             "///  - Baz\n" // Here we keep the relative indentation
             "///  - End\n"
             "///    of the inner list\n"
             "///  .\n"
             "///.\n"
TEST_F(FormatTestComments, SpaceAtLineCommentBegin) {
   EXPECT_EQ("//  Free comment without space\n"
             "\n"
             "//   Free comment with 3 spaces\n"
             "\n"
             "///  Free Doxygen without space\n"
             "\n"
             "///   Free Doxygen with 3 spaces\n"
             "\n"
+            "//  🐉 A nice dragon\n"
+            "\n"
+            "//\t abccba\n"
+            "\n"
+            "//\\t deffed\n"
+            "\n"
+            "//   🐉 Another nice dragon\n"
+            "\n"
+            "//   \t Three leading spaces following tab\n"
+            "\n"
+            "//   \\t Three leading spaces following backslash\n"
+            "\n"
             "///  A Doxygen Comment with a nested list:\n"
             "///  - Foo\n"
             "///  - Bar\n"
             "///    - Baz\n"
             "///    - End\n"
             "///      of the inner list\n"
             "///    .\n"
             "///  .\n"