[clang-format] Adds a canonical delimiter to raw string formatting

[krasimir]

Summary

This patch adds canonical delimiter support to the raw string formatting.
This allows matching delimiters to be updated to the canonical one.

[bkramer]

lg

lib/Format/ContinuationIndenter.cpp

1336

Can this be a StringRef? Can RawText outlive the Current token?

[krasimir]

• Add a comment about std::string

[krasimir]

lib/Format/ContinuationIndenter.cpp

1336

Nice catch! The reformatting code creates a virtual source code environment which expects a null-terminated string. Added a comment why this is necessary.

docs/ClangFormatStyleOptions.rst

**RawStringFormats** (``std::vector<RawStringFormat>``)
   name will be reformatted assuming the specified language based on the
   style for that language defined in the .clang-format file. If no style has
   been defined in the .clang-format file for the specific language, a
   predefined style given by 'BasedOnStyle' is used. If 'BasedOnStyle' is not
   found, the formatting is based on llvm style. A matching delimiter takes
   precedence over a matching enclosing function name for determining the
   language of the raw string contents.
 
+  If a canonical delimiter is specified, occurences of other delimiters for
+  the same language will be updated to the canonical if possible.
+
   There should be at most one specification per language and each delimiter
   and enclosing function should not occur in multiple specifications.
 
   To configure this in the .clang-format file, use:
 
   .. code-block:: yaml
 
     RawStringFormats:
       - Language: TextProto
           Delimiters:
             - 'pb'
             - 'proto'
           EnclosingFunctions:
             - 'PARSE_TEXT_PROTO'
           BasedOnStyle: google
       - Language: Cpp
           Delimiters:
             - 'cc'
             - 'cpp'
           BasedOnStyle: llvm
+          CanonicalDelimiter: 'cc'
 
 **ReflowComments** (``bool``)
   If ``true``, clang-format will attempt to re-flow comments.
 
   .. code-block:: c++
 
      false:
      // veryVeryVeryVeryVeryVeryVeryVeryVeryVeryVeryLongComment with plenty of information

include/clang/Format/Format.h

   /// See documentation of ``RawStringFormats``.
   struct RawStringFormat {
     /// \brief The language of this raw string.
     LanguageKind Language;
     /// \brief A list of raw string delimiters that match this language.
     std::vector<std::string> Delimiters;
     /// \brief A list of enclosing function names that match this language.
     std::vector<std::string> EnclosingFunctions;
+    /// \brief The canonical delimiter for this language.
+    std::string CanonicalDelimiter;
     /// \brief The style name on which this raw string format is based on.
     /// If not specified, the raw string format is based on the style that this
     /// format is based on.
     std::string BasedOnStyle;
     bool operator==(const RawStringFormat &Other) const {
       return Language == Other.Language && Delimiters == Other.Delimiters &&
              EnclosingFunctions == Other.EnclosingFunctions &&
+             CanonicalDelimiter == Other.CanonicalDelimiter &&
              BasedOnStyle == Other.BasedOnStyle;
     }
   };
 
   /// \brief Defines hints for detecting supported languages code blocks in raw
   /// strings.
   ///
   /// A raw string with a matching delimiter or a matching enclosing function
   /// name will be reformatted assuming the specified language based on the
   /// style for that language defined in the .clang-format file. If no style has
   /// been defined in the .clang-format file for the specific language, a
   /// predefined style given by 'BasedOnStyle' is used. If 'BasedOnStyle' is not
   /// found, the formatting is based on llvm style. A matching delimiter takes
   /// precedence over a matching enclosing function name for determining the
   /// language of the raw string contents.
   ///
+  /// If a canonical delimiter is specified, occurences of other delimiters for
+  /// the same language will be updated to the canonical if possible.
+  ///
   /// There should be at most one specification per language and each delimiter
   /// and enclosing function should not occur in multiple specifications.
   ///
   /// To configure this in the .clang-format file, use:
   /// \code{.yaml}
   ///   RawStringFormats:
   ///     - Language: TextProto
   ///         Delimiters:
   ///           - 'pb'
   ///           - 'proto'
   ///         EnclosingFunctions:
   ///           - 'PARSE_TEXT_PROTO'
   ///         BasedOnStyle: google
   ///     - Language: Cpp
   ///         Delimiters:
   ///           - 'cc'
   ///           - 'cpp'
   ///         BasedOnStyle: llvm
+  ///         CanonicalDelimiter: 'cc'
   /// \endcode
   std::vector<RawStringFormat> RawStringFormats;
 
   /// \brief If ``true``, clang-format will attempt to re-flow comments.
   /// \code
   ///    false:
   ///    // veryVeryVeryVeryVeryVeryVeryVeryVeryVeryVeryLongComment with plenty of information
   ///    /* second veryVeryVeryVeryVeryVeryVeryVeryVeryVeryVeryLongComment with plenty of information */

lib/Format/ContinuationIndenter.cpp

static llvm::Optional<StringRef> getRawStringDelimiter(StringRef TokenText) {
   size_t RParenPos = TokenText.size() - Delimiter.size() - 2;
   if (TokenText[RParenPos] != ')')
     return None;
   if (!TokenText.substr(RParenPos + 1).startswith(Delimiter))
     return None;
   return Delimiter;
 }
 
+// Returns the canonical delimiter for \p Language, or the empty string if no
+// canonical delimiter is specified.
+static StringRef
+getCanonicalRawStringDelimiter(const FormatStyle &Style,
+                               FormatStyle::LanguageKind Language) {
+  for (const auto &Format : Style.RawStringFormats) {
+    if (Format.Language == Language)
+      return StringRef(Format.CanonicalDelimiter);
+  }
+  return "";
+}
+
 RawStringFormatStyleManager::RawStringFormatStyleManager(
     const FormatStyle &CodeStyle) {
   for (const auto &RawStringFormat : CodeStyle.RawStringFormats) {
     llvm::Optional<FormatStyle> LanguageStyle =
         CodeStyle.GetLanguageStyle(RawStringFormat.Language);
     if (!LanguageStyle) {
       FormatStyle PredefinedStyle;
       if (!getPredefinedStyle(RawStringFormat.BasedOnStyle,
return encoding::columnWidthWithTabs(Text.substr(LastNewlinePos),
                                          /*StartColumn=*/0, TabWidth, Encoding);
   }
 }
 
 unsigned ContinuationIndenter::reformatRawStringLiteral(
     const FormatToken &Current, LineState &State,
     const FormatStyle &RawStringStyle, bool DryRun) {
   unsigned StartColumn = State.Column - Current.ColumnWidth;
-  auto Delimiter = *getRawStringDelimiter(Current.TokenText);
+  StringRef OldDelimiter = *getRawStringDelimiter(Current.TokenText);
+  StringRef NewDelimiter =
+      getCanonicalRawStringDelimiter(Style, RawStringStyle.Language);
+  if (NewDelimiter.empty() || OldDelimiter.empty())
+    NewDelimiter = OldDelimiter;
   // The text of a raw string is between the leading 'R"delimiter(' and the
   // trailing 'delimiter)"'.
-  unsigned PrefixSize = 3 + Delimiter.size();
-  unsigned SuffixSize = 2 + Delimiter.size();
+  unsigned OldPrefixSize = 3 + OldDelimiter.size();
+  unsigned OldSuffixSize = 2 + OldDelimiter.size();
+  // We create a virtual text environment which expects a null-terminated
+  // string, so we cannot use StringRef.
+  std::string RawText =
+      Current.TokenText.substr(OldPrefixSize).drop_back(OldSuffixSize);
+  if (NewDelimiter != OldDelimiter) {
+    // Don't update to the canonical delimiter 'deli' if ')deli"' occurs in the
+    // raw string.
+    std::string CanonicalDelimiterSuffix = (")" + NewDelimiter + "\"").str();
+    if (StringRef(RawText).contains(CanonicalDelimiterSuffix))
+      NewDelimiter = OldDelimiter;
+  }
 
-  // The first start column is the column the raw text starts.
-  unsigned FirstStartColumn = StartColumn + PrefixSize;
+  unsigned NewPrefixSize = 3 + NewDelimiter.size();
+  unsigned NewSuffixSize = 2 + NewDelimiter.size();
+
+  // The first start column is the column the raw text starts after formatting.
+  unsigned FirstStartColumn = StartColumn + NewPrefixSize;
 
   // The next start column is the intended indentation a line break inside
   // the raw string at level 0. It is determined by the following rules:
   //   - if the content starts on newline, it is one level more than the current
   //     indent, and
   //   - if the content does not start on a newline, it is the first start
   //     column.
   // These rules have the advantage that the formatted content both does not
   // violate the rectangle rule and visually flows within the surrounding
   // source.
-  bool ContentStartsOnNewline = Current.TokenText[PrefixSize] == '\n';
+  bool ContentStartsOnNewline = Current.TokenText[OldPrefixSize] == '\n';
   unsigned NextStartColumn = ContentStartsOnNewline
                                  ? State.Stack.back().Indent + Style.IndentWidth
                                  : FirstStartColumn;
 
   // The last start column is the column the raw string suffix starts if it is
   // put on a newline.
   // The last start column is the intended indentation of the raw string postfix
   // if it is put on a newline. It is determined by the following rules:
   //   - if the raw string prefix starts on a newline, it is the column where
   //     that raw string prefix starts, and
   //   - if the raw string prefix does not start on a newline, it is the current
   //     indent.
   unsigned LastStartColumn = Current.NewlinesBefore
-                                 ? FirstStartColumn - PrefixSize
+                                 ? FirstStartColumn - NewPrefixSize
                                  : State.Stack.back().Indent;
 
-  std::string RawText =
-      Current.TokenText.substr(PrefixSize).drop_back(SuffixSize);
-
   std::pair<tooling::Replacements, unsigned> Fixes = internal::reformat(
       RawStringStyle, RawText, {tooling::Range(0, RawText.size())},
       FirstStartColumn, NextStartColumn, LastStartColumn, "<stdin>",
       /*Status=*/nullptr);
 
   auto NewCode = applyAllReplacements(RawText, Fixes.first);
   tooling::Replacements NoFixes;
   if (!NewCode) {
     State.Column += Current.ColumnWidth;
     return 0;
   }
   if (!DryRun) {
+    if (NewDelimiter != OldDelimiter) {
+      // In 'R"delimiter(...', the delimiter starts 2 characters after the start
+      // of the token.
+      SourceLocation PrefixDelimiterStart =
+          Current.Tok.getLocation().getLocWithOffset(2);
+      auto PrefixErr = Whitespaces.addReplacement(tooling::Replacement(
+          SourceMgr, PrefixDelimiterStart, OldDelimiter.size(), NewDelimiter));
+      if (PrefixErr) {
+        llvm::errs()
+            << "Failed to update the prefix delimiter of a raw string: "
+            << llvm::toString(std::move(PrefixErr)) << "\n";
+      }
+      // In 'R"delimiter(...)delimiter"', the suffix delimiter starts at
+      // position length - 1 - |delimiter|.
+      SourceLocation SuffixDelimiterStart =
+          Current.Tok.getLocation().getLocWithOffset(Current.TokenText.size() -
+                                                     1 - OldDelimiter.size());
+      auto SuffixErr = Whitespaces.addReplacement(tooling::Replacement(
+          SourceMgr, SuffixDelimiterStart, OldDelimiter.size(), NewDelimiter));
+      if (SuffixErr) {
+        llvm::errs()
+            << "Failed to update the suffix delimiter of a raw string: "
+            << llvm::toString(std::move(SuffixErr)) << "\n";
+      }
+    }
     SourceLocation OriginLoc =
-        Current.Tok.getLocation().getLocWithOffset(PrefixSize);
+        Current.Tok.getLocation().getLocWithOffset(OldPrefixSize);
     for (const tooling::Replacement &Fix : Fixes.first) {
       auto Err = Whitespaces.addReplacement(tooling::Replacement(
           SourceMgr, OriginLoc.getLocWithOffset(Fix.getOffset()),
           Fix.getLength(), Fix.getReplacementText()));
       if (Err) {
         llvm::errs() << "Failed to reformat raw string: "
                      << llvm::toString(std::move(Err)) << "\n";
       }
     }
   }
   unsigned RawLastLineEndColumn = getLastLineEndColumn(
       *NewCode, FirstStartColumn, Style.TabWidth, Encoding);
-  State.Column = RawLastLineEndColumn + SuffixSize;
+  State.Column = RawLastLineEndColumn + NewSuffixSize;
   return Fixes.second;
 }
 
 unsigned ContinuationIndenter::addMultilineToken(const FormatToken &Current,
                                                  LineState &State) {
   // Break before further function parameters on all levels.
   for (unsigned i = 0, e = State.Stack.size(); i != e; ++i)
     State.Stack[i].BreakBeforeParameter = true;

lib/Format/Format.cpp

template <> struct ScalarEnumerationTraits<FormatStyle::IncludeBlocksStyle> {
   }
 };
 
 template <> struct MappingTraits<FormatStyle::RawStringFormat> {
   static void mapping(IO &IO, FormatStyle::RawStringFormat &Format) {
     IO.mapOptional("Language", Format.Language);
     IO.mapOptional("Delimiters", Format.Delimiters);
     IO.mapOptional("EnclosingFunctions", Format.EnclosingFunctions);
+    IO.mapOptional("CanonicalDelimiter", Format.CanonicalDelimiter);
     IO.mapOptional("BasedOnStyle", Format.BasedOnStyle);
   }
 };
 
 // Allows to read vector<FormatStyle> while keeping default values.
 // IO.getContext() should contain a pointer to the FormatStyle structure, that
 // will be used to get default values for missing keys.
 // If the first element has no Language specified, it will be treated as the
GoogleStyle.RawStringFormats = {{
           "TEXTPROTO",
       },
       /*EnclosingFunctionNames=*/
        {
            "EqualsProto",
            "PARSE_TEXT_PROTO",
            "ParseTextProto",
        },
+      /*CanonicalDelimiter=*/"",
       /*BasedOnStyle=*/"google",
   }};
   GoogleStyle.SpacesBeforeTrailingComments = 2;
   GoogleStyle.Standard = FormatStyle::LS_Auto;
 
   GoogleStyle.PenaltyReturnTypeOnItsOwnLine = 200;
   GoogleStyle.PenaltyBreakBeforeFirstCallParameter = 1;
 

unittests/Format/FormatTest.cpp

TEST_F(FormatTest, ParsesConfiguration) {
   CHECK_PARSE("IncludeIsMainRegex: 'abc$'", IncludeIsMainRegex, "abc$");
 
   Style.RawStringFormats.clear();
   std::vector<FormatStyle::RawStringFormat> ExpectedRawStringFormats = {
       {
           FormatStyle::LK_TextProto,
           {"pb", "proto"},
           {"PARSE_TEXT_PROTO"},
+          /*CanonicalDelimiter=*/"",
           "llvm",
       },
       {
           FormatStyle::LK_Cpp,
           {"cc", "cpp"},
           {"C_CODEBLOCK", "CPPEVAL"},
-          "",
+          /*CanonicalDelimiter=*/"cc",
+          /*BasedOnStyle=*/"",
       },
   };
 
   CHECK_PARSE("RawStringFormats:\n"
               "  - Language: TextProto\n"
               "    Delimiters:\n"
               "      - 'pb'\n"
               "      - 'proto'\n"
               "    EnclosingFunctions:\n"
               "      - 'PARSE_TEXT_PROTO'\n"
               "    BasedOnStyle: llvm\n"
               "  - Language: Cpp\n"
               "    Delimiters:\n"
               "      - 'cc'\n"
               "      - 'cpp'\n"
               "    EnclosingFunctions:\n"
               "      - 'C_CODEBLOCK'\n"
-              "      - 'CPPEVAL'\n",
+              "      - 'CPPEVAL'\n"
+              "    CanonicalDelimiter: 'cc'",
               RawStringFormats, ExpectedRawStringFormats);
 }
 
 TEST_F(FormatTest, ParsesConfigurationWithLanguages) {
   FormatStyle Style = {};
   Style.Language = FormatStyle::LK_Cpp;
   CHECK_PARSE("Language: Cpp\n"
               "IndentWidth: 12",

unittests/Format/FormatTestRawStrings.cpp

protected:
   }
 
   int ReplacementCount;
 
   FormatStyle getRawStringPbStyleWithColumns(unsigned ColumnLimit) {
     FormatStyle Style = getLLVMStyle();
     Style.ColumnLimit = ColumnLimit;
     Style.RawStringFormats = {
-        {/*Language=*/FormatStyle::LK_TextProto,
+        {
+            /*Language=*/FormatStyle::LK_TextProto,
             /*Delimiters=*/{"pb"},
             /*EnclosingFunctions=*/{},
-         /*BasedOnStyle=*/"google"},
+            /*CanonicalDelimiter=*/"",
+            /*BasedOnStyle=*/"google",
+        },
     };
     return Style;
   }
 
   FormatStyle getRawStringLLVMCppStyleBasedOn(std::string BasedOnStyle) {
     FormatStyle Style = getLLVMStyle();
     Style.RawStringFormats = {
-        {/*Language=*/FormatStyle::LK_Cpp,
+        {
+            /*Language=*/FormatStyle::LK_Cpp,
             /*Delimiters=*/{"cpp"},
-         /*EnclosingFunctions=*/{}, BasedOnStyle},
+            /*EnclosingFunctions=*/{},
+            /*CanonicalDelimiter=*/"",
+            BasedOnStyle,
+        },
     };
     return Style;
   }
 
   FormatStyle getRawStringGoogleCppStyleBasedOn(std::string BasedOnStyle) {
     FormatStyle Style = getGoogleStyle(FormatStyle::LK_Cpp);
     Style.RawStringFormats = {
-        {/*Language=*/FormatStyle::LK_Cpp,
+        {
+            /*Language=*/FormatStyle::LK_Cpp,
             /*Delimiters=*/{"cpp"},
-         /*EnclosingFunctions=*/{}, BasedOnStyle},
+            /*EnclosingFunctions=*/{},
+            /*CanonicalDelimiter=*/"",
+            BasedOnStyle,
+        },
     };
     return Style;
   }
 
   // Gcc 4.8 doesn't support raw string literals in macros, which breaks some
   // build bots. We use this function instead.
   void expect_eq(const std::string Expected, const std::string Actual) {
     EXPECT_EQ(Expected, Actual);
TEST_F(FormatTestRawStrings, UsesConfigurationOverBaseStyle) {
   // google style puts '*' on the left.
 
   // Uses the configured google style inside raw strings even if BasedOnStyle in
   // the raw string format is llvm.
   FormatStyle Style = getGoogleStyle(FormatStyle::LK_Cpp);
   EXPECT_EQ(0, parseConfiguration("---\n"
                                   "Language: Cpp\n"
                                   "BasedOnStyle: Google", &Style).value());
-  Style.RawStringFormats = {{FormatStyle::LK_Cpp, {"cpp"}, {}, "llvm"}};
+  Style.RawStringFormats = {{
+      FormatStyle::LK_Cpp,
+      {"cpp"},
+      {},
+      /*CanonicalDelimiter=*/"",
+      /*BasedOnStyle=*/"llvm",
+  }};
   expect_eq(R"test(int* i = R"cpp(int* j = 0;)cpp";)test",
             format(R"test(int * i = R"cpp(int * j = 0;)cpp";)test", Style));
 }
 
 TEST_F(FormatTestRawStrings, MatchesDelimitersCaseSensitively) {
   // Don't touch the 'PB' raw string, format the 'pb' raw string.
   expect_eq(R"test(
 s = R"PB(item:1)PB";
a = PARSE_TEXT_PROTO/**/(/**/R"(key:value)");)test",
   expect_eq(R"test(
 a = ParseTextProto<ProtoType>(
     R"(key: value)");)test",
             format(R"test(
 a = ParseTextProto<ProtoType>(R"(key:value)");)test",
                    Style));
 }
 
+TEST_F(FormatTestRawStrings, UpdatesToCanonicalDelimiters) {
+  FormatStyle Style = getRawStringPbStyleWithColumns(25);
+  Style.RawStringFormats[0].CanonicalDelimiter = "proto";
+  expect_eq(R"test(a = R"proto(key: value)proto";)test",
+            format(R"test(a = R"pb(key:value)pb";)test", Style));
+
+  // Don't update to canonical delimiter if it occurs as a raw string suffix in
+  // the raw string content.
+  expect_eq(R"test(a = R"pb(key: ")proto")pb";)test",
+            format(R"test(a = R"pb(key:")proto")pb";)test", Style));
+}
+
 } // end namespace
 } // end namespace format
 } // end namespace clang