[clang-format] adds enclosing function detection to raw string formatting

[krasimir]

Summary

This patch adds enclosing function detection to raw string formatting.

[krasimir]

• Update comments

[bkramer]

lib/Format/ContinuationIndenter.cpp

1454

This function doesn't look clang-formatted.

[krasimir]

• Format a function

[bkramer]

lgtm

docs/ClangFormatStyleOptions.rst

**BreakConstructorInitializers** (``BreakConstructorInitializersStyle``)
 
   Possible values:
 
   * ``BCIS_BeforeColon`` (in configuration: ``BeforeColon``)
     Break constructor initializers before the colon and after the commas.
 
     .. code-block:: c++
 
-      Constructor()
-          : initializer1(),
-            initializer2()
 
   * ``BCIS_BeforeComma`` (in configuration: ``BeforeComma``)
     Break constructor initializers before the colon and commas, and align
     the commas with the colon.
 
     .. code-block:: c++
 
-      Constructor()
-          : initializer1()
-          , initializer2()
 
   * ``BCIS_AfterColon`` (in configuration: ``AfterColon``)
     Break constructor initializers after the colon and commas.
 
     .. code-block:: c++
 
-      Constructor() :
-          initializer1(),
-          initializer2()
 
 
 
 **BreakStringLiterals** (``bool``)
   Allow breaking string literals when formatting.
 
 **ColumnLimit** (``unsigned``)
   The column limit.
.. code-block:: c++
 
        #include "b.h"               into      #include "a.h"
                                               #include "b.h"
        #include <lib/main.h>                  #include <lib/main.h>
        #include "a.h"
 
   * ``IBS_Regroup`` (in configuration: ``Regroup``)
     Merge multiple ``#include`` blocks together and sort as one.
-    Then split into groups based on category priority. See ``IncludeCategories``.
+    Then split into groups based on category priority. See
+    ``IncludeCategories``.
 
     .. code-block:: c++
 
        #include "b.h"               into      #include "a.h"
                                               #include "b.h"
        #include <lib/main.h>
        #include "a.h"                         #include <lib/main.h>
 
* ``PAS_Middle`` (in configuration: ``Middle``)
 
     .. code-block:: c++
 
       int * a;
 
 
 
 **RawStringFormats** (``std::vector<RawStringFormat>``)
-  Raw string delimiters denoting that the raw string contents are
-  code in a particular language and can be reformatted.
+  Defines hints for detecting supported languages code blocks in raw
+  strings.
 
-  A raw string with a matching delimiter will be reformatted assuming the
-  specified language based on a predefined style given by 'BasedOnStyle'.
-  If 'BasedOnStyle' is not found, the formatting is based on llvm style.
+  A raw string with a matching delimiter or a matching enclosing function
+  name will be reformatted assuming the specified language based on the
+  style for that language defined in the .clang-format file. If no style has
+  been defined in the .clang-format file for the specific language, a
+  predefined style given by 'BasedOnStyle' is used. If 'BasedOnStyle' is not
+  found, the formatting is based on llvm style. A matching delimiter takes
+  precedence over a matching enclosing function name for determining the
+  language of the raw string contents.
+
+  There should be at most one specification per language and each delimiter
+  and enclosing function should not occur in multiple specifications.
 
   To configure this in the .clang-format file, use:
 
   .. code-block:: yaml
 
     RawStringFormats:
-      - Delimiter: 'pb'
-        Language:  TextProto
-        BasedOnStyle: llvm
-      - Delimiter: 'proto'
-        Language:  TextProto
+      - Language: TextProto
+          Delimiters:
+            - 'pb'
+            - 'proto'
+          EnclosingFunctions:
+            - 'PARSE_TEXT_PROTO'
           BasedOnStyle: google
+      - Language: Cpp
+          Delimiters:
+            - 'cc'
+            - 'cpp'
+          BasedOnStyle: llvm
 
 **ReflowComments** (``bool``)
   If ``true``, clang-format will attempt to re-flow comments.
 
   .. code-block:: c++
 
      false:
      // veryVeryVeryVeryVeryVeryVeryVeryVeryVeryVeryLongComment with plenty of information

include/clang/Format/Format.h

   PointerAlignmentStyle PointerAlignment;
 
   /// See documentation of ``RawStringFormats``.
   struct RawStringFormat {
     /// \brief The language of this raw string.
     LanguageKind Language;
     /// \brief A list of raw string delimiters that match this language.
     std::vector<std::string> Delimiters;
+    /// \brief A list of enclosing function names that match this language.
+    std::vector<std::string> EnclosingFunctions;
     /// \brief The style name on which this raw string format is based on.
     /// If not specified, the raw string format is based on the style that this
     /// format is based on.
     std::string BasedOnStyle;
     bool operator==(const RawStringFormat &Other) const {
       return Language == Other.Language && Delimiters == Other.Delimiters &&
+             EnclosingFunctions == Other.EnclosingFunctions &&
              BasedOnStyle == Other.BasedOnStyle;
     }
   };
 
   /// \brief Defines hints for detecting supported languages code blocks in raw
   /// strings.
   ///
-  /// A raw string with a matching delimiter will be reformatted assuming the
-  /// specified language based on the style for that language defined in the
-  /// .clang-format file. If no style has been defined in the .clang-format file
-  /// for the specific language, a predefined style given by 'BasedOnStyle' is
-  /// used. If 'BasedOnStyle' is not found, the formatting is based on llvm
-  /// style.
+  /// A raw string with a matching delimiter or a matching enclosing function
+  /// name will be reformatted assuming the specified language based on the
+  /// style for that language defined in the .clang-format file. If no style has
+  /// been defined in the .clang-format file for the specific language, a
+  /// predefined style given by 'BasedOnStyle' is used. If 'BasedOnStyle' is not
+  /// found, the formatting is based on llvm style. A matching delimiter takes
+  /// precedence over a matching enclosing function name for determining the
+  /// language of the raw string contents.
+  ///
+  /// There should be at most one specification per language and each delimiter
+  /// and enclosing function should not occur in multiple specifications.
   ///
   /// To configure this in the .clang-format file, use:
   /// \code{.yaml}
   ///   RawStringFormats:
   ///     - Language: TextProto
   ///         Delimiters:
   ///           - 'pb'
   ///           - 'proto'
+  ///         EnclosingFunctions:
+  ///           - 'PARSE_TEXT_PROTO'
   ///         BasedOnStyle: google
   ///     - Language: Cpp
   ///         Delimiters:
   ///           - 'cc'
   ///           - 'cpp'
   ///         BasedOnStyle: llvm
   /// \endcode
   std::vector<RawStringFormat> RawStringFormats;

lib/Format/ContinuationIndenter.h

 struct FormatToken;
 struct LineState;
 struct ParenState;
 struct RawStringFormatStyleManager;
 class WhitespaceManager;
 
 struct RawStringFormatStyleManager {
   llvm::StringMap<FormatStyle> DelimiterStyle;
+  llvm::StringMap<FormatStyle> EnclosingFunctionStyle;
 
   RawStringFormatStyleManager(const FormatStyle &CodeStyle);
 
-  llvm::Optional<FormatStyle> get(StringRef Delimiter) const;
+  llvm::Optional<FormatStyle> getDelimiterStyle(StringRef Delimiter) const;
+
+  llvm::Optional<FormatStyle>
+  getEnclosingFunctionStyle(StringRef EnclosingFunction) const;
 };
 
 class ContinuationIndenter {
 public:
   /// \brief Constructs a \c ContinuationIndenter to format \p Line starting in
   /// column \p FirstIndent.
   ContinuationIndenter(const FormatStyle &Style,
                        const AdditionalKeywords &Keywords,

lib/Format/ContinuationIndenter.cpp

static llvm::Optional<StringRef> getRawStringDelimiter(StringRef TokenText) {
   if (!TokenText.substr(RParenPos + 1).startswith(Delimiter))
     return None;
   return Delimiter;
 }
 
 RawStringFormatStyleManager::RawStringFormatStyleManager(
     const FormatStyle &CodeStyle) {
   for (const auto &RawStringFormat : CodeStyle.RawStringFormats) {
-    for (StringRef Delimiter : RawStringFormat.Delimiters) {
-      llvm::Optional<FormatStyle> LanguageStyle =
-          CodeStyle.GetLanguageStyle(RawStringFormat.Language);
-      if (!LanguageStyle) {
-        FormatStyle PredefinedStyle;
-        if (!getPredefinedStyle(RawStringFormat.BasedOnStyle,
-                                RawStringFormat.Language, &PredefinedStyle)) {
-          PredefinedStyle = getLLVMStyle();
-          PredefinedStyle.Language = RawStringFormat.Language;
-        }
-        LanguageStyle = PredefinedStyle;
-      }
-      LanguageStyle->ColumnLimit = CodeStyle.ColumnLimit;
+    for (StringRef Delimiter : RawStringFormat.Delimiters) {
       DelimiterStyle.insert({Delimiter, *LanguageStyle});
     }
+    for (StringRef EnclosingFunction : RawStringFormat.EnclosingFunctions) {
+      EnclosingFunctionStyle.insert({EnclosingFunction, *LanguageStyle});
+    }
   }
 }
 
 llvm::Optional<FormatStyle>
-RawStringFormatStyleManager::get(StringRef Delimiter) const {
+RawStringFormatStyleManager::getDelimiterStyle(StringRef Delimiter) const {
   auto It = DelimiterStyle.find(Delimiter);
   if (It == DelimiterStyle.end())
     return None;
   return It->second;
 }
 
+llvm::Optional<FormatStyle>
+RawStringFormatStyleManager::getEnclosingFunctionStyle(
+    StringRef EnclosingFunction) const {
+  auto It = EnclosingFunctionStyle.find(EnclosingFunction);
+  if (It == EnclosingFunctionStyle.end())
+    return None;
+  return It->second;
+}
+
 ContinuationIndenter::ContinuationIndenter(const FormatStyle &Style,
                                            const AdditionalKeywords &Keywords,
                                            const SourceManager &SourceMgr,
                                            WhitespaceManager &Whitespaces,
                                            encoding::Encoding Encoding,
                                            bool BinPackInconclusiveFunctions)
     : Style(Style), Keywords(Keywords), SourceMgr(SourceMgr),
       Whitespaces(Whitespaces), Encoding(Encoding),
unsigned ContinuationIndenter::handleEndOfLine(const FormatToken &Current,
   }
   if (State.Column > getColumnLimit(State)) {
     unsigned ExcessCharacters = State.Column - getColumnLimit(State);
     Penalty += Style.PenaltyExcessCharacter * ExcessCharacters;
   }
   return Penalty;
 }
 
+// Returns the enclosing function name of a token, or the empty string if not
+// found.
+static StringRef getEnclosingFunctionName(const FormatToken &Current) {
+  // Look for: 'function(' or 'function<templates>(' before Current.
+  auto Tok = Current.getPreviousNonComment();
+  if (!Tok || !Tok->is(tok::l_paren))
+    return "";
+  Tok = Tok->getPreviousNonComment();
+  if (!Tok)
+    return "";
+  if (Tok->is(TT_TemplateCloser)) {
+    Tok = Tok->MatchingParen;
+    if (Tok)
+      Tok = Tok->getPreviousNonComment();
+  }
+  if (!Tok || !Tok->is(tok::identifier))
+    return "";
+  return Tok->TokenText;
+}
+
 llvm::Optional<FormatStyle>
 ContinuationIndenter::getRawStringStyle(const FormatToken &Current,
                                         const LineState &State) {
   if (!Current.isStringLiteral())
     return None;
   auto Delimiter = getRawStringDelimiter(Current.TokenText);
   if (!Delimiter)
     return None;
-  auto RawStringStyle = RawStringFormats.get(*Delimiter);
+  auto RawStringStyle = RawStringFormats.getDelimiterStyle(*Delimiter);
+  if (!RawStringStyle)
+    RawStringStyle = RawStringFormats.getEnclosingFunctionStyle(
+        getEnclosingFunctionName(Current));
   if (!RawStringStyle)
     return None;
   RawStringStyle->ColumnLimit = getColumnLimit(State);
   return RawStringStyle;
 }
 
 std::unique_ptr<BreakableToken> ContinuationIndenter::createBreakableToken(
     const FormatToken &Current, LineState &State, bool AllowBreak) {

lib/Format/Format.cpp

static void enumeration(IO &IO, FormatStyle::IncludeBlocksStyle &Value) {
     IO.enumCase(Value, "Regroup", FormatStyle::IBS_Regroup);
   }
 };
 
 template <> struct MappingTraits<FormatStyle::RawStringFormat> {
   static void mapping(IO &IO, FormatStyle::RawStringFormat &Format) {
     IO.mapOptional("Language", Format.Language);
     IO.mapOptional("Delimiters", Format.Delimiters);
+    IO.mapOptional("EnclosingFunctions", Format.EnclosingFunctions);
     IO.mapOptional("BasedOnStyle", Format.BasedOnStyle);
   }
 };
 
 // Allows to read vector<FormatStyle> while keeping default values.
 // IO.getContext() should contain a pointer to the FormatStyle structure, that
 // will be used to get default values for missing keys.
 // If the first element has no Language specified, it will be treated as the
GoogleStyle.RawStringFormats = {{
       {
           "pb",
           "PB",
           "proto",
           "PROTO",
           "textproto",
           "TEXTPROTO",
       },
+      /*EnclosingFunctionNames=*/
+       {
+           "EqualsProto",
+           "PARSE_TEXT_PROTO",
+           "ParseTextProto",
+       },
       /*BasedOnStyle=*/"google",
   }};
   GoogleStyle.SpacesBeforeTrailingComments = 2;
   GoogleStyle.Standard = FormatStyle::LS_Auto;
 
   GoogleStyle.PenaltyReturnTypeOnItsOwnLine = 200;
   GoogleStyle.PenaltyBreakBeforeFirstCallParameter = 1;
 

unittests/Format/FormatTest.cpp

CHECK_PARSE("IncludeCategories:\n"
               "    Priority: 2\n"
               "  - Regex: .*\n"
               "    Priority: 1",
               IncludeCategories, ExpectedCategories);
   CHECK_PARSE("IncludeIsMainRegex: 'abc$'", IncludeIsMainRegex, "abc$");
 
   Style.RawStringFormats.clear();
   std::vector<FormatStyle::RawStringFormat> ExpectedRawStringFormats = {
-      {FormatStyle::LK_TextProto, {"pb", "proto"}, "llvm"},
-      {FormatStyle::LK_Cpp, {"cc", "cpp"}, "google"},
+      {
+          FormatStyle::LK_TextProto,
+          {"pb", "proto"},
+          {"PARSE_TEXT_PROTO"},
+          "llvm",
+      },
+      {
+          FormatStyle::LK_Cpp,
+          {"cc", "cpp"},
+          {"C_CODEBLOCK", "CPPEVAL"},
+          "",
+      },
   };
 
   CHECK_PARSE("RawStringFormats:\n"
               "  - Language: TextProto\n"
               "    Delimiters:\n"
               "      - 'pb'\n"
               "      - 'proto'\n"
+              "    EnclosingFunctions:\n"
+              "      - 'PARSE_TEXT_PROTO'\n"
               "    BasedOnStyle: llvm\n"
               "  - Language: Cpp\n"
               "    Delimiters:\n"
               "      - 'cc'\n"
               "      - 'cpp'\n"
-              "    BasedOnStyle: google\n",
+              "    EnclosingFunctions:\n"
+              "      - 'C_CODEBLOCK'\n"
+              "      - 'CPPEVAL'\n",
               RawStringFormats, ExpectedRawStringFormats);
 }
 
 TEST_F(FormatTest, ParsesConfigurationWithLanguages) {
   FormatStyle Style = {};
   Style.Language = FormatStyle::LK_Cpp;
   CHECK_PARSE("Language: Cpp\n"
               "IndentWidth: 12",

unittests/Format/FormatTestRawStrings.cpp

FormatStyle getLLVMStyleWithColumns(unsigned ColumnLimit) {
     return getStyleWithColumns(getLLVMStyle(), ColumnLimit);
   }
 
   int ReplacementCount;
 
   FormatStyle getRawStringPbStyleWithColumns(unsigned ColumnLimit) {
     FormatStyle Style = getLLVMStyle();
     Style.ColumnLimit = ColumnLimit;
-    Style.RawStringFormats = {{/*Language=*/FormatStyle::LK_TextProto,
-                               /*Delimiters=*/{"pb"},
-                               /*BasedOnStyle=*/"google"}};
+    Style.RawStringFormats = {
+        {/*Language=*/FormatStyle::LK_TextProto,
+         /*EnclosingFunctions=*/{},
+         /*BasedOnStyle=*/"google"},
+    };
     return Style;
   }
 
   FormatStyle getRawStringLLVMCppStyleBasedOn(std::string BasedOnStyle) {
     FormatStyle Style = getLLVMStyle();
-    Style.RawStringFormats = {{/*Language=*/FormatStyle::LK_Cpp,
-                               /*Delimiters=*/{"cpp"}, BasedOnStyle}};
+    Style.RawStringFormats = {
+        {/*Language=*/FormatStyle::LK_Cpp,
+         /*Delimiters=*/{"cpp"},
+         /*EnclosingFunctions=*/{}, BasedOnStyle},
+    };
     return Style;
   }
 
   FormatStyle getRawStringGoogleCppStyleBasedOn(std::string BasedOnStyle) {
     FormatStyle Style = getGoogleStyle(FormatStyle::LK_Cpp);
-    Style.RawStringFormats = {{/*Language=*/FormatStyle::LK_Cpp,
-                               /*Delimiters=*/{"cpp"}, BasedOnStyle}};
+    Style.RawStringFormats = {
+        {/*Language=*/FormatStyle::LK_Cpp,
+         /*Delimiters=*/{"cpp"},
+         /*EnclosingFunctions=*/{}, BasedOnStyle},
+    };
     return Style;
   }
 
   // Gcc 4.8 doesn't support raw string literals in macros, which breaks some
   // build bots. We use this function instead.
   void expect_eq(const std::string Expected, const std::string Actual) {
     EXPECT_EQ(Expected, Actual);
   }
TEST_F(FormatTestRawStrings, UsesConfigurationOverBaseStyle) {
   // google style puts '*' on the left.
 
   // Uses the configured google style inside raw strings even if BasedOnStyle in
   // the raw string format is llvm.
   FormatStyle Style = getGoogleStyle(FormatStyle::LK_Cpp);
   EXPECT_EQ(0, parseConfiguration("---\n"
                                   "Language: Cpp\n"
                                   "BasedOnStyle: Google", &Style).value());
-  Style.RawStringFormats = {{FormatStyle::LK_Cpp, {"cpp"}, "llvm"}};
+  Style.RawStringFormats = {{FormatStyle::LK_Cpp, {"cpp"}, {}, "llvm"}};
   expect_eq(R"test(int* i = R"cpp(int* j = 0;)cpp";)test",
             format(R"test(int * i = R"cpp(int * j = 0;)cpp";)test", Style));
 }
 
 TEST_F(FormatTestRawStrings, MatchesDelimitersCaseSensitively) {
   // Don't touch the 'PB' raw string, format the 'pb' raw string.
   expect_eq(R"test(
 s = R"PB(item:1)PB";

 }
 
 TEST_F(FormatTestRawStrings, DontFormatNonRawStrings) {
   expect_eq(R"test(a = R"pb(key:value)";)test",
             format(R"test(a = R"pb(key:value)";)test",
                    getRawStringPbStyleWithColumns(20)));
 }
 
+TEST_F(FormatTestRawStrings, FormatsRawStringsWithEnclosingFunctionName) {
+  FormatStyle Style = getRawStringPbStyleWithColumns(40);
+  Style.RawStringFormats[0].EnclosingFunctions.push_back(
+      "PARSE_TEXT_PROTO");
+  Style.RawStringFormats[0].EnclosingFunctions.push_back("ParseTextProto");
+  expect_eq(R"test(a = PARSE_TEXT_PROTO(R"(key: value)");)test",
+            format(R"test(a = PARSE_TEXT_PROTO(R"(key:value)");)test", Style));
+
+  expect_eq(R"test(
+a = PARSE_TEXT_PROTO /**/ (
+    /**/ R"(key: value)");)test",
+            format(R"test(
+a = PARSE_TEXT_PROTO/**/(/**/R"(key:value)");)test",
+                   Style));
+
+  expect_eq(R"test(
+a = ParseTextProto<ProtoType>(
+    R"(key: value)");)test",
+            format(R"test(
+a = ParseTextProto<ProtoType>(R"(key:value)");)test",
+                   Style));
+}
+
 } // end namespace
 } // end namespace format
 } // end namespace clang