[clang-format] Reorganize raw string delimiters

[krasimir]

Summary

OBSOLETE
This patch reorganizes the way raw string formats are handled:

• all styles defined in a .clang-format file per language are propagated to the formatting library for the purposes of formatting code blocks
• the raw string styles are organized around the language, with a list of delimiters and enclosing function names matching that language
• there is an optional canonical delimiter per language, which can be used to update other delimiters for that language to the canonical one.

[krasimir]

• Added unsafe canonical delimiter update handling

[krasimir]

• Added support for enclosing function names

[krasimir]

• Updated documentation

[djasper]

include/clang/Format/Format.h

1216

Lets find a way to implement without this in the public header file.

1375

Can you pull apart this patch?

In my view, it has three parts that have an ordering, but are actually fairly independent:

1. Propagate all configured languages to the formatting library. First patch to land, should not affect the visible behavior.
2. Restructure RawStringFormat to be centered around each language. This is a restructuring to make things easier and use #1.
3. Add a CanonicalDelimiter and make clang-format canonicalize it.

I'll focus my comments on what's required for #1 for now as that is already complicated (IMO).

lib/Format/Format.cpp

919

Prefer early exit, i.e.

if (!LanguageFound)
  return make_error_code(ParseError::Unsuitable);
...

920

I think this is getting a bit convoluted and I don't even understand whether we are doing what is document (even before this patch).

So in lines 892-905, we verify that:

• Only the first Style in the file is allowed be LK_None.
• No language is duplicated.

That seems good.

According to the documentation: "The first section may have no language set, it will set the default style options for all lanugages.". Does the latter part actually happen? Seems to me that we are just setting "Style" to the style configured for a specific language, completely ignoring values that might have been set in the LK_None style. Or is that somehow happening when reading the JSON?

Independent of that, I think we should use this structure more explicitly. I think we should create an additional class (FormatStyles or FormatStyleSet or something) that is returned by this function and handed to the formatting library. This function then doesn't need to look at the language anymore.

That class should then have a function getFormatStyle(LanguageKind Language); that returns the style for a particular language (doing the default logic, etc.). Internally, it can likely just have a map<LK, Style> and I don't think you need to pre-fill that for all language kinds. If a language kind is not in the map, you can just return what's stored for LK_None. WDYT?

[krasimir]

include/clang/Format/Format.h

1375

I believe these should all go together: the reason that we propagate all configured languages to the formatting library is to be able to use them as a replacement for the BasedOnStyle in RawStringFormat. To make this possible, we need to update the internal structure of RawStringFormat itself to base it around each language. The canonical delimiter part is just a convenience for this I guess, which could be split.

My biggest concern with (1) is that since it has no visible behavior and no other uses except for the adaptation of (2), it is not testable by itself and it's not evident that a patch doing just (1) would handle the things correctly.

[krasimir]

lib/Format/Format.cpp

920

Yes, defaulting to the None for missing language specifications is handled at line 912:

if (!LanguageFound && (Styles[i].Language == Language ||
                       Styles[i].Language == FormatStyle::LK_None

I was thinking of the FormatStyleSet approach but the problem is that this has repercussions all over the library. We could indeed update this specific function that way, but fundamentally the additional language styles are part of the FormatStyle and need to somehow be recorded inside there. That's why I went with KISS and directly made this function handle that case.

[krasimir]

• Address review comments

[djasper]

include/clang/Format/Format.h

1375

Ok, if you wish, this is not an unreasonable argument. But let's still do the code review in two steps: Lets for now just get the part of handling multiple languages straight and figure out the rest once we are sure that that part is fine.

(I do think you can test it, though - but it depends on whether I can convince you to go with the FormatStyleSet approach ;) )

lib/Format/Format.cpp

920

But it's not just defaulting to LK_None what we are saying we are implementing. I think the documentation suggestion that we implement very basic inheritance. E.g. if the style for LK_None set the ColumnLimit to 42, I would expect that the styles for all other languages that don't explicitly set a ColumnLimit would also use 42. I don't think this is currently implemented and I don't think this patch implements it. But I think we should :).

I agree that the FormatStyleSet approach would have some consequences, but I also think that it is much cleaner. Your current solution feels like we us working around technical debt and creating more technical debt to do it :(. Maybe Manuel has thoughts here?

[klimek]

include/clang/Format/Format.h

1375

On a philosophical level, something that has no visible behavior, and just restructures the code, should be tested by existing tests?

Enclosing function names also seems like an extra feature that could be pulled out, btw.

lib/Format/Format.cpp

920

I agree that we should test the inheritance if it's documented :)

I don't have super strong feelings where the logic of implementing that inheritance lives - both the function that parses the data and the data structure we hand around seem fine for that, as it's easy to change

That said, I do agree that FormatStyle being a recursive graph data structure is weird and unexpected; I had to ask Daniel what he actually meant here, and then went "how does this even work??!" before realizing that FormatStyle is both a single style and also containing all other styles.

-> in conclusion, I agree with Daniel: we should have a FormatStyleSet and pass that around everywhere. That's a pure refactoring that looks like it would already make the current code better, so I also agree it should be done first and be covered by existing tests.

docs/ClangFormatStyleOptions.rst

**BreakConstructorInitializers** (``BreakConstructorInitializersStyle``)
 
   Possible values:
 
   * ``BCIS_BeforeColon`` (in configuration: ``BeforeColon``)
     Break constructor initializers before the colon and after the commas.
 
     .. code-block:: c++
 
-      Constructor()
-          : initializer1(),
-            initializer2()
 
   * ``BCIS_BeforeComma`` (in configuration: ``BeforeComma``)
     Break constructor initializers before the colon and commas, and align
     the commas with the colon.
 
     .. code-block:: c++
 
-      Constructor()
-          : initializer1()
-          , initializer2()
 
   * ``BCIS_AfterColon`` (in configuration: ``AfterColon``)
     Break constructor initializers after the colon and commas.
 
     .. code-block:: c++
 
-      Constructor() :
-          initializer1(),
-          initializer2()
 
 
 
 **BreakStringLiterals** (``bool``)
   Allow breaking string literals when formatting.
 
 **ColumnLimit** (``unsigned``)
   The column limit.
.. code-block:: c++
 
        #include "b.h"               into      #include "a.h"
                                               #include "b.h"
        #include <lib/main.h>                  #include <lib/main.h>
        #include "a.h"
 
   * ``IBS_Regroup`` (in configuration: ``Regroup``)
     Merge multiple ``#include`` blocks together and sort as one.
-    Then split into groups based on category priority. See ``IncludeCategories``.
+    Then split into groups based on category priority. See
+    ``IncludeCategories``.
 
     .. code-block:: c++
 
        #include "b.h"               into      #include "a.h"
                                               #include "b.h"
        #include <lib/main.h>
        #include "a.h"                         #include <lib/main.h>
 
**Language** (``LanguageKind``)
 
   * ``LK_TableGen`` (in configuration: ``TableGen``)
     Should be used for TableGen code.
 
   * ``LK_TextProto`` (in configuration: ``TextProto``)
     Should be used for Protocol Buffer messages in text format
     (https://developers.google.com/protocol-buffers/).
 
+  * ``LK_End`` (in configuration: ``End``)
+    Do not use. Keep at last position.
+
 
 
 **MacroBlockBegin** (``std::string``)
   A regular expression matching macros that start a block.
 
   .. code-block:: c++
 
      # With:
* ``PAS_Middle`` (in configuration: ``Middle``)
 
     .. code-block:: c++
 
       int * a;
 
 
 
 **RawStringFormats** (``std::vector<RawStringFormat>``)
-  Raw string delimiters denoting that the raw string contents are
-  code in a particular language and can be reformatted.
+  Defines hints for detecting supported languages code blocks in
+  raw strings.
+
+  A raw string with a matching delimiter or a matching enclosing function
+  name will be reformatted assuming the specified language based on a
+  predefined style for that language defined in the .clang-format file.
+
+  A matching delimiter of a raw string takes precedence over a matching
+  enclosing function name for determining the language of the raw string
+  contents.
+
+  If a canonical delimiter is specified, occurences of other delimiters for
+  the same language will be updated to the canonical if possible.
 
-  A raw string with a matching delimiter will be reformatted assuming the
-  specified language based on a predefined style given by 'BasedOnStyle'.
-  If 'BasedOnStyle' is not found, the formatting is based on llvm style.
+  For a particular supported language at most one specification should
+  exist.
 
   To configure this in the .clang-format file, use:
 
   .. code-block:: yaml
 
     RawStringFormats:
-      - Delimiter: 'pb'
-        Language:  TextProto
-        BasedOnStyle: llvm
-      - Delimiter: 'proto'
-        Language:  TextProto
-        BasedOnStyle: google
+      - Language: TextProto
+          Delimiters:
+            - 'pb'
+            - 'proto'
+          EnclosingFunctionNames:
+            - 'PARSE_TEXT_PROTO'
+      - Language: Cpp
+          Delimiters:
+            - 'cc'
+            - 'cpp'
+          CanonicalDelimiter: 'cc'
 
 **ReflowComments** (``bool``)
   If ``true``, clang-format will attempt to re-flow comments.
 
   .. code-block:: c++
 
      false:
      // veryVeryVeryVeryVeryVeryVeryVeryVeryVeryVeryLongComment with plenty of information

include/clang/Format/Format.h

 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_FORMAT_FORMAT_H
 #define LLVM_CLANG_FORMAT_FORMAT_H
 
 #include "clang/Basic/LangOptions.h"
 #include "clang/Tooling/Core/Replacement.h"
 #include "llvm/ADT/ArrayRef.h"
+#include <array>
 #include <system_error>
 
 namespace clang {
 
 class Lexer;
 class SourceManager;
 class DiagnosticConsumer;
 
enum LanguageKind {
     LK_ObjC,
     /// Should be used for Protocol Buffers
     /// (https://developers.google.com/protocol-buffers/).
     LK_Proto,
     /// Should be used for TableGen code.
     LK_TableGen,
     /// Should be used for Protocol Buffer messages in text format
     /// (https://developers.google.com/protocol-buffers/).
-    LK_TextProto
+    LK_TextProto,
+    /// Do not use. Keep at last position.
+    LK_End,
   };
   bool isCpp() const { return Language == LK_Cpp || Language == LK_ObjC; }
 
   /// \brief Language, this format style is targeted at.
   LanguageKind Language;
 
   /// \brief A regular expression matching macros that start a block.
   /// \code
enum PointerAlignmentStyle {
     PAS_Middle
   };
 
   /// \brief Pointer and reference alignment style.
   PointerAlignmentStyle PointerAlignment;
 
   /// See documentation of ``RawStringFormats``.
   struct RawStringFormat {
-    /// \brief The delimiter that this raw string format matches.
-    std::string Delimiter;
-    /// \brief The language of this raw string.
+    /// \brief The language of this raw string format.
     LanguageKind Language;
-    /// \brief The style name on which this raw string format is based on.
-    /// If not specified, the raw string format is based on the style that this
-    /// format is based on.
-    std::string BasedOnStyle;
+    /// \brief A list of raw string delimiters that match this language.
+    std::vector<std::string> Delimiters;
+    /// \brief A list of enclosing function names that match this language.
+    std::vector<std::string> EnclosingFunctionNames;
+    /// \brief The canonical delimiter for this language.
+    std::string CanonicalDelimiter;
+
     bool operator==(const RawStringFormat &Other) const {
-      return Delimiter == Other.Delimiter && Language == Other.Language &&
-             BasedOnStyle == Other.BasedOnStyle;
+      return Language == Other.Language && Delimiters == Other.Delimiters &&
+             EnclosingFunctionNames == Other.EnclosingFunctionNames &&
+             CanonicalDelimiter == Other.CanonicalDelimiter;
     }
   };
 
-  /// \brief Raw string delimiters denoting that the raw string contents are
-  /// code in a particular language and can be reformatted.
+  /// \brief Defines hints for detecting supported languages code blocks in
+  /// raw strings.
+  ///
+  /// A raw string with a matching delimiter or a matching enclosing function
+  /// name will be reformatted assuming the specified language based on a
+  /// predefined style for that language defined in the .clang-format file.
+  ///
+  /// A matching delimiter of a raw string takes precedence over a matching
+  /// enclosing function name for determining the language of the raw string
+  /// contents.
+  ///
+  /// If a canonical delimiter is specified, occurences of other delimiters for
+  /// the same language will be updated to the canonical if possible.
   ///
-  /// A raw string with a matching delimiter will be reformatted assuming the
-  /// specified language based on a predefined style given by 'BasedOnStyle'.
-  /// If 'BasedOnStyle' is not found, the formatting is based on llvm style.
+  /// For a particular supported language at most one specification should
+  /// exist.
   ///
   /// To configure this in the .clang-format file, use:
   /// \code{.yaml}
   ///   RawStringFormats:
-  ///     - Delimiter: 'pb'
-  ///       Language:  TextProto
-  ///       BasedOnStyle: llvm
-  ///     - Delimiter: 'proto'
-  ///       Language:  TextProto
-  ///       BasedOnStyle: google
+  ///     - Language: TextProto
+  ///         Delimiters:
+  ///           - 'pb'
+  ///           - 'proto'
+  ///         EnclosingFunctionNames:
+  ///           - 'PARSE_TEXT_PROTO'
+  ///     - Language: Cpp
+  ///         Delimiters:
+  ///           - 'cc'
+  ///           - 'cpp'
+  ///         CanonicalDelimiter: 'cc'
   /// \endcode
   std::vector<RawStringFormat> RawStringFormats;
 
   /// \brief If ``true``, clang-format will attempt to re-flow comments.
   /// \code
   ///    false:
   ///    // veryVeryVeryVeryVeryVeryVeryVeryVeryVeryVeryLongComment with plenty of information
   ///    /* second veryVeryVeryVeryVeryVeryVeryVeryVeryVeryVeryLongComment with plenty of information */
return AccessModifierOffset == R.AccessModifierOffset &&
            SpacesInAngles == R.SpacesInAngles &&
            SpacesInContainerLiterals == R.SpacesInContainerLiterals &&
            SpacesInCStyleCastParentheses == R.SpacesInCStyleCastParentheses &&
            SpacesInParentheses == R.SpacesInParentheses &&
            SpacesInSquareBrackets == R.SpacesInSquareBrackets &&
            Standard == R.Standard && TabWidth == R.TabWidth &&
            UseTab == R.UseTab;
   }
-};
 
+  // A mapping from FormatStyle::LanguageKind to a format style to use for
+  // code blocks in that language.
+  std::array<std::shared_ptr<FormatStyle>, FormatStyle::LK_End>
+      AdditionalLanguageStyles;
+};
 /// \brief Returns a format style complying with the LLVM coding standards:
 /// http://llvm.org/docs/CodingStandards.html.
 FormatStyle getLLVMStyle();
 
 /// \brief Returns a format style complying with one of Google's style guides:
 /// http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml.
 /// http://google-styleguide.googlecode.com/svn/trunk/javascriptguide.xml.
 /// https://developers.google.com/protocol-buffers/docs/style.

lib/Format/ContinuationIndenter.h

 class BreakableToken;
 struct FormatToken;
 struct LineState;
 struct ParenState;
 struct RawStringFormatStyleManager;
 class WhitespaceManager;
 
 struct RawStringFormatStyleManager {
-  llvm::StringMap<FormatStyle> DelimiterStyle;
+  llvm::StringMap<const FormatStyle*> DelimiterStyle;
+  llvm::StringMap<const FormatStyle*> EnclosingFunctionNameStyle;
 
   RawStringFormatStyleManager(const FormatStyle &CodeStyle);
 
-  llvm::Optional<FormatStyle> get(StringRef Delimiter) const;
+  const FormatStyle *getDelimiterStyle(StringRef Delimiter) const;
+
+  const FormatStyle *
+  getEnclosingFunctionNameStyle(StringRef EnclosingFunctionName) const;
 };
 
 class ContinuationIndenter {
 public:
   /// \brief Constructs a \c ContinuationIndenter to format \p Line starting in
   /// column \p FirstIndent.
   ContinuationIndenter(const FormatStyle &Style,
                        const AdditionalKeywords &Keywords,

lib/Format/ContinuationIndenter.cpp

static llvm::Optional<StringRef> getRawStringDelimiter(StringRef TokenText) {
   size_t RParenPos = TokenText.size() - Delimiter.size() - 2;
   if (TokenText[RParenPos] != ')')
     return None;
   if (!TokenText.substr(RParenPos + 1).startswith(Delimiter))
     return None;
   return Delimiter;
 }
 
+static StringRef
+getCanonicalRawStringDelimiter(const FormatStyle &Style,
+                               FormatStyle::LanguageKind Language) {
+  for (const auto &Format : llvm::reverse(Style.RawStringFormats)) {
+    if (Format.Language == Language)
+      return StringRef(Format.CanonicalDelimiter);
+  }
+  return "";
+}
+
 RawStringFormatStyleManager::RawStringFormatStyleManager(
     const FormatStyle &CodeStyle) {
   for (const auto &RawStringFormat : CodeStyle.RawStringFormats) {
-    FormatStyle Style;
-    if (!getPredefinedStyle(RawStringFormat.BasedOnStyle,
-                            RawStringFormat.Language, &Style)) {
-      Style = getLLVMStyle();
-      Style.Language = RawStringFormat.Language;
+    for (StringRef Delimiter : RawStringFormat.Delimiters) {
+      DelimiterStyle.insert(
+          {Delimiter,
+           CodeStyle.AdditionalLanguageStyles[RawStringFormat.Language].get()});
+    }
+    for (StringRef EnclosingFunctionName :
+         RawStringFormat.EnclosingFunctionNames) {
+      EnclosingFunctionNameStyle.insert(
+          {EnclosingFunctionName,
+           CodeStyle.AdditionalLanguageStyles[RawStringFormat.Language].get()});
     }
-    Style.ColumnLimit = CodeStyle.ColumnLimit;
-    DelimiterStyle.insert({RawStringFormat.Delimiter, Style});
   }
 }
 
-llvm::Optional<FormatStyle>
-RawStringFormatStyleManager::get(StringRef Delimiter) const {
+const FormatStyle *
+RawStringFormatStyleManager::getDelimiterStyle(StringRef Delimiter) const {
   auto It = DelimiterStyle.find(Delimiter);
   if (It == DelimiterStyle.end())
-    return None;
+    return nullptr;
+  return It->second;
+}
+
+const FormatStyle *RawStringFormatStyleManager::getEnclosingFunctionNameStyle(
+    StringRef EnclosingFunctionName) const {
+  auto It = EnclosingFunctionNameStyle.find(EnclosingFunctionName);
+  if (It == EnclosingFunctionNameStyle.end())
+    return nullptr;
   return It->second;
 }
 
 ContinuationIndenter::ContinuationIndenter(const FormatStyle &Style,
                                            const AdditionalKeywords &Keywords,
                                            const SourceManager &SourceMgr,
                                            WhitespaceManager &Whitespaces,
                                            encoding::Encoding Encoding,
return encoding::columnWidthWithTabs(Text.substr(LastNewlinePos),
                                          /*StartColumn=*/0, TabWidth, Encoding);
   }
 }
 
 unsigned ContinuationIndenter::reformatRawStringLiteral(
     const FormatToken &Current, LineState &State,
     const FormatStyle &RawStringStyle, bool DryRun) {
   unsigned StartColumn = State.Column - Current.ColumnWidth;
-  auto Delimiter = *getRawStringDelimiter(Current.TokenText);
+  StringRef OldDelimiter = *getRawStringDelimiter(Current.TokenText);
+  StringRef NewDelimiter =
+      getCanonicalRawStringDelimiter(Style, RawStringStyle.Language);
+  if (NewDelimiter.empty() || OldDelimiter.empty())
+    NewDelimiter = OldDelimiter;
   // The text of a raw string is between the leading 'R"delimiter(' and the
   // trailing 'delimiter)"'.
-  unsigned PrefixSize = 3 + Delimiter.size();
-  unsigned SuffixSize = 2 + Delimiter.size();
+  unsigned OldPrefixSize = 3 + OldDelimiter.size();
+  unsigned OldSuffixSize = 2 + OldDelimiter.size();
+  std::string RawText =
+      Current.TokenText.substr(OldPrefixSize).drop_back(OldSuffixSize);
+  if (NewDelimiter != OldDelimiter) {
+    // Don't update to the canonical delimiter 'deli' if ')deli"' occurs in the
+    // raw string.
+    std::string CanonicalDelimiterSuffix = (")" + NewDelimiter + "\"").str();
+    if (StringRef(RawText).contains(CanonicalDelimiterSuffix))
+      NewDelimiter = OldDelimiter;
+  }
+
+  unsigned NewPrefixSize = 3 + NewDelimiter.size();
+  unsigned NewSuffixSize = 2 + NewDelimiter.size();
 
-  // The first start column is the column the raw text starts.
-  unsigned FirstStartColumn = StartColumn + PrefixSize;
+  // The first start column is the column the raw text starts after formatting.
+  unsigned FirstStartColumn = StartColumn + NewPrefixSize;
 
   // The next start column is the intended indentation a line break inside
   // the raw string at level 0. It is determined by the following rules:
   //   - if the content starts on newline, it is one level more than the current
   //     indent, and
   //   - if the content does not start on a newline, it is the first start
   //     column.
   // These rules have the advantage that the formatted content both does not
   // violate the rectangle rule and visually flows within the surrounding
   // source.
-  bool ContentStartsOnNewline = Current.TokenText[PrefixSize] == '\n';
+  bool ContentStartsOnNewline = Current.TokenText[OldPrefixSize] == '\n';
   unsigned NextStartColumn = ContentStartsOnNewline
                                  ? State.Stack.back().Indent + Style.IndentWidth
                                  : FirstStartColumn;
 
   // The last start column is the column the raw string suffix starts if it is
   // put on a newline.
   // The last start column is the intended indentation of the raw string postfix
   // if it is put on a newline. It is determined by the following rules:
   //   - if the raw string prefix starts on a newline, it is the column where
   //     that raw string prefix starts, and
   //   - if the raw string prefix does not start on a newline, it is the current
   //     indent.
   unsigned LastStartColumn = Current.NewlinesBefore
-                                 ? FirstStartColumn - PrefixSize
+                                 ? FirstStartColumn - NewPrefixSize
                                  : State.Stack.back().Indent;
 
-  std::string RawText =
-      Current.TokenText.substr(PrefixSize).drop_back(SuffixSize);
-
   std::pair<tooling::Replacements, unsigned> Fixes = internal::reformat(
       RawStringStyle, RawText, {tooling::Range(0, RawText.size())},
       FirstStartColumn, NextStartColumn, LastStartColumn, "<stdin>",
       /*Status=*/nullptr);
 
   auto NewCode = applyAllReplacements(RawText, Fixes.first);
   tooling::Replacements NoFixes;
   if (!NewCode) {
     State.Column += Current.ColumnWidth;
     return 0;
   }
   if (!DryRun) {
+    if (NewDelimiter != OldDelimiter) {
+      // In 'R"delimiter(...', the delimiter starts 2 characters after the start
+      // of the token.
+      SourceLocation PrefixDelimiterStart =
+          Current.Tok.getLocation().getLocWithOffset(2);
+      auto PrefixErr = Whitespaces.addReplacement(tooling::Replacement(
+          SourceMgr, PrefixDelimiterStart, OldDelimiter.size(), NewDelimiter));
+      if (PrefixErr) {
+        llvm::errs()
+            << "Failed to update the prefix delimiter of a raw string: "
+            << llvm::toString(std::move(PrefixErr)) << "\n";
+      }
+      // In 'R"delimiter(...)delimiter"', the suffix delimiter starts at
+      // position length - 1 - |delimiter|.
+      SourceLocation SuffixDelimiterStart =
+          Current.Tok.getLocation().getLocWithOffset(Current.TokenText.size() -
+                                                     1 - OldDelimiter.size());
+      auto SuffixErr = Whitespaces.addReplacement(tooling::Replacement(
+          SourceMgr, SuffixDelimiterStart, OldDelimiter.size(), NewDelimiter));
+      if (SuffixErr) {
+        llvm::errs()
+            << "Failed to update the suffix delimiter of a raw string: "
+            << llvm::toString(std::move(SuffixErr)) << "\n";
+      }
+    }
     SourceLocation OriginLoc =
-        Current.Tok.getLocation().getLocWithOffset(PrefixSize);
+        Current.Tok.getLocation().getLocWithOffset(OldPrefixSize);
     for (const tooling::Replacement &Fix : Fixes.first) {
       auto Err = Whitespaces.addReplacement(tooling::Replacement(
           SourceMgr, OriginLoc.getLocWithOffset(Fix.getOffset()),
           Fix.getLength(), Fix.getReplacementText()));
       if (Err) {
         llvm::errs() << "Failed to reformat raw string: "
                      << llvm::toString(std::move(Err)) << "\n";
       }
     }
   }
   unsigned RawLastLineEndColumn = getLastLineEndColumn(
       *NewCode, FirstStartColumn, Style.TabWidth, Encoding);
-  State.Column = RawLastLineEndColumn + SuffixSize;
+  State.Column = RawLastLineEndColumn + NewSuffixSize;
   return Fixes.second;
 }
 
 unsigned ContinuationIndenter::addMultilineToken(const FormatToken &Current,
                                                  LineState &State) {
   // Break before further function parameters on all levels.
   for (unsigned i = 0, e = State.Stack.size(); i != e; ++i)
     State.Stack[i].BreakBeforeParameter = true;
unsigned ContinuationIndenter::handleEndOfLine(const FormatToken &Current,
   }
   if (State.Column > getColumnLimit(State)) {
     unsigned ExcessCharacters = State.Column - getColumnLimit(State);
     Penalty += Style.PenaltyExcessCharacter * ExcessCharacters;
   }
   return Penalty;
 }
 
+static StringRef getEnclosingFunctionName(const FormatToken& Current) {
+  // Look for: 'function(' or 'function<templates>(' before Current.
+  auto Tok = Current.getPreviousNonComment();
+  if (!Tok || !Tok->is(tok::l_paren)) return "";
+  Tok = Tok->getPreviousNonComment();
+  if (!Tok) return "";
+  if (Tok->is(TT_TemplateCloser)) {
+    Tok = Tok->MatchingParen;
+    if (Tok)
+      Tok = Tok->getPreviousNonComment();
+  }
+  if (!Tok || !Tok->is(tok::identifier)) return "";
+  return Tok->TokenText;
+}
+
 llvm::Optional<FormatStyle>
 ContinuationIndenter::getRawStringStyle(const FormatToken &Current,
                                         const LineState &State) {
   if (!Current.isStringLiteral())
     return None;
   auto Delimiter = getRawStringDelimiter(Current.TokenText);
   if (!Delimiter)
     return None;
-  auto RawStringStyle = RawStringFormats.get(*Delimiter);
+  auto RawStringStyle = RawStringFormats.getDelimiterStyle(*Delimiter);
+  if (!RawStringStyle)
+    RawStringStyle = RawStringFormats.getEnclosingFunctionNameStyle(
+        getEnclosingFunctionName(Current));
   if (!RawStringStyle)
     return None;
-  RawStringStyle->ColumnLimit = getColumnLimit(State);
-  return RawStringStyle;
+
+  FormatStyle ResultStyle = *RawStringStyle;
+  ResultStyle.ColumnLimit = getColumnLimit(State);
+  return ResultStyle;
 }
 
 std::unique_ptr<BreakableToken> ContinuationIndenter::createBreakableToken(
     const FormatToken &Current, LineState &State, bool AllowBreak) {
   unsigned StartColumn = State.Column - Current.ColumnWidth;
   if (Current.isStringLiteral()) {
     // FIXME: String literal breaking is currently disabled for Java and JS, as
     // it requires strings to be merged using "+" which we don't support.

lib/Format/Format.cpp

 #include "clang/Lex/Lexer.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/Support/Allocator.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/Regex.h"
 #include "llvm/Support/YAMLTraits.h"
 #include <algorithm>
-#include <memory>
 #include <string>
 
 #define DEBUG_TYPE "format-formatter"
 
 using clang::format::FormatStyle;
 
 LLVM_YAML_IS_SEQUENCE_VECTOR(clang::format::FormatStyle::IncludeCategory)
 LLVM_YAML_IS_SEQUENCE_VECTOR(clang::format::FormatStyle::RawStringFormat)
static void enumeration(IO &IO, FormatStyle::IncludeBlocksStyle &Value) {
     IO.enumCase(Value, "Preserve", FormatStyle::IBS_Preserve);
     IO.enumCase(Value, "Merge", FormatStyle::IBS_Merge);
     IO.enumCase(Value, "Regroup", FormatStyle::IBS_Regroup);
   }
 };
 
 template <> struct MappingTraits<FormatStyle::RawStringFormat> {
   static void mapping(IO &IO, FormatStyle::RawStringFormat &Format) {
-    IO.mapOptional("Delimiter", Format.Delimiter);
     IO.mapOptional("Language", Format.Language);
-    IO.mapOptional("BasedOnStyle", Format.BasedOnStyle);
+    IO.mapOptional("Delimiters", Format.Delimiters);
+    IO.mapOptional("EnclosingFunctionNames", Format.EnclosingFunctionNames);
+    IO.mapOptional("CanonicalDelimiter", Format.CanonicalDelimiter);
   }
 };
 
 // Allows to read vector<FormatStyle> while keeping default values.
 // IO.getContext() should contain a pointer to the FormatStyle structure, that
 // will be used to get default values for missing keys.
 // If the first element has no Language specified, it will be treated as the
 // default one for the following elements.
FormatStyle getLLVMStyle() {
   LLVMStyle.NamespaceIndentation = FormatStyle::NI_None;
   LLVMStyle.ObjCBlockIndentWidth = 2;
   LLVMStyle.ObjCSpaceAfterProperty = false;
   LLVMStyle.ObjCSpaceBeforeProtocolList = true;
   LLVMStyle.PointerAlignment = FormatStyle::PAS_Right;
   LLVMStyle.SpacesBeforeTrailingComments = 1;
   LLVMStyle.Standard = FormatStyle::LS_Cpp11;
   LLVMStyle.UseTab = FormatStyle::UT_Never;
-  LLVMStyle.RawStringFormats = {{"pb", FormatStyle::LK_TextProto, "google"}};
   LLVMStyle.ReflowComments = true;
   LLVMStyle.SpacesInParentheses = false;
   LLVMStyle.SpacesInSquareBrackets = false;
   LLVMStyle.SpaceInEmptyParentheses = false;
   LLVMStyle.SpacesInContainerLiterals = true;
   LLVMStyle.SpacesInCStyleCastParentheses = false;
   LLVMStyle.SpaceAfterCStyleCast = false;
   LLVMStyle.SpaceAfterTemplateKeyword = true;
FormatStyle getGoogleStyle(FormatStyle::LanguageKind Language) {
   GoogleStyle.IncludeCategories = {
       {"^<ext/.*\\.h>", 2}, {"^<.*\\.h>", 1}, {"^<.*", 2}, {".*", 3}};
   GoogleStyle.IncludeIsMainRegex = "([-_](test|unittest))?$";
   GoogleStyle.IndentCaseLabels = true;
   GoogleStyle.KeepEmptyLinesAtTheStartOfBlocks = false;
   GoogleStyle.ObjCSpaceAfterProperty = false;
   GoogleStyle.ObjCSpaceBeforeProtocolList = false;
   GoogleStyle.PointerAlignment = FormatStyle::PAS_Left;
+  GoogleStyle.RawStringFormats = {
+      {FormatStyle::LK_TextProto,
+       /*Delimiters=*/
+       {
+           "pb",
+           "PB",
+           "proto",
+           "PROTO",
+           "textproto",
+           "TEXTPROTO",
+       },
+       /*EnclosingFunctionNames=*/
+       {
+           "EqualsProto",
+           "PARSE_TEXT_PROTO",
+           "ParseTextProto",
+       },
+       /*CanonicalDelimiter=*/""},
+  };
   GoogleStyle.SpacesBeforeTrailingComments = 2;
   GoogleStyle.Standard = FormatStyle::LS_Auto;
 
   GoogleStyle.PenaltyReturnTypeOnItsOwnLine = 200;
   GoogleStyle.PenaltyBreakBeforeFirstCallParameter = 1;
 
   if (Language == FormatStyle::LK_Java) {
     GoogleStyle.AlignAfterOpenBracket = FormatStyle::BAS_DontAlign;
for (unsigned j = 0; j < i; ++j) {
               << " and " << i << "\n");
         return make_error_code(ParseError::Error);
       }
     }
   }
   // Look for a suitable configuration starting from the end, so we can
   // find the configuration for the specific language first, and the default
   // configuration (which can only be at slot 0) after it.
+  bool LanguageFound = false;
   for (int i = Styles.size() - 1; i >= 0; --i) {
-    if (Styles[i].Language == Language ||
-        Styles[i].Language == FormatStyle::LK_None) {
+    if (!LanguageFound && (Styles[i].Language == Language ||
+                           Styles[i].Language == FormatStyle::LK_None)) {
       *Style = Styles[i];
       Style->Language = Language;
-      return make_error_code(ParseError::Success);
+      LanguageFound = true;
+      break;
+    }
+  }
+  if (LanguageFound) {
+    for (int i = Styles.size() - 1; i >= 0; --i) {
+      if (Styles[i].Language == FormatStyle::LK_None) {
+        for (unsigned AdditionalLanguage = 0;
+             AdditionalLanguage < FormatStyle::LK_End; ++AdditionalLanguage) {
+          FormatStyle AdditionalLanguageStyle = Styles[i];
+          AdditionalLanguageStyle.Language =
+              static_cast<FormatStyle::LanguageKind>(AdditionalLanguage);
+          if (!Style->AdditionalLanguageStyles[AdditionalLanguage])
+            Style->AdditionalLanguageStyles[AdditionalLanguage].reset(
+                new FormatStyle(AdditionalLanguageStyle));
+        }
+        continue;
       }
+      Style->AdditionalLanguageStyles[Styles[i].Language].reset(
+          new FormatStyle(Styles[i]));
+    }
+    return make_error_code(ParseError::Success);
   }
   return make_error_code(ParseError::Unsuitable);
 }
 
 std::string configurationAsText(const FormatStyle &Style) {
   std::string Text;
   llvm::raw_string_ostream Stream(Text);
   llvm::yaml::Output Output(Stream);

unittests/Format/FormatTest.cpp

CHECK_PARSE("IncludeCategories:\n"
               "    Priority: 2\n"
               "  - Regex: .*\n"
               "    Priority: 1",
               IncludeCategories, ExpectedCategories);
   CHECK_PARSE("IncludeIsMainRegex: 'abc$'", IncludeIsMainRegex, "abc$");
 
   Style.RawStringFormats.clear();
   std::vector<FormatStyle::RawStringFormat> ExpectedRawStringFormats = {
-      {"pb", FormatStyle::LK_TextProto, "llvm"},
-      {"cpp", FormatStyle::LK_Cpp, "google"}};
+      {FormatStyle::LK_TextProto,
+       {"pb", "proto"},
+       {"PARSE_TEXT_PROTO"},
+       "textproto"},
+      {FormatStyle::LK_Cpp, {"cc", "cpp"}, {"C_CODEBLOCK", "CPPEVAL"}, ""}};
 
   CHECK_PARSE("RawStringFormats:\n"
-              "  - Delimiter: 'pb'\n"
-              "    Language: TextProto\n"
-              "    BasedOnStyle: llvm\n"
-              "  - Delimiter: 'cpp'\n"
-              "    Language: Cpp\n"
-              "    BasedOnStyle: google",
+              "  - Language: TextProto\n"
+              "    Delimiters:\n"
+              "      - 'pb'\n"
+              "      - 'proto'\n"
+              "    EnclosingFunctionNames:\n"
+              "      - 'PARSE_TEXT_PROTO'\n"
+              "    CanonicalDelimiter: 'textproto'\n"
+              "  - Language: Cpp\n"
+              "    Delimiters:\n"
+              "      - 'cc'\n"
+              "      - 'cpp'\n"
+              "    EnclosingFunctionNames:\n"
+              "      - 'C_CODEBLOCK'\n"
+              "      - 'CPPEVAL'",
               RawStringFormats, ExpectedRawStringFormats);
 }
 
 TEST_F(FormatTest, ParsesConfigurationWithLanguages) {
   FormatStyle Style = {};
   Style.Language = FormatStyle::LK_Cpp;
   CHECK_PARSE("Language: Cpp\n"
               "IndentWidth: 12",

unittests/Format/FormatTestRawStrings.cpp

FormatStyle getLLVMStyleWithColumns(unsigned ColumnLimit) {
     return getStyleWithColumns(getLLVMStyle(), ColumnLimit);
   }
 
   int ReplacementCount;
 
   FormatStyle getRawStringPbStyleWithColumns(unsigned ColumnLimit) {
     FormatStyle Style = getLLVMStyle();
     Style.ColumnLimit = ColumnLimit;
-    Style.RawStringFormats = {{/*Delimiter=*/"pb",
-                               /*Kind=*/FormatStyle::LK_TextProto,
-                               /*BasedOnStyle=*/"google"}};
+    Style.AdditionalLanguageStyles[FormatStyle::LK_TextProto] =
+        std::make_shared<FormatStyle>(
+            getGoogleStyle(FormatStyle::LK_TextProto));
+    Style.RawStringFormats = {{/*Language=*/FormatStyle::LK_TextProto,
+                               /*Delimiters=*/{"pb"},
+                               /*EnclosingFunctionNames=*/{},
+                               /*CanonicalDelimiter=*/""}};
     return Style;
   }
 
-  FormatStyle getRawStringLLVMCppStyleBasedOn(std::string BasedOnStyle) {
+  FormatStyle getRawStringLLVMCppStyleBasedOn(std::string Name) {
     FormatStyle Style = getLLVMStyle();
-    Style.RawStringFormats = {{/*Delimiter=*/"cpp",
-                               /*Kind=*/FormatStyle::LK_Cpp, BasedOnStyle}};
+    FormatStyle BasedOnStyle = getLLVMStyle();
+    getPredefinedStyle(Name, FormatStyle::LK_Cpp, &BasedOnStyle);
+    Style.AdditionalLanguageStyles[FormatStyle::LK_Cpp] =
+        std::make_shared<FormatStyle>(BasedOnStyle);
+    Style.RawStringFormats = {{/*Language=*/FormatStyle::LK_Cpp,
+                               /*Delimiters=*/{"cpp"},
+                               /*EnclosingFunctionNames=*/{},
+                               /*CanonicalDelimiter=*/""}};
     return Style;
   }
 
-  FormatStyle getRawStringGoogleCppStyleBasedOn(std::string BasedOnStyle) {
+  FormatStyle getRawStringGoogleCppStyleBasedOn(std::string Name) {
     FormatStyle Style = getGoogleStyle(FormatStyle::LK_Cpp);
-    Style.RawStringFormats = {{/*Delimiter=*/"cpp",
-                               /*Kind=*/FormatStyle::LK_Cpp, BasedOnStyle}};
+    FormatStyle BasedOnStyle = getLLVMStyle();
+    getPredefinedStyle(Name, FormatStyle::LK_Cpp, &BasedOnStyle);
+    Style.AdditionalLanguageStyles[FormatStyle::LK_Cpp] =
+        std::make_shared<FormatStyle>(BasedOnStyle);
+    Style.RawStringFormats = {{/*Language=*/FormatStyle::LK_Cpp,
+                               /*Delimiters=*/{"cpp"},
+                               /*EnclosingFunctionNames=*/{},
+                               /*CanonicalDelimiter=*/""}};
     return Style;
   }
 
   // Gcc 4.8 doesn't support raw string literals in macros, which breaks some
   // build bots. We use this function instead.
   void expect_eq(const std::string Expected, const std::string Actual) {
     EXPECT_EQ(Expected, Actual);
   }
 };
 
 TEST_F(FormatTestRawStrings, ReformatsAccordingToBaseStyle) {
   // llvm style puts '*' on the right.
   // google style puts '*' on the left.
 
-  // Use the llvm style if the raw string style has no BasedOnStyle.
-  expect_eq(R"test(int *i = R"cpp(int *p = nullptr;)cpp")test",
+  // Use llvm style outside and the google style inside if the raw string style
+  // is based on google.
+  expect_eq(R"test(int *i = R"cpp(int* p = nullptr;)cpp")test",
             format(R"test(int * i = R"cpp(int * p = nullptr;)cpp")test",
-                   getRawStringLLVMCppStyleBasedOn("")));
+                   getRawStringLLVMCppStyleBasedOn("google")));
 
-  // Use the google style if the raw string style has BasedOnStyle=google.
-  expect_eq(R"test(int *i = R"cpp(int* p = nullptr;)cpp")test",
+  // Use llvm style if the raw string style has no BasedOnStyle.
+  expect_eq(R"test(int *i = R"cpp(int *p = nullptr;)cpp")test",
             format(R"test(int * i = R"cpp(int * p = nullptr;)cpp")test",
-                   getRawStringLLVMCppStyleBasedOn("google")));
+                   getRawStringLLVMCppStyleBasedOn("")));
 
-  // Use the llvm style if the raw string style has no BasedOnStyle=llvm.
+  // Use google style outside and the llvm style inside if the raw string style
+  // is based on llvm.
   expect_eq(R"test(int* i = R"cpp(int *p = nullptr;)cpp")test",
             format(R"test(int * i = R"cpp(int * p = nullptr;)cpp")test",
                    getRawStringGoogleCppStyleBasedOn("llvm")));
 }
 
 TEST_F(FormatTestRawStrings, MatchesDelimitersCaseSensitively) {
   // Don't touch the 'PB' raw string, format the 'pb' raw string.
   expect_eq(R"test(
 s = R"PB(item:1)PB";
 t = R"pb(item: 1)pb";)test",
             format(R"test(
 s = R"PB(item:1)PB";
 t = R"pb(item:1)pb";)test",
                    getRawStringPbStyleWithColumns(40)));
-
-  FormatStyle MixedStyle = getLLVMStyle();
-  MixedStyle.RawStringFormats = {
-      {/*Delimiter=*/"cpp", /*Kind=*/FormatStyle::LK_Cpp,
-       /*BasedOnStyle=*/"llvm"},
-      {/*Delimiter=*/"CPP", /*Kind=*/FormatStyle::LK_Cpp,
-       /*BasedOnStyle=*/"google"}};
-
-  // Format the 'cpp' raw string with '*' on the right.
-  // Format the 'CPP' raw string with '*' on the left.
-  // Do not format the 'Cpp' raw string.
-  // Do not format non-raw strings.
-  expect_eq(R"test(
-a = R"cpp(int *i = 0;)cpp";
-b = R"CPP(int* j = 0;)CPP";
-c = R"Cpp(int * k = 0;)Cpp";
-d = R"cpp(int * k = 0;)Cpp";)test",
-            format(R"test(
-a = R"cpp(int * i = 0;)cpp";
-b = R"CPP(int * j = 0;)CPP";
-c = R"Cpp(int * k = 0;)Cpp";
-d = R"cpp(int * k = 0;)Cpp";)test",
-                   MixedStyle));
 }
 
 TEST_F(FormatTestRawStrings, ReformatsShortRawStringsOnSingleLine) {
   expect_eq(
       R"test(P p = TP(R"pb()pb");)test",
       format(
           R"test(P p = TP(R"pb( )pb");)test",
           getRawStringPbStyleWithColumns(40)));
TPPPPPPPPPPPPPPP(
             format(R"test(
 P p = TPPPPPPPPPPPPPPP(R"pb(item_1: 1, item_2: 2, item_3: 3)pb");)test",
                    getRawStringPbStyleWithColumns(40)));
 
   expect_eq(R"test(
 P p = TP(R"pb(item_1 <1>
               item_2: <2>
               item_3 {})pb");)test",
             format(R"test(
 P p = TP(R"pb(item_1<1> item_2:<2> item_3{ })pb");)test",
                    getRawStringPbStyleWithColumns(40)));
 
   expect_eq(
       R"test(
 P p = TP(R"pb(item_1: 1,
               item_2: 2,
               item_3: 3,
               item_4: 4)pb");)test",
       format(
ASSERT_TRUE(ParseFromString(R"pb(item_1: 1, item_2: 2)pb"), ptr);)test",
   expect_eq(R"test(
 ASSERT_TRUE(
     ParseFromString(
         R"pb(item_1: 1 item_2: 2)pb"),
     ptr);)test",
             format(R"test(
 ASSERT_TRUE(ParseFromString(R"pb(item_1: 1 item_2: 2)pb"), ptr);)test",
                    getRawStringPbStyleWithColumns(40)));
-
 }
 
 TEST_F(FormatTestRawStrings, RawStringsInOperands) {
   // Formats the raw string first operand of a binary operator expression.
   expect_eq(R"test(auto S = R"pb(item_1: 1)pb" + rest;)test",
             format(R"test(auto S = R"pb(item_1:1)pb" + rest;)test",
                    getRawStringPbStyleWithColumns(40)));
 
auto S =
     (count < 3)
         ? R"pb(item_1: 1)pb"
         : R"pb(item_2: 2, item_3: 3)pb";
 )test",
             format(R"test(
 auto S=(count<3)?R"pb(item_1:1)pb":R"pb(item_2:2,item_3:3)pb";
 )test",
                    getRawStringPbStyleWithColumns(40)));
-
 }
 
 TEST_F(FormatTestRawStrings, PrefixAndSuffixAlignment) {
   // Keep the suffix at the end of line if not on newline.
   expect_eq(R"test(
 int s() {
   auto S = PTP(
       R"pb(

 }
 
 TEST_F(FormatTestRawStrings, DontFormatNonRawStrings) {
   expect_eq(R"test(a = R"pb(key:value)";)test",
             format(R"test(a = R"pb(key:value)";)test",
                    getRawStringPbStyleWithColumns(20)));
 }
 
+TEST_F(FormatTestRawStrings, UpdatesToCanonicalDelimiters) {
+  FormatStyle Style = getRawStringPbStyleWithColumns(25);
+  Style.RawStringFormats[0].CanonicalDelimiter = "proto";
+  expect_eq(R"test(a = R"proto(key: value)proto";)test",
+            format(R"test(a = R"pb(key:value)pb";)test", Style));
+
+  // Don't update to canonical delimiter if it occurs as a raw string suffix in
+  // the raw string content.
+  expect_eq(R"test(a = R"pb(key: ")proto")pb";)test",
+            format(R"test(a = R"pb(key:")proto")pb";)test", Style));
+}
+
+TEST_F(FormatTestRawStrings, FormatsRawStringsWithEnclosingFunctionName) {
+  FormatStyle Style = getRawStringPbStyleWithColumns(40);
+  Style.RawStringFormats[0].EnclosingFunctionNames.push_back(
+      "PARSE_TEXT_PROTO");
+  Style.RawStringFormats[0].EnclosingFunctionNames.push_back("ParseTextProto");
+  expect_eq(R"test(a = PARSE_TEXT_PROTO(R"(key: value)");)test",
+            format(R"test(a = PARSE_TEXT_PROTO(R"(key:value)");)test", Style));
+
+  expect_eq(R"test(
+a = PARSE_TEXT_PROTO /**/ (
+    /**/ R"(key: value)");)test",
+            format(R"test(
+a = PARSE_TEXT_PROTO/**/(/**/R"(key:value)");)test",
+                   Style));
+
+  expect_eq(R"test(
+a = ParseTextProto<ProtoType>(
+    R"(key: value)");)test",
+            format(R"test(
+a = ParseTextProto<ProtoType>(R"(key:value)");)test",
+                   Style));
+}
+
 } // end namespace
 } // end namespace format
 } // end namespace clang