[clang-format] Format raw string literals

[krasimir]

Summary

This patch adds raw string literal formatting.

It's still work in progress, however feedback is welcome.

[klimek]

lib/Format/ContinuationIndenter.cpp

83

Why 19?

1279–1282

Why 3? Generally, when you use magic values other than 0, -1, 1, comment on what exactly they mean.

lib/Format/ContinuationIndenter.h

43

Why getGoogleStyle?

[krasimir]

• Fixed formatting

[krasimir]

• Add comments on magic

[krasimir]

• Add tests for delimiters and make delimiter search case-insensitive

[krasimir]

• Add tests for raw strings in arguments

[krasimir]

• Make reformatRawStringLiteral return the new state column
• Add FormattedRawString cache

[krasimir]

• Add BasedOnStyle to RawStringFormat

[krasimir]

lib/Format/ContinuationIndenter.h

43

Added BasedOnStyle to the RawStringStyle type, now it's not arbitrary anymore. Thank you!

[krasimir]

• Add tests for case-sensitive delimiter matching

[krasimir]

• Add tests for raw strings starting on own line

[krasimir]

• Add tests for breaks before raw strings

[krasimir]

• Add tests for raw string operands

[krasimir]

• Add more tests for raw string operands

[krasimir]

• Add LastStartColumn and prefix and postfix raw string alignment

[krasimir]

• Adjust the indent of raw string suffixes

[krasimir]

• Add a test demonstrating exceeding the column limit

[krasimir]

• Propagate estimated raw string penalty to the reflow optimization

[krasimir]

• Rebase with master

[krasimir]

I think this is ready for a round of reviews.

[klimek]

lib/Format/ContinuationIndenter.cpp

83

If it == 5, it''s also trivially empty. It can't really be 6 (as it needs to be repeated), so only 5 + 2n are valid sizes? So < 7 would work here?

95

After tokenization, can't we assume that's true?

138

Why do we need a new parameter to pass through instead of using FirstIndent?

1006

I'd have expected the raw string formatting to happen here, as breakProtrudingToken is very similar (semantically) to formatting raw strings.

1018

Add a comment why this happens (-> reading this, I assume the State.Column is adapted in the call to reformatRawStringLiteral).

1277–1280

Why do we need to estimate the penalty, as opposed to getting it from the recursive formatting step?

1316–1317

Why do we need an extra cache and can't let the State take care of this?

lib/Format/ContinuationIndenter.h

167

"A key in for" doesn't parse.

[krasimir]

lib/Format/ContinuationIndenter.cpp

83

The delimiter could be the empty string. Added a comment for that.

95

No. The tokenizer is pretty liberal and will recognize R"pb(text)" as a string literal, for example (this is because of some handling of the strings occurring as header names). Added a test for that.

[krasimir]

lib/Format/ContinuationIndenter.cpp

138

There are two parameters in State that are initialized: State.FirstIndent, which is always initialized as FirstIndent and Stare.Column, which is initialized appropriately.

The raw string formatting introduces the ability to format a piece of code as if it's first line started at FirstStartColumn, and this new parameter needs to be propagated appropriately where it matters.

1006

I explored this approach initially. The problem is that breakProtrudingToken internally assumes that the token has been put on the state (the protruding part) and reformatting conceptually takes place before that.

1277–1280

That would be the other option. I went with the simplest approach.
Here's some problems with getting the penalty from the recursive step:

1. This breaks the abstraction of reformat. Conceptually, the penalty stuff is an implementation detail of the Optimizing Formatter. For other implementations of Formatter-s, we don't have a meaningful penalty.
2. The penalty is computed in the context of a different FormatStyle, so it might be incompatible with the way the penalties are computed in the current context.

1316–1317

How can the State take care of this? We want to save the edits to the raw string text so that we don't call reformat again in non-dry-run-mode. Conceptually, there might be different State-s having the same FormattedRawStringCacheKey, where the edits could be reused.

[krasimir]

• Address review comments

[krasimir]

• Update comments

[klimek]

lib/Format/ContinuationIndenter.cpp

95

I mean, that's a string literal, just not a raw string literal :) I'm wondering why the tokanizer doesn't have to make that distinction. Anyway, if it can happen, of course this looks fine.

138

I thought that's what FirstIndent means? I'm still really confused :)

1006

Why? I guess my question is: Where do the side effects of breakProtrudingToken and reformatRawStringLiteral differ?

1277–1280

(2) is an interesting argument. I don't see why (1) holds, though, as we're already doing recursive formatting for child blocks within statements. I'd like to get Daniel's opinion on this point.

1316–1317

What's the performance difference there?

[djasper]

lib/Format/ContinuationIndenter.cpp

1277–1280

Ok, as for the function itself: It needs a better description of what the estimation does and why.

For the points:

1. The optimizing formatter can compute the penalty. If the NoLineBreakFormatter is used, then the penalty should be 0 (there should be no line break or column limit violation). The NoColumnLimitFormatter doesn't care about penalties at all, so we can also assume 0.

2. In what way can it be incompatible?

[krasimir]

lib/Format/ContinuationIndenter.cpp

138

FirstIndent is the expected indent of the first token of the unwrapped line, which gets propagated, for example, in line breaks. The formatter itself controls it.
FirstStartColumn is where this line starts. The formatter doesn't control it.

Consider:

void f() {
  auto s = R"pb(key1: value1,
      key2: value2,
      key3: value3)pb";
}

To reformat the raw string literal, we need to take into account both the column where key1 starts (that is FirstStartColumn is 16) and the indent we'd like the first unwrapped line to have (that is FirstIndent is 6).

Maybe we need to rename these and add comments about them?

1006

You're right. This belongs to breakProtrudingToken. I'll move this to an implementation of BreakableToken.

1277–1280

I have another point about switching to use the recursive formatting of the penalties:

3. The penalty for the last line of a multiline token is handled at a later stage. If we use the penalty recursively, that will include the penalty for the last line, so we'll need to somehow propagate to the later stage that we don't wanna handle the penalty for the last line there.

As an example for (2) penalty incompatibility, you could use a style inside the raw strings that has different PenaltyExcessCharacter, for example.

Also, consider another possible use-case of formatting SQL raw string literals, for example. clang-format doesn't understand how to format these and could possibly only touch the leading indent of the lines. In that case, the penalty estimation would work.

1316–1317

Without the cache, the edits to each raw string token are computed at least twice: once in DryRun mode, once in wet mode.
So, I can come up with an example where this cache makes formatting twice faster.

Switching the implementation to a BreakableToken would eliminate the need of the cache, since the edits will be stored in the BreakableToken itself.

[krasimir]

• Move reformatRawStringLiteral to breakProtrudingToken

[krasimir]

lib/Format/ContinuationIndenter.cpp

1316–1317

I was wrong. I moved the implementation to breakProtrudingToken, but that doesn't eliminate the need for the cache.

[krasimir]

• Document estimatedPenalty

[krasimir]

• Fix formatting around a newly added block

[krasimir]

lib/Format/ContinuationIndenter.cpp

1277–1280

Added a description for estimatedPenalty.

1314

@djasper, @klimek: what's your opinion about this decision? It seems logical, but most of the examples I see in the wild are formatted more like State.FirstIndent + Style.IndentWidth.

[klimek]

lib/Format/ContinuationIndenter.cpp

138

The same thing is true for blocks inside expressions, though - what's the difference?

1006

Wait wait :) I'm not arguing it needs to be using the same implementation. I'm just saying it should be at the same spot in the code.

1316–1317

What's the performance difference without the cache?

[krasimir]

lib/Format/ContinuationIndenter.cpp

138

I don't know how blocks inside expressions are handled. Could you please give me an example or point me to that code?

1316–1317

Without the cache, the edits to each raw string token are computed at least twice: once in DryRun mode, once in wet mode.
So, I can come up with an example where this cache makes formatting twice faster.

[klimek]

lib/Format/ContinuationIndenter.cpp

138

void f() {
  otherfunc([] {
    innerStatement();
    auto letsgo = [] {
      deeper();
    };
  });
}

[krasimir]

lib/Format/ContinuationIndenter.cpp

138

Thank you! Here's how formatting of blocks inside expressions works now:
Consider this formatted code (with column limit 20):

a = [] {
  bbbb = 1;
};

This is parsed as two unwrapped lines, one like a = [] {}; and a child line attached to the { token like bbbb = 1;. Since these are two independent lines, they have two different values of FirstIndent: 0 and 2, respectively. Also, since these are on two separate unwrapped lines, the formatting of the { is independent of the formatting inside (except for the first indent that gets propagated through).

The situation with raw string literals is a bit different. Consider the following formatted code (with column limit 20):

auto s = R"pb(key1: value1,
    key2: value2,
    key3: value3)pb";

This is parsed as a single unwrapped line, like auto s = R"pb(...)pb"; with a single value of FirstIndent: 0. We need to somehow propagate the information about the additional indent that a line break at line level 0 inside the raw string literal would get, and that's how we end up with an additional parameter. From the point of view of the surrounding formatting, the formatting of the raw string literal is a black box returning replacements (and possibly additional penalty). So the surrounding formatting code needs to inform the raw string formatting code about the context in which the formatting occurs, which is achieved by passing the newly introduced FirstStartColumn, NextStartColumn and LastStartColumn parameters.

[klimek]

lib/Format/ContinuationIndenter.cpp

138

I'm wondering whether the inner indent should come from the style of the inner formatter. I'm not sure yet. I think if it shouldn't, that's a good argument for passing it through (and we need to document that decision).

1277–1280

As long as the penalty is applied consistently for every different possibility, and just adds up, won't it still be correct (in selecting the best solution), even if we have totally different excess char penalties? Can you give an example where that's not true?

1316–1317

I understand that without the cache it's going to be evaluated twice. We do that in lots of cases, though. Which is why I'm curious how much difference in runtime this actually makes (we usually dry-run a bazillion of states and then apply only one, so the dry- vs. non-dry-run runtime usually doesn't matter).

[krasimir]

lib/Format/ContinuationIndenter.cpp

138

The inner indent already comes from the style of the inner formatter. The offset where the inner column 0 starts, however, is controlled by the outer formatter.
Consider this code:

string s = R"pb(item: 1,
    msg_field {
      key: value
    })pb";

void f() {
  string s = R"pb(item: 1,
      msg_field {
        key: value
      })pb";
}

The 4 column relative indent of msg_field is the ContinuationIndentWidth of the outer format style.
The additional 2 column relative indent of key comes from the IndentWidth of the inner format style.
The outer formatter sees this code as:

string s = R"pb(<...stuff that the inner formatter will take care of>
####<...stuff that the inner formatter will take care of>
####<...stuff that the inner formatter will take care of>
####<...stuff that the inner formatter will take care of>)pb";

void f() {
  string s = R"pb(<...stuff that the inner formatter will take care of>
######<...stuff that the inner formatter will take care of>
######<...stuff that the inner formatter will take care of>
######<...stuff that the inner formatter will take care of>
######})pb";
}

where # signifies the offset where column 0 starts in the raw strings.
I'll refine the information about this new option with a set of rules that roughly determine the indentation decisions taken.

1277–1280

Good point. Thank you!
I can't come up with an example.

1316–1317

Do you have any ideas on how to gather some meaningful measurements on the runtime impact of this patch? I'm thinking of running it over a bunch of real-world source code that does proto processing and noting the runtime.

[klimek]

lib/Format/ContinuationIndenter.cpp

1316–1317

Yep, that's exactly what I'd expect. Also try to find some "worst case" scenarios, where you think you have an example that would benefit the most from the caching and measure that - that often gives a good upper bound.

[krasimir]

lib/Format/ContinuationIndenter.cpp

1316–1317

The cache adds ~4% performance improvement on a bunch of real-world examples.

[krasimir]

• Get the penalty from the recursive invocation

[krasimir]

• Update the section in moveStateToNextToken to match the original

[krasimir]

• Updated indent to IndentWidth
• Revised the rules for formatting the prefix and suffix

[krasimir]

I think this is ready for another round of reviews.

[klimek]

lib/Format/ContinuationIndenter.cpp

138

Ok, so the problem is that if we want to start an inner formatter, FirstStartColumn will be the column at which we currently are at the state of the parent formatter?
(If that's the case, please document it :)

1316–1317

I'm not sure whether that's worth it, but I'm happy to let Daniel tip the scale.

[krasimir]

lib/Format/ContinuationIndenter.cpp

1316–1317

I believe that the value of the cache is that it shows explicitly and mechanically the exact state that is propagated to the inner formatter. As such, I would argue that the cache increases maintainability.

[krasimir]

• Rebase with master

[djasper]

lib/Format/ContinuationIndenter.cpp

1316–1317

I agree that the performance gain aren't worth the additional effort here. We would need a cache to prevent computing the same breaks in various paths of the combinatorial formatting, but I haven't studied the code enough to know whether we have this separately yet.

On maintainability:
We similarly don't do this in other places (breakProtrudingToken, formatChildren, etc.) and so I think doing this differently for raw string literals will make things less maintainable. Moreover, I see your point that caching makes it in some way easier to see that we are doing the exact same thing in the second phase. At the same time ensuring that caching is correct might be hard to maintain. E.g. finding bugs that are caused by us storing incorrect information in the cache or something might be extraordinarily hard to track down.

[krasimir]

• Fix a regression after preprocessor indentation

[krasimir]

• Remove raw string cache

[krasimir]

• Document FirstStateColumn @ getInitialState

[krasimir]

I think this is ready for another round of reviews.

[krasimir]

ping

[krasimir]

ping

[klimek]

Needs a bit more comments, otherwise LG.

include/clang/Format/Format.h

1689–1693

Needs a comment. Also, this should probably go into an internal namespace somewhere below all the public interfaces?

lib/Format/ContinuationIndenter.cpp

1325

No else after return.

lib/Format/TokenAnalyzer.h

71–75

I think you'll want to document these.

[krasimir]

• Address review comments

[krasimir]

include/clang/Format/Format.h

1689–1693

Good point. Moved to a new internal header FormatInternal.h and added documentation.

[krasimir]

I'll need a bit of refactoring on the test side of things, since old gcc doesn't like raw string literals in macros.

[klimek]

lib/Format/FormatInternal.h

27–28 ↗

(On Diff #120481)

"and at which column the fragment's last line should end".

I don't understand the "if it is a newline" part.

29–30 ↗

(On Diff #120481)

"specifies the column at which the first line of \p Code starts".

33–34 ↗

(On Diff #120481)

"the column at which the last line of \p Code should end".

Also, s/it/if/. Does "just a newline" mean "is empty"?

[krasimir]

• Improve comments

[krasimir]

• Refactor tests to use a function instead of a macro

[krasimir]

• Update example

include/clang/Format/Format.h

enum PointerAlignmentStyle {
     ///   int * a;
     /// \endcode
     PAS_Middle
   };
 
   /// \brief Pointer and reference alignment style.
   PointerAlignmentStyle PointerAlignment;
 
+  /// See documentation of ``RawStringFormats``.
+  struct RawStringFormat {
+    /// \brief The delimiter that this raw string format matches.
+    std::string Delimiter;
+    /// \brief The language of this raw string.
+    LanguageKind Language;
+    /// \brief The style name on which this raw string format is based on.
+    /// If not specified, the raw string format is based on the style that this
+    /// format is based on.
+    std::string BasedOnStyle;
+    bool operator==(const RawStringFormat &Other) const {
+      return Delimiter == Other.Delimiter && Language == Other.Language &&
+             BasedOnStyle == Other.BasedOnStyle;
+    }
+  };
+
+  /// \brief Raw string delimiters denoting that the raw string contents are
+  /// code in a particular language and can be reformatted.
+  ///
+  /// A raw string with a matching delimiter will be reformatted assuming the
+  /// specified language based on a predefined style given by 'BasedOnStyle'.
+  /// If 'BasedOnStyle' is not found, the formatting is based on llvm style.
+  ///
+  /// To configure this in the .clang-format file, use:
+  /// \code{.yaml}
+  ///   RawStringFormats:
+  ///     - Delimiter: 'pb'
+  ///       Language:  TextProto
+  ///       BasedOnStyle: llvm
+  ///     - Delimiter: 'proto'
+  ///       Language:  TextProto
+  ///       BasedOnStyle: google
+  /// \endcode
+  std::vector<RawStringFormat> RawStringFormats;
+
   /// \brief If ``true``, clang-format will attempt to re-flow comments.
   /// \code
   ///    false:
   ///    // veryVeryVeryVeryVeryVeryVeryVeryVeryVeryVeryLongComment with plenty of information
   ///    /* second veryVeryVeryVeryVeryVeryVeryVeryVeryVeryVeryLongComment with plenty of information */
   ///
   ///    true:
   ///    // veryVeryVeryVeryVeryVeryVeryVeryVeryVeryVeryLongComment with plenty of
return AccessModifierOffset == R.AccessModifierOffset &&
            PenaltyBreakBeforeFirstCallParameter ==
                R.PenaltyBreakBeforeFirstCallParameter &&
            PenaltyBreakComment == R.PenaltyBreakComment &&
            PenaltyBreakFirstLessLess == R.PenaltyBreakFirstLessLess &&
            PenaltyBreakString == R.PenaltyBreakString &&
            PenaltyExcessCharacter == R.PenaltyExcessCharacter &&
            PenaltyReturnTypeOnItsOwnLine == R.PenaltyReturnTypeOnItsOwnLine &&
            PointerAlignment == R.PointerAlignment &&
+           RawStringFormats == R.RawStringFormats &&
            SpaceAfterCStyleCast == R.SpaceAfterCStyleCast &&
            SpaceAfterTemplateKeyword == R.SpaceAfterTemplateKeyword &&
            SpaceBeforeAssignmentOperators == R.SpaceBeforeAssignmentOperators &&
            SpaceBeforeParens == R.SpaceBeforeParens &&
            SpaceInEmptyParentheses == R.SpaceInEmptyParentheses &&
            SpacesBeforeTrailingComments == R.SpacesBeforeTrailingComments &&
            SpacesInAngles == R.SpacesInAngles &&
            SpacesInContainerLiterals == R.SpacesInContainerLiterals &&
struct FormattingAttemptStatus {
   bool FormatComplete = true;
 
   /// \brief If ``FormatComplete`` is false, ``Line`` records a one-based
   /// original line number at which a syntax error might have occurred. This is
   /// based on a best-effort analysis and could be imprecise.
   unsigned Line = 0;
 };
 
+tooling::Replacements reformat(const FormatStyle &Style, StringRef Code,
+                               ArrayRef<tooling::Range> Ranges,
+                               unsigned FirstStartColumn,
+                               unsigned NextStartColumn,
+                               StringRef FileName,
+                               FormattingAttemptStatus *Status);
+
 /// \brief Reformats the given \p Ranges in \p Code.
 ///
 /// Each range is extended on either end to its next bigger logic unit, i.e.
 /// everything that might influence its formatting or might be influenced by its
 /// formatting.
 ///
 /// Returns the ``Replacements`` necessary to make all \p Ranges comply with
 /// \p Style.

lib/Format/ContinuationIndenter.h

 
 #ifndef LLVM_CLANG_LIB_FORMAT_CONTINUATIONINDENTER_H
 #define LLVM_CLANG_LIB_FORMAT_CONTINUATIONINDENTER_H
 
 #include "Encoding.h"
 #include "FormatToken.h"
 #include "clang/Format/Format.h"
 #include "llvm/Support/Regex.h"
+#include <map>
+#include <tuple>
 
 namespace clang {
 class SourceManager;
 
 namespace format {
 
 class AnnotatedLine;
 struct FormatToken;
 struct LineState;
 struct ParenState;
+struct RawStringFormatStyleManager;
 class WhitespaceManager;
 
+struct RawStringFormatStyleManager {
+  llvm::StringMap<FormatStyle> DelimiterStyle;
+
+  RawStringFormatStyleManager(const FormatStyle &CodeStyle);
+
+  llvm::Optional<FormatStyle> get(StringRef Delimiter) const;
+};
+
 class ContinuationIndenter {
 public:
   /// \brief Constructs a \c ContinuationIndenter to format \p Line starting in
   /// column \p FirstIndent.
   ContinuationIndenter(const FormatStyle &Style,
                        const AdditionalKeywords &Keywords,
                        const SourceManager &SourceMgr,
                        WhitespaceManager &Whitespaces,
                        encoding::Encoding Encoding,
                        bool BinPackInconclusiveFunctions);
 
   /// \brief Get the initial state, i.e. the state after placing \p Line's
   /// first token at \p FirstIndent.
-  LineState getInitialState(unsigned FirstIndent, const AnnotatedLine *Line,
-                            bool DryRun);
+  LineState getInitialState(unsigned FirstIndent, unsigned FirstStartColumn,
+                            const AnnotatedLine *Line, bool DryRun);
 
   // FIXME: canBreak and mustBreak aren't strictly indentation-related. Find a
   // better home.
   /// \brief Returns \c true, if a line break after \p State is allowed.
   bool canBreak(const LineState &State);
 
   /// \brief Returns \c true, if a line break after \p State is mandatory.
   bool mustBreak(const LineState &State);
private:
 
   /// \brief Update 'State' according to the next token being one of "(<{[".
   void moveStatePastScopeOpener(LineState &State, bool Newline);
   /// \brief Update 'State' according to the next token being one of ")>}]".
   void moveStatePastScopeCloser(LineState &State);
   /// \brief Update 'State' with the next token opening a nested block.
   void moveStateToNewBlock(LineState &State);
 
+  /// \brief Reformats a raw string literal.
+  /// 
+  /// Returns column after the end of the raw string literal after reformat.
+  unsigned reformatRawStringLiteral(const FormatToken &Current,
+                                    LineState &State, StringRef Delimiter,
+                                    const FormatStyle &RawStringStyle,
+                                    bool DryRun);
+
   /// \brief If the current token sticks out over the end of the line, break
   /// it if possible.
   ///
   /// \returns An extra penalty if a token was broken, otherwise 0.
   ///
-  /// The returned penalty will cover the cost of the additional line breaks and
-  /// column limit violation in all lines except for the last one. The penalty
-  /// for the column limit violation in the last line (and in single line
-  /// tokens) is handled in \c addNextStateToQueue.
+  /// The returned penalty will cover the cost of the additional line breaks
+  /// and column limit violation in all lines except for the last one. The
+  /// penalty for the column limit violation in the last line (and in single
+  /// line tokens) is handled in \c addNextStateToQueue.
   unsigned breakProtrudingToken(const FormatToken &Current, LineState &State,
                                 bool DryRun);
 
   /// \brief Appends the next token to \p State and updates information
   /// necessary for indentation.
   ///
   /// Puts the token on the current line.
   ///
private:
 
   FormatStyle Style;
   const AdditionalKeywords &Keywords;
   const SourceManager &SourceMgr;
   WhitespaceManager &Whitespaces;
   encoding::Encoding Encoding;
   bool BinPackInconclusiveFunctions;
   llvm::Regex CommentPragmasRegex;
+  const RawStringFormatStyleManager RawStringFormats;
+  
+  /// A key in for the formatted raw string cache has the format
+  /// (RawStringToken, FirstStartColumn, NextStartColumn).
+  typedef std::tuple<const FormatToken*, unsigned, unsigned>
+      FormattedRawStringCacheKey;
+
+  struct FormattedRawStringCacheValue {
+    unsigned LastLineEndColumn;
+    tooling::Replacements Fixes;
+  };
+
+  std::map<FormattedRawStringCacheKey, FormattedRawStringCacheValue>
+      FormattedRawStringCache;
 };
 
 struct ParenState {
   ParenState(unsigned Indent, unsigned LastSpace, bool AvoidBinPacking,
              bool NoLineBreak)
       : Indent(Indent), LastSpace(LastSpace), NestedBlockIndent(Indent),
         BreakBeforeClosingBrace(false), AvoidBinPacking(AvoidBinPacking),
         BreakBeforeParameter(false), NoLineBreak(NoLineBreak),

lib/Format/ContinuationIndenter.cpp

static bool opensProtoMessageField(const FormatToken &LessTok,
   if (LessTok.isNot(tok::less))
     return false;
   return Style.Language == FormatStyle::LK_TextProto ||
          (Style.Language == FormatStyle::LK_Proto &&
           (LessTok.NestingLevel > 0 ||
            (LessTok.Previous && LessTok.Previous->is(tok::equal))));
 }
 
+// Returns the delimiter of a raw string literal, or None if TokenText is not
+// the text of a raw string literal.
+// For example, the delimiter of R"deli(cont)deli" is deli.
+static llvm::Optional<StringRef> getRawStringDelimiter(StringRef TokenText) {
+  if (TokenText.size() < 5 // The smallest raw string is R"()"
+      || !TokenText.startswith("R\"") || !TokenText.endswith("\""))
+    return None;
+
+  // A raw string starts with 'R"<delimiter>(' and delimiter is ascii and has
+  // size at most 16 by the standard, so the first '(' must be among the first
+  // 19 bytes.
+  size_t LParenPos = TokenText.substr(0, 19).find_first_of('(');
+  if (LParenPos == StringRef::npos)
+    return None;
+  StringRef Delimiter = TokenText.substr(2, LParenPos - 2);
+
+  // Check that the string ends in ')Delimiter"'
+  size_t RParenPos = TokenText.size() - Delimiter.size() - 2;
+  if (TokenText[RParenPos] != ')')
+    return None;
+  if (!TokenText.substr(RParenPos + 1).startswith(Delimiter))
+    return None;
+  return Delimiter;
+}
+
+RawStringFormatStyleManager::RawStringFormatStyleManager(
+    const FormatStyle &CodeStyle) {
+  for (const auto &RawStringFormat : CodeStyle.RawStringFormats) {
+    FormatStyle Style;
+    if (!getPredefinedStyle(RawStringFormat.BasedOnStyle,
+                            RawStringFormat.Language, &Style)) {
+      Style = getLLVMStyle();
+      Style.Language = RawStringFormat.Language;
+    }
+    Style.ColumnLimit = CodeStyle.ColumnLimit;
+    DelimiterStyle.insert({RawStringFormat.Delimiter, Style});
+  }
+}
+
+llvm::Optional<FormatStyle>
+RawStringFormatStyleManager::get(StringRef Delimiter) const {
+  auto It = DelimiterStyle.find(Delimiter);
+  if (It == DelimiterStyle.end())
+    return None;
+  return It->second;
+}
+
 ContinuationIndenter::ContinuationIndenter(const FormatStyle &Style,
                                            const AdditionalKeywords &Keywords,
                                            const SourceManager &SourceMgr,
                                            WhitespaceManager &Whitespaces,
                                            encoding::Encoding Encoding,
                                            bool BinPackInconclusiveFunctions)
     : Style(Style), Keywords(Keywords), SourceMgr(SourceMgr),
       Whitespaces(Whitespaces), Encoding(Encoding),
       BinPackInconclusiveFunctions(BinPackInconclusiveFunctions),
-      CommentPragmasRegex(Style.CommentPragmas) {}
+      CommentPragmasRegex(Style.CommentPragmas), RawStringFormats(Style) {}
 
 LineState ContinuationIndenter::getInitialState(unsigned FirstIndent,
+                                                unsigned FirstStartColumn,
                                                 const AnnotatedLine *Line,
                                                 bool DryRun) {
   LineState State;
   State.FirstIndent = FirstIndent;
-  State.Column = FirstIndent;
+  State.Column = FirstStartColumn ? FirstStartColumn : FirstIndent;
   State.Line = Line;
   State.NextToken = Line->First;
   State.Stack.push_back(ParenState(FirstIndent, FirstIndent,
                                    /*AvoidBinPacking=*/false,
                                    /*NoLineBreak=*/false));
   State.LineContainsContinuedForLoopSection = false;
   State.StartOfStringLiteral = 0;
   State.StartOfLineLevel = 0;
unsigned ContinuationIndenter::moveStateToNextToken(LineState &State,
   if (Current.is(TT_ObjCStringLiteral) && State.StartOfStringLiteral == 0)
     State.StartOfStringLiteral = State.Column + 1;
   else if (Current.isStringLiteral() && State.StartOfStringLiteral == 0)
     State.StartOfStringLiteral = State.Column;
   else if (!Current.isOneOf(tok::comment, tok::identifier, tok::hash) &&
            !Current.isStringLiteral())
     State.StartOfStringLiteral = 0;
 
-  State.Column += Current.ColumnWidth;
+  unsigned NewStateColumn = State.Column + Current.ColumnWidth;
+  if (Current.isStringLiteral()) {
+    if (llvm::Optional<StringRef> Delimiter =
+            getRawStringDelimiter(Current.TokenText)) {
+      if (llvm::Optional<FormatStyle> RawStringStyle =
+              RawStringFormats.get(*Delimiter))
+        NewStateColumn = reformatRawStringLiteral(Current, State, *Delimiter,
+                                                  *RawStringStyle, DryRun);
+    }
+  }
+
+  State.Column = NewStateColumn;
   State.NextToken = State.NextToken->Next;
   unsigned Penalty = 0;
   if (CanBreakProtrudingToken)
     Penalty = breakProtrudingToken(Current, State, DryRun);
   if (State.Column > getColumnLimit(State)) {
     unsigned ExcessCharacters = State.Column - getColumnLimit(State);
     Penalty += Style.PenaltyExcessCharacter * ExcessCharacters;
   }
 
   if (Current.Role)
     Current.Role->formatFromToken(State, this, DryRun);
   // If the previous has a special role, let it consume tokens as appropriate.
   // It is necessary to start at the previous token for the only implemented
   // role (comma separated list). That way, the decision whether or not to break
   // after the "{" is already done and both options are tried and evaluated.
   // FIXME: This is ugly, find a better way.
   if (Previous && Previous->Role)
     Penalty += Previous->Role->formatAfterToken(State, this, DryRun);
 
   return Penalty;
 }
 
 void ContinuationIndenter::moveStatePastFakeLParens(LineState &State,
                                                     bool Newline) {
unsigned NewIndent =
                                : Style.IndentWidth);
   State.Stack.push_back(ParenState(NewIndent, State.Stack.back().LastSpace,
                                    /*AvoidBinPacking=*/true,
                                    /*NoLineBreak=*/false));
   State.Stack.back().NestedBlockIndent = NestedBlockIndent;
   State.Stack.back().BreakBeforeParameter = true;
 }
 
+static unsigned getLastLineEndColumn(StringRef Text, unsigned StartColumn,
+                                     unsigned TabWidth,
+                                     encoding::Encoding Encoding) {
+  size_t LastNewlinePos = Text.find_last_of("\n");
+  if (LastNewlinePos == StringRef::npos) {
+    return StartColumn +
+           encoding::columnWidthWithTabs(Text, StartColumn, TabWidth, Encoding);
+  } else {
+    return encoding::columnWidthWithTabs(Text.substr(LastNewlinePos),
+                                         /*StartColumn=*/0, TabWidth, Encoding);
+  }
+}
+
+unsigned ContinuationIndenter::reformatRawStringLiteral(
+    const FormatToken &Current, LineState &State, StringRef Delimiter,
+    const FormatStyle &RawStringStyle, bool DryRun) {
+  // The text of a raw string is between the leading 'R"delimiter(' and the
+  // trailing 'delimiter)"'.
+  unsigned PrefixSize = 3 + Delimiter.size();
+  unsigned SuffixSize = 2 + Delimiter.size();
+  // The first start column is the column the raw text starts.
+  unsigned FirstStartColumn = State.Column + PrefixSize;
+  // The next start column is the intended indentation a line break inside
+  // the raw string at level 0.
+  unsigned NextStartColumn = State.FirstIndent + Style.ContinuationIndentWidth;
+
+  FormattedRawStringCacheKey CacheKey(&Current, FirstStartColumn,
+                                      NextStartColumn);
+  auto It = FormattedRawStringCache.find(CacheKey);
+  if (It == FormattedRawStringCache.end()) {
+    std::string RawText =
+        Current.TokenText.substr(PrefixSize).drop_back(SuffixSize);
+
+    tooling::Replacements Fixes =
+        reformat(RawStringStyle, RawText, {tooling::Range(0, RawText.size())},
+                 FirstStartColumn, NextStartColumn, "<stdin>",
+                 /*FormattingAttemptStatus=*/nullptr);
+
+    auto NewCode = applyAllReplacements(RawText, Fixes);
+    if (!NewCode) {
+      It = FormattedRawStringCache.insert(
+          It, {CacheKey,
+               {/*LastLineEndColumn=*/
+                State.Column + Current.ColumnWidth, Fixes}});
+    } else {
+      unsigned RawLastLineEndColumn = getLastLineEndColumn(
+          *NewCode, FirstStartColumn, Style.TabWidth, Encoding);
+      It = FormattedRawStringCache.insert(
+          It,
+          {CacheKey,
+           {/*LastLineEndColumn=*/RawLastLineEndColumn + SuffixSize, Fixes}});
+    }
+  }
+
+  if (!DryRun) {
+    SourceLocation OriginLoc =
+        Current.Tok.getLocation().getLocWithOffset(PrefixSize);
+    for (const tooling::Replacement &Fix : It->second.Fixes) {
+      auto Err = Whitespaces.addReplacement(tooling::Replacement(
+          SourceMgr, OriginLoc.getLocWithOffset(Fix.getOffset()),
+          Fix.getLength(), Fix.getReplacementText()));
+      if (Err) {
+        llvm::errs() << "Failed to reformat raw string: "
+                     << llvm::toString(std::move(Err)) << "\n";
+      }
+    }
+  }
+
+  return It->second.LastLineEndColumn;
+}
+
 unsigned ContinuationIndenter::addMultilineToken(const FormatToken &Current,
                                                  LineState &State) {
   if (!Current.IsMultiline)
     return 0;
 
   // Break before further function parameters on all levels.
   for (unsigned i = 0, e = State.Stack.size(); i != e; ++i)
     State.Stack[i].BreakBeforeParameter = true;

lib/Format/Format.cpp

 #include <algorithm>
 #include <memory>
 #include <string>
 
 #define DEBUG_TYPE "format-formatter"
 
 using clang::format::FormatStyle;
 
-LLVM_YAML_IS_SEQUENCE_VECTOR(clang::format::FormatStyle::IncludeCategory)
+LLVM_YAML_IS_SEQUENCE_VECTOR(clang::format::FormatStyle::IncludeCategory);
+LLVM_YAML_IS_SEQUENCE_VECTOR(clang::format::FormatStyle::RawStringFormat);
 
 namespace llvm {
 namespace yaml {
 template <> struct ScalarEnumerationTraits<FormatStyle::LanguageKind> {
   static void enumeration(IO &IO, FormatStyle::LanguageKind &Value) {
     IO.enumCase(Value, "Cpp", FormatStyle::LK_Cpp);
     IO.enumCase(Value, "Java", FormatStyle::LK_Java);
     IO.enumCase(Value, "JavaScript", FormatStyle::LK_JavaScript);
static void mapping(IO &IO, FormatStyle &Style) {
     IO.mapOptional("PenaltyBreakComment", Style.PenaltyBreakComment);
     IO.mapOptional("PenaltyBreakFirstLessLess",
                    Style.PenaltyBreakFirstLessLess);
     IO.mapOptional("PenaltyBreakString", Style.PenaltyBreakString);
     IO.mapOptional("PenaltyExcessCharacter", Style.PenaltyExcessCharacter);
     IO.mapOptional("PenaltyReturnTypeOnItsOwnLine",
                    Style.PenaltyReturnTypeOnItsOwnLine);
     IO.mapOptional("PointerAlignment", Style.PointerAlignment);
+    IO.mapOptional("RawStringFormats", Style.RawStringFormats);
     IO.mapOptional("ReflowComments", Style.ReflowComments);
     IO.mapOptional("SortIncludes", Style.SortIncludes);
     IO.mapOptional("SortUsingDeclarations", Style.SortUsingDeclarations);
     IO.mapOptional("SpaceAfterCStyleCast", Style.SpaceAfterCStyleCast);
     IO.mapOptional("SpaceAfterTemplateKeyword", Style.SpaceAfterTemplateKeyword);
     IO.mapOptional("SpaceBeforeAssignmentOperators",
                    Style.SpaceBeforeAssignmentOperators);
     IO.mapOptional("SpaceBeforeParens", Style.SpaceBeforeParens);

 
 template <> struct MappingTraits<FormatStyle::IncludeCategory> {
   static void mapping(IO &IO, FormatStyle::IncludeCategory &Category) {
     IO.mapOptional("Regex", Category.Regex);
     IO.mapOptional("Priority", Category.Priority);
   }
 };
 
+template <> struct MappingTraits<FormatStyle::RawStringFormat> {
+  static void mapping(IO &IO, FormatStyle::RawStringFormat &Format) {
+    IO.mapOptional("Delimiter", Format.Delimiter);
+    IO.mapOptional("Language", Format.Language);
+    IO.mapOptional("BasedOnStyle", Format.BasedOnStyle);
+  }
+};
+
 // Allows to read vector<FormatStyle> while keeping default values.
 // IO.getContext() should contain a pointer to the FormatStyle structure, that
 // will be used to get default values for missing keys.
 // If the first element has no Language specified, it will be treated as the
 // default one for the following elements.
 template <> struct DocumentListTraits<std::vector<FormatStyle>> {
   static size_t size(IO &IO, std::vector<FormatStyle> &Seq) {
     return Seq.size();
FormatStyle getLLVMStyle() {
   LLVMStyle.NamespaceIndentation = FormatStyle::NI_None;
   LLVMStyle.ObjCBlockIndentWidth = 2;
   LLVMStyle.ObjCSpaceAfterProperty = false;
   LLVMStyle.ObjCSpaceBeforeProtocolList = true;
   LLVMStyle.PointerAlignment = FormatStyle::PAS_Right;
   LLVMStyle.SpacesBeforeTrailingComments = 1;
   LLVMStyle.Standard = FormatStyle::LS_Cpp11;
   LLVMStyle.UseTab = FormatStyle::UT_Never;
+  LLVMStyle.RawStringFormats = {{"pb", FormatStyle::LK_TextProto, "llvm"}};
   LLVMStyle.ReflowComments = true;
   LLVMStyle.SpacesInParentheses = false;
   LLVMStyle.SpacesInSquareBrackets = false;
   LLVMStyle.SpaceInEmptyParentheses = false;
   LLVMStyle.SpacesInContainerLiterals = true;
   LLVMStyle.SpacesInCStyleCastParentheses = false;
   LLVMStyle.SpaceAfterCStyleCast = false;
   LLVMStyle.SpaceAfterTemplateKeyword = true;
analyze(TokenAnnotator &Annotator,
     WhitespaceManager Whitespaces(
         Env.getSourceManager(), Style,
         inputUsesCRLF(Env.getSourceManager().getBufferData(Env.getFileID())));
     ContinuationIndenter Indenter(Style, Tokens.getKeywords(),
                                   Env.getSourceManager(), Whitespaces, Encoding,
                                   BinPackInconclusiveFunctions);
     UnwrappedLineFormatter(&Indenter, &Whitespaces, Style, Tokens.getKeywords(),
                            Env.getSourceManager(), Status)
-        .format(AnnotatedLines);
+        .format(AnnotatedLines, /*DryRun=*/false,
+                /*AdditionalIndent=*/0,
+                /*FixBadIndentation=*/false,
+                /*FirstStartColumn=*/Env.getFirstStartColumn(),
+                /*NextStartColumn=*/Env.getNextStartColumn());
     for (const auto &R : Whitespaces.generateReplacements())
       if (Result.add(R))
         return Result;
     return Result;
   }
 
 private:
 
cleanupAroundReplacements(StringRef Code, const tooling::Replacements &Replaces,
   // Make header insertion replacements insert new headers into correct blocks.
   tooling::Replacements NewReplaces =
       fixCppIncludeInsertions(Code, Replaces, Style);
   return processReplacements(Cleanup, Code, NewReplaces, Style);
 }
 
 tooling::Replacements reformat(const FormatStyle &Style, StringRef Code,
                                ArrayRef<tooling::Range> Ranges,
+                               unsigned FirstStartColumn,
+                               unsigned NextStartColumn,
                                StringRef FileName,
                                FormattingAttemptStatus *Status) {
   FormatStyle Expanded = expandPresets(Style);
   if (Expanded.DisableFormat)
     return tooling::Replacements();
   if (Expanded.Language == FormatStyle::LK_JavaScript && isMpegTS(Code))
     return tooling::Replacements();
 
if (Style.Language == FormatStyle::LK_JavaScript &&
     Passes.emplace_back([&](const Environment &Env) {
       return JavaScriptRequoter(Env, Expanded).process();
     });
 
   Passes.emplace_back([&](const Environment &Env) {
     return Formatter(Env, Expanded, Status).process();
   });
 
-  std::unique_ptr<Environment> Env =
-      Environment::CreateVirtualEnvironment(Code, FileName, Ranges);
+  std::unique_ptr<Environment> Env = Environment::CreateVirtualEnvironment(
+      Code, FileName, Ranges, FirstStartColumn, NextStartColumn);
   llvm::Optional<std::string> CurrentCode = None;
   tooling::Replacements Fixes;
   for (size_t I = 0, E = Passes.size(); I < E; ++I) {
     tooling::Replacements PassFixes = Passes[I](*Env);
     auto NewCode = applyAllReplacements(
         CurrentCode ? StringRef(*CurrentCode) : Code, PassFixes);
     if (NewCode) {
       Fixes = Fixes.merge(PassFixes);
       if (I + 1 < E) {
         CurrentCode = std::move(*NewCode);
         Env = Environment::CreateVirtualEnvironment(
             *CurrentCode, FileName,
-            tooling::calculateRangesAfterReplacements(Fixes, Ranges));
+            tooling::calculateRangesAfterReplacements(Fixes, Ranges),
+            FirstStartColumn, NextStartColumn);
       }
     }
   }
 
   return Fixes;
 }
 
+tooling::Replacements reformat(const FormatStyle &Style, StringRef Code,
+                               ArrayRef<tooling::Range> Ranges,
+                               StringRef FileName,
+                               FormattingAttemptStatus *Status) {
+  return reformat(Style, Code, Ranges,
+                  /*FirstStartColumn=*/0,
+                  /*NextStartColumn=*/0, FileName, Status);
+}
+
 tooling::Replacements cleanup(const FormatStyle &Style, StringRef Code,
                               ArrayRef<tooling::Range> Ranges,
                               StringRef FileName) {
   // cleanups only apply to C++ (they mostly concern ctor commas etc.)
   if (Style.Language != FormatStyle::LK_Cpp)
     return tooling::Replacements();
   std::unique_ptr<Environment> Env =
       Environment::CreateVirtualEnvironment(Code, FileName, Ranges);

lib/Format/FormatTokenLexer.h

 enum LexerState {
   NORMAL,
   TEMPLATE_STRING,
   TOKEN_STASHED,
 };
 
 class FormatTokenLexer {
 public:
-  FormatTokenLexer(const SourceManager &SourceMgr, FileID ID,
+  FormatTokenLexer(const SourceManager &SourceMgr, FileID ID, unsigned Column,
                    const FormatStyle &Style, encoding::Encoding Encoding);
 
   ArrayRef<FormatToken *> lex();
 
   const AdditionalKeywords &getKeywords() { return Keywords; }
 
 private:
   void tryMergePreviousTokens();

lib/Format/FormatTokenLexer.cpp

 #include "clang/Basic/SourceManager.h"
 #include "clang/Format/Format.h"
 #include "llvm/Support/Regex.h"
 
 namespace clang {
 namespace format {
 
 FormatTokenLexer::FormatTokenLexer(const SourceManager &SourceMgr, FileID ID,
-                                   const FormatStyle &Style,
+                                   unsigned Column, const FormatStyle &Style,
                                    encoding::Encoding Encoding)
     : FormatTok(nullptr), IsFirstToken(true), StateStack({LexerState::NORMAL}),
-      Column(0), TrailingWhitespace(0), SourceMgr(SourceMgr), ID(ID),
+      Column(Column), TrailingWhitespace(0), SourceMgr(SourceMgr), ID(ID),
       Style(Style), IdentTable(getFormattingLangOpts(Style)),
       Keywords(IdentTable), Encoding(Encoding), FirstInLineIndex(0),
       FormattingDisabled(false), MacroBlockBeginRegex(Style.MacroBlockBegin),
       MacroBlockEndRegex(Style.MacroBlockEnd) {
   Lex.reset(new Lexer(ID, SourceMgr.getBuffer(ID), SourceMgr,
                       getFormattingLangOpts(Style)));
   Lex->SetKeepWhitespaceMode(true);
 

lib/Format/TokenAnalyzer.h

 #include "llvm/Support/Debug.h"
 
 namespace clang {
 namespace format {
 
 class Environment {
 public:
   Environment(SourceManager &SM, FileID ID, ArrayRef<CharSourceRange> Ranges)
-      : ID(ID), CharRanges(Ranges.begin(), Ranges.end()), SM(SM) {}
+      : ID(ID), CharRanges(Ranges.begin(), Ranges.end()), SM(SM),
+      FirstStartColumn(0),
+      NextStartColumn(0) {}
 
   Environment(FileID ID, std::unique_ptr<FileManager> FileMgr,
               std::unique_ptr<SourceManager> VirtualSM,
               std::unique_ptr<DiagnosticsEngine> Diagnostics,
-              const std::vector<CharSourceRange> &CharRanges)
+              const std::vector<CharSourceRange> &CharRanges,
+              unsigned FirstStartColumn,
+              unsigned NextStartColumn)
       : ID(ID), CharRanges(CharRanges.begin(), CharRanges.end()),
-        SM(*VirtualSM), FileMgr(std::move(FileMgr)),
+        SM(*VirtualSM), 
+        FirstStartColumn(FirstStartColumn),
+        NextStartColumn(NextStartColumn),
+        FileMgr(std::move(FileMgr)),
         VirtualSM(std::move(VirtualSM)), Diagnostics(std::move(Diagnostics)) {}
 
   // This sets up an virtual file system with file \p FileName containing \p
   // Code.
   static std::unique_ptr<Environment>
   CreateVirtualEnvironment(StringRef Code, StringRef FileName,
-                           ArrayRef<tooling::Range> Ranges);
+                           ArrayRef<tooling::Range> Ranges,
+                           unsigned FirstStartColumn = 0,
+                           unsigned NextStartColumn = 0);
 
   FileID getFileID() const { return ID; }
 
   ArrayRef<CharSourceRange> getCharRanges() const { return CharRanges; }
 
   const SourceManager &getSourceManager() const { return SM; }
 
+  unsigned getFirstStartColumn() const { return FirstStartColumn; }
+
+  unsigned getNextStartColumn() const { return NextStartColumn; }
+
 private:
   FileID ID;
   SmallVector<CharSourceRange, 8> CharRanges;
   SourceManager &SM;
+  unsigned FirstStartColumn;
+  unsigned NextStartColumn;
 
   // The order of these fields are important - they should be in the same order
   // as they are created in `CreateVirtualEnvironment` so that they can be
   // deleted in the reverse order as they are created.
   std::unique_ptr<FileManager> FileMgr;
   std::unique_ptr<SourceManager> VirtualSM;
   std::unique_ptr<DiagnosticsEngine> Diagnostics;
 };

lib/Format/TokenAnalyzer.cpp

 
 namespace clang {
 namespace format {
 
 // This sets up an virtual file system with file \p FileName containing \p
 // Code.
 std::unique_ptr<Environment>
 Environment::CreateVirtualEnvironment(StringRef Code, StringRef FileName,
-                                      ArrayRef<tooling::Range> Ranges) {
+                                      ArrayRef<tooling::Range> Ranges,
+                                      unsigned FirstStartColumn,
+                                      unsigned NextStartColumn) {
   // This is referenced by `FileMgr` and will be released by `FileMgr` when it
   // is deleted.
   IntrusiveRefCntPtr<vfs::InMemoryFileSystem> InMemoryFileSystem(
       new vfs::InMemoryFileSystem);
   // This is passed to `SM` as reference, so the pointer has to be referenced
   // in `Environment` so that `FileMgr` can out-live this function scope.
   std::unique_ptr<FileManager> FileMgr(
       new FileManager(FileSystemOptions(), InMemoryFileSystem));
Environment::CreateVirtualEnvironment(StringRef Code, StringRef FileName,
   assert(ID.isValid());
   SourceLocation StartOfFile = VirtualSM->getLocForStartOfFile(ID);
   std::vector<CharSourceRange> CharRanges;
   for (const tooling::Range &Range : Ranges) {
     SourceLocation Start = StartOfFile.getLocWithOffset(Range.getOffset());
     SourceLocation End = Start.getLocWithOffset(Range.getLength());
     CharRanges.push_back(CharSourceRange::getCharRange(Start, End));
   }
-  return llvm::make_unique<Environment>(ID, std::move(FileMgr),
-                                        std::move(VirtualSM),
-                                        std::move(Diagnostics), CharRanges);
+  return llvm::make_unique<Environment>(
+      ID, std::move(FileMgr), std::move(VirtualSM), std::move(Diagnostics),
+      CharRanges, FirstStartColumn, NextStartColumn);
 }
 
 TokenAnalyzer::TokenAnalyzer(const Environment &Env, const FormatStyle &Style)
     : Style(Style), Env(Env),
       AffectedRangeMgr(Env.getSourceManager(), Env.getCharRanges()),
       UnwrappedLines(1),
       Encoding(encoding::detectEncoding(
           Env.getSourceManager().getBufferData(Env.getFileID()))) {
   DEBUG(
       llvm::dbgs() << "File encoding: "
                    << (Encoding == encoding::Encoding_UTF8 ? "UTF8" : "unknown")
                    << "\n");
   DEBUG(llvm::dbgs() << "Language: " << getLanguageName(Style.Language)
                      << "\n");
 }
 
 tooling::Replacements TokenAnalyzer::process() {
   tooling::Replacements Result;
-  FormatTokenLexer Tokens(Env.getSourceManager(), Env.getFileID(), Style,
-                          Encoding);
+  FormatTokenLexer Tokens(Env.getSourceManager(), Env.getFileID(),
+                          Env.getFirstStartColumn(), Style, Encoding);
 
-  UnwrappedLineParser Parser(Style, Tokens.getKeywords(), Tokens.lex(), *this);
+  UnwrappedLineParser Parser(Style, Tokens.getKeywords(),
+                             Env.getFirstStartColumn(), Tokens.lex(), *this);
   Parser.parse();
   assert(UnwrappedLines.rbegin()->empty());
   for (unsigned Run = 0, RunE = UnwrappedLines.size(); Run + 1 != RunE; ++Run) {
     DEBUG(llvm::dbgs() << "Run " << Run << "...\n");
     SmallVector<AnnotatedLine *, 16> AnnotatedLines;
 
     TokenAnnotator Annotator(Style, Tokens.getKeywords());
     for (unsigned i = 0, e = UnwrappedLines[Run].size(); i != e; ++i) {

lib/Format/TokenAnnotator.h

 class AnnotatedLine {
 public:
   AnnotatedLine(const UnwrappedLine &Line)
       : First(Line.Tokens.front().Tok), Level(Line.Level),
         MatchingOpeningBlockLineIndex(Line.MatchingOpeningBlockLineIndex),
         InPPDirective(Line.InPPDirective),
         MustBeDeclaration(Line.MustBeDeclaration), MightBeFunctionDecl(false),
         IsMultiVariableDeclStmt(false), Affected(false),
-        LeadingEmptyLinesAffected(false), ChildrenAffected(false) {
+        LeadingEmptyLinesAffected(false), ChildrenAffected(false),
+        FirstStartColumn(Line.FirstStartColumn) {
     assert(!Line.Tokens.empty());
 
     // Calculate Next and Previous for all tokens. Note that we must overwrite
     // Next and Previous for every token, as previous formatting runs might have
     // left them in a different state.
     First->Previous = nullptr;
     FormatToken *Current = First;
     for (std::list<UnwrappedLineNode>::const_iterator I = ++Line.Tokens.begin(),
public:
 
   /// \c True if the leading empty lines of this line intersect with one of the
   /// input ranges.
   bool LeadingEmptyLinesAffected;
 
   /// \c True if one of this line's children intersects with an input range.
   bool ChildrenAffected;
 
+  unsigned FirstStartColumn;
+
 private:
   // Disallow copying.
   AnnotatedLine(const AnnotatedLine &) = delete;
   void operator=(const AnnotatedLine &) = delete;
 };
 
 /// \brief Determines extra information about the tokens comprising an
 /// \c UnwrappedLine.

lib/Format/TokenAnnotator.cpp

 void TokenAnnotator::calculateFormattingInformation(AnnotatedLine &Line) {
   for (SmallVectorImpl<AnnotatedLine *>::iterator I = Line.Children.begin(),
                                                   E = Line.Children.end();
        I != E; ++I) {
     calculateFormattingInformation(**I);
   }
 
   Line.First->TotalLength =
-      Line.First->IsMultiline ? Style.ColumnLimit : Line.First->ColumnWidth;
+      Line.First->IsMultiline ? Style.ColumnLimit
+                              : Line.FirstStartColumn + Line.First->ColumnWidth;
   FormatToken *Current = Line.First->Next;
   bool InFunctionDecl = Line.MightBeFunctionDecl;
   while (Current) {
     if (isFunctionDeclarationName(*Current, Line))
       Current->Type = TT_FunctionDeclarationName;
     if (Current->is(TT_LineComment)) {
       if (Current->Previous->BlockKind == BK_BracedInit &&
           Current->Previous->opensScope())

lib/Format/UnwrappedLineFormatter.h

UnwrappedLineFormatter(ContinuationIndenter *Indenter,
                          FormattingAttemptStatus *Status)
       : Indenter(Indenter), Whitespaces(Whitespaces), Style(Style),
         Keywords(Keywords), SourceMgr(SourceMgr),
         Status(Status) {}
 
   /// \brief Format the current block and return the penalty.
   unsigned format(const SmallVectorImpl<AnnotatedLine *> &Lines,
                   bool DryRun = false, int AdditionalIndent = 0,
-                  bool FixBadIndentation = false);
+                  bool FixBadIndentation = false,
+                  unsigned FirstStartColumn = 0,
+                  unsigned NextStartColumn = 0);
 
 private:
   /// \brief Add a new line and the required indent before the first Token
   /// of the \c UnwrappedLine if there was no structural parsing error.
   void formatFirstToken(const AnnotatedLine &Line,
                         const AnnotatedLine *PreviousLine, unsigned Indent);
 
   /// \brief Returns the column limit for a line, taking into account whether we

lib/Format/UnwrappedLineFormatter.cpp

LineFormatter(ContinuationIndenter *Indenter, WhitespaceManager *Whitespaces,
                 UnwrappedLineFormatter *BlockFormatter)
       : Indenter(Indenter), Whitespaces(Whitespaces), Style(Style),
         BlockFormatter(BlockFormatter) {}
   virtual ~LineFormatter() {}
 
   /// \brief Formats an \c AnnotatedLine and returns the penalty.
   ///
   /// If \p DryRun is \c false, directly applies the changes.
-  virtual unsigned formatLine(const AnnotatedLine &Line, unsigned FirstIndent,
+  virtual unsigned formatLine(const AnnotatedLine &Line,
+                              unsigned FirstIndent,
+                              unsigned FirstStartColumn,
                               bool DryRun) = 0;
 
 protected:
   /// \brief If the \p State's next token is an r_brace closing a nested block,
   /// format the nested block before it.
   ///
   /// Returns \c true if all children could be placed successfully and adapts
   /// \p Penalty as well as \p State. If \p DryRun is false, also directly
if (Style.ColumnLimit > 0 &&
         Child->Last->TotalLength + State.Column + 2 > Style.ColumnLimit)
       return false;
 
     if (!DryRun) {
       Whitespaces->replaceWhitespace(
           *Child->First, /*Newlines=*/0, /*Spaces=*/1,
           /*StartOfTokenColumn=*/State.Column, State.Line->InPPDirective);
     }
-    Penalty += formatLine(*Child, State.Column + 1, DryRun);
+    Penalty +=
+        formatLine(*Child, State.Column + 1, /*FirstStartColumn=*/0, DryRun);
 
     State.Column += 1 + Child->Last->TotalLength;
     return true;
   }
 
   ContinuationIndenter *Indenter;
 
 private:
NoColumnLimitLineFormatter(ContinuationIndenter *Indenter,
                              WhitespaceManager *Whitespaces,
                              const FormatStyle &Style,
                              UnwrappedLineFormatter *BlockFormatter)
       : LineFormatter(Indenter, Whitespaces, Style, BlockFormatter) {}
 
   /// \brief Formats the line, simply keeping all of the input's line breaking
   /// decisions.
   unsigned formatLine(const AnnotatedLine &Line, unsigned FirstIndent,
-                      bool DryRun) override {
+                      unsigned FirstStartColumn, bool DryRun) override {
     assert(!DryRun);
-    LineState State =
-        Indenter->getInitialState(FirstIndent, &Line, /*DryRun=*/false);
+    LineState State = Indenter->getInitialState(FirstIndent, FirstStartColumn,
+                                                &Line, /*DryRun=*/false);
     while (State.NextToken) {
       bool Newline =
           Indenter->mustBreak(State) ||
           (Indenter->canBreak(State) && State.NextToken->NewlinesBefore > 0);
       unsigned Penalty = 0;
       formatChildren(State, Newline, /*DryRun=*/false, Penalty);
       Indenter->addTokenToState(State, Newline, /*DryRun=*/false);
     }
     return 0;
   }
 };
 
 /// \brief Formatter that puts all tokens into a single line without breaks.
 class NoLineBreakFormatter : public LineFormatter {
 public:
   NoLineBreakFormatter(ContinuationIndenter *Indenter,
                        WhitespaceManager *Whitespaces, const FormatStyle &Style,
                        UnwrappedLineFormatter *BlockFormatter)
       : LineFormatter(Indenter, Whitespaces, Style, BlockFormatter) {}
 
   /// \brief Puts all tokens into a single line.
   unsigned formatLine(const AnnotatedLine &Line, unsigned FirstIndent,
-                      bool DryRun) override {
+                      unsigned FirstStartColumn, bool DryRun) override {
     unsigned Penalty = 0;
-    LineState State = Indenter->getInitialState(FirstIndent, &Line, DryRun);
+    LineState State =
+        Indenter->getInitialState(FirstIndent, FirstStartColumn, &Line, DryRun);
     while (State.NextToken) {
       formatChildren(State, /*Newline=*/false, DryRun, Penalty);
       Indenter->addTokenToState(
           State, /*Newline=*/State.NextToken->MustBreakBefore, DryRun);
     }
     return Penalty;
   }
 };
 
 /// \brief Finds the best way to break lines.
 class OptimizingLineFormatter : public LineFormatter {
 public:
   OptimizingLineFormatter(ContinuationIndenter *Indenter,
                           WhitespaceManager *Whitespaces,
                           const FormatStyle &Style,
                           UnwrappedLineFormatter *BlockFormatter)
       : LineFormatter(Indenter, Whitespaces, Style, BlockFormatter) {}
 
   /// \brief Formats the line by finding the best line breaks with line lengths
   /// below the column limit.
   unsigned formatLine(const AnnotatedLine &Line, unsigned FirstIndent,
-                      bool DryRun) override {
-    LineState State = Indenter->getInitialState(FirstIndent, &Line, DryRun);
+                      unsigned FirstStartColumn, bool DryRun) override {
+    LineState State =
+        Indenter->getInitialState(FirstIndent, FirstStartColumn, &Line, DryRun);
 
     // If the ObjC method declaration does not fit on a line, we should format
     // it with one arg per line.
     if (State.Line->Type == LT_ObjCMethodDecl)
       State.Stack.back().BreakBeforeParameter = true;
 
     // Find best solution in solution space.
     return analyzeSolutionSpace(State, DryRun);
private:
   llvm::SpecificBumpPtrAllocator<StateNode> Allocator;
 };
 
 } // anonymous namespace
 
 unsigned
 UnwrappedLineFormatter::format(const SmallVectorImpl<AnnotatedLine *> &Lines,
                                bool DryRun, int AdditionalIndent,
-                               bool FixBadIndentation) {
+                               bool FixBadIndentation,
+                               unsigned FirstStartColumn,
+                               unsigned NextStartColumn) {
   LineJoiner Joiner(Style, Keywords, Lines);
 
   // Try to look up already computed penalty in DryRun-mode.
   std::pair<const SmallVectorImpl<AnnotatedLine *> *, unsigned> CacheKey(
       &Lines, AdditionalIndent);
   auto CacheIt = PenaltyCache.find(CacheKey);
   if (DryRun && CacheIt != PenaltyCache.end())
     return CacheIt->second;
 
   assert(!Lines.empty());
   unsigned Penalty = 0;
   LevelIndentTracker IndentTracker(Style, Keywords, Lines[0]->Level,
                                    AdditionalIndent);
   const AnnotatedLine *PreviousLine = nullptr;
   const AnnotatedLine *NextLine = nullptr;
 
   // The minimum level of consecutive lines that have been formatted.
   unsigned RangeMinLevel = UINT_MAX;
 
+  bool FirstLine = true;
   for (const AnnotatedLine *Line =
            Joiner.getNextMergedLine(DryRun, IndentTracker);
-       Line; Line = NextLine) {
+       Line; Line = NextLine, FirstLine = false) {
     const AnnotatedLine &TheLine = *Line;
     unsigned Indent = IndentTracker.getIndent();
 
     // We continue formatting unchanged lines to adjust their indent, e.g. if a
     // scope was added. However, we need to carefully stop doing this when we
     // exit the scope of affected lines to prevent indenting a the entire
     // remaining file if it currently missing a closing brace.
     bool ContinueFormatting =
if (ShouldFormat && TheLine.Type != LT_Invalid) {
 
       NextLine = Joiner.getNextMergedLine(DryRun, IndentTracker);
       unsigned ColumnLimit = getColumnLimit(TheLine.InPPDirective, NextLine);
       bool FitsIntoOneLine =
           TheLine.Last->TotalLength + Indent <= ColumnLimit ||
           (TheLine.Type == LT_ImportStatement &&
            (Style.Language != FormatStyle::LK_JavaScript ||
             !Style.JavaScriptWrapImports));
-
       if (Style.ColumnLimit == 0)
         NoColumnLimitLineFormatter(Indenter, Whitespaces, Style, this)
-            .formatLine(TheLine, Indent, DryRun);
+            .formatLine(TheLine, NextStartColumn + Indent,
+                        FirstLine ? FirstStartColumn : 0, DryRun);
       else if (FitsIntoOneLine)
         Penalty += NoLineBreakFormatter(Indenter, Whitespaces, Style, this)
-                       .formatLine(TheLine, Indent, DryRun);
+                       .formatLine(TheLine, NextStartColumn + Indent,
+                                   FirstLine ? FirstStartColumn : 0, DryRun);
       else
         Penalty += OptimizingLineFormatter(Indenter, Whitespaces, Style, this)
-                       .formatLine(TheLine, Indent, DryRun);
+                       .formatLine(TheLine, NextStartColumn + Indent,
+                                   FirstLine ? FirstStartColumn : 0, DryRun);
       RangeMinLevel = std::min(RangeMinLevel, TheLine.Level);
     } else {
       // If no token in the current line is affected, we still need to format
       // affected children.
       if (TheLine.ChildrenAffected)
         for (const FormatToken *Tok = TheLine.First; Tok; Tok = Tok->Next)
           if (!Tok->Children.empty())
             format(Tok->Children, DryRun);

lib/Format/UnwrappedLineParser.h

struct UnwrappedLine {
 
   /// \brief If this \c UnwrappedLine closes a block in a sequence of lines,
   /// \c MatchingOpeningBlockLineIndex stores the index of the corresponding
   /// opening line. Otherwise, \c MatchingOpeningBlockLineIndex must be
   /// \c kInvalidIndex.
   size_t MatchingOpeningBlockLineIndex;
 
   static const size_t kInvalidIndex = -1;
+
+  unsigned FirstStartColumn = 0;
 };
 
 class UnwrappedLineConsumer {
 public:
   virtual ~UnwrappedLineConsumer() {}
   virtual void consumeUnwrappedLine(const UnwrappedLine &Line) = 0;
   virtual void finishRun() = 0;
 };
 
 class FormatTokenSource;
 
 class UnwrappedLineParser {
 public:
   UnwrappedLineParser(const FormatStyle &Style,
                       const AdditionalKeywords &Keywords,
+                      unsigned FirstStartColumn,
                       ArrayRef<FormatToken *> Tokens,
                       UnwrappedLineConsumer &Callback);
 
   void parse();
 
 private:
   void reset();
   void parseFile();
private:
   SmallVector<int, 8> PPLevelBranchCount;
 
   // Contains the number of branches per nesting level we are currently
   // in while parsing a preprocessor branch sequence.
   // This is used to update PPLevelBranchCount at the end of a branch
   // sequence.
   std::stack<int> PPChainBranchIndex;
 
+  unsigned FirstStartColumn;
+
   friend class ScopedLineState;
   friend class CompoundStatementIndenter;
 };
 
 struct UnwrappedLineNode {
   UnwrappedLineNode() : Tok(nullptr) {}
   UnwrappedLineNode(FormatToken *Tok) : Tok(Tok) {}
 

lib/Format/UnwrappedLineParser.cpp

private:
   ArrayRef<FormatToken *> Tokens;
   int Position;
 };
 
 } // end anonymous namespace
 
 UnwrappedLineParser::UnwrappedLineParser(const FormatStyle &Style,
                                          const AdditionalKeywords &Keywords,
+                                         unsigned FirstStartColumn,
                                          ArrayRef<FormatToken *> Tokens,
                                          UnwrappedLineConsumer &Callback)
     : Line(new UnwrappedLine), MustBreakBeforeNextToken(false),
       CurrentLines(&Lines), Style(Style), Keywords(Keywords),
       CommentPragmasRegex(Style.CommentPragmas), Tokens(nullptr),
-      Callback(Callback), AllTokens(Tokens), PPBranchLevel(-1) {}
+      Callback(Callback), AllTokens(Tokens), PPBranchLevel(-1),
+      FirstStartColumn(FirstStartColumn) {}
 
 void UnwrappedLineParser::reset() {
   PPBranchLevel = -1;
   Line.reset(new UnwrappedLine);
   CommentsBeforeNextToken.clear();
   FormatTok = nullptr;
   MustBreakBeforeNextToken = false;
   PreprocessorDirectives.clear();
   CurrentLines = &Lines;
   DeclarationScopeStack.clear();
   PPStack.clear();
+  Line->FirstStartColumn = FirstStartColumn;
 }
 
 void UnwrappedLineParser::parse() {
   IndexedTokenSource TokenSource(AllTokens);
+  Line->FirstStartColumn = FirstStartColumn;
   do {
     DEBUG(llvm::dbgs() << "----\n");
     reset();
     Tokens = &TokenSource;
     TokenSource.reset();
 
     readToken();
     parseFile();
while (!eof()) {
     } else {
       nextToken();
     }
   }
 }
 
 LLVM_ATTRIBUTE_UNUSED static void printDebugInfo(const UnwrappedLine &Line,
                                                  StringRef Prefix = "") {
-  llvm::dbgs() << Prefix << "Line(" << Line.Level << ")"
+  llvm::dbgs() << Prefix << "Line(" << Line.Level
+               << ", FSC=" << Line.FirstStartColumn << ")"
                << (Line.InPPDirective ? " MACRO" : "") << ": ";
   for (std::list<UnwrappedLineNode>::const_iterator I = Line.Tokens.begin(),
                                                     E = Line.Tokens.end();
        I != E; ++I) {
     llvm::dbgs() << I->Tok->Tok.getName() << "["
                  << "T=" << I->Tok->Type
                  << ", OC=" << I->Tok->OriginalColumn << "] ";
   }
if (Line->Tokens.empty())
     return;
   DEBUG({
     if (CurrentLines == &Lines)
       printDebugInfo(*Line);
   });
   CurrentLines->push_back(std::move(*Line));
   Line->Tokens.clear();
   Line->MatchingOpeningBlockLineIndex = UnwrappedLine::kInvalidIndex;
+  Line->FirstStartColumn = 0;
   if (CurrentLines == &Lines && !PreprocessorDirectives.empty()) {
     CurrentLines->append(
         std::make_move_iterator(PreprocessorDirectives.begin()),
         std::make_move_iterator(PreprocessorDirectives.end()));
     PreprocessorDirectives.clear();
   }
 }
 

lib/Format/WhitespaceManager.h

void replaceWhitespace(FormatToken &Tok, unsigned Newlines, unsigned Spaces,
                          bool InPPDirective = false);
 
   /// \brief Adds information about an unchangeable token's whitespace.
   ///
   /// Needs to be called for every token for which \c replaceWhitespace
   /// was not called.
   void addUntouchableToken(const FormatToken &Tok, bool InPPDirective);
 
+  llvm::Error addReplacement(const tooling::Replacement &Replacement);
+
   /// \brief Inserts or replaces whitespace in the middle of a token.
   ///
   /// Inserts \p PreviousPostfix, \p Newlines, \p Spaces and \p CurrentPrefix
   /// (in this order) at \p Offset inside \p Tok, replacing \p ReplaceChars
   /// characters.
   ///
   /// Note: \p Spaces can be negative to retain information about initial
   /// relative column offset between a line of a block comment and the start of

lib/Format/WhitespaceManager.cpp

if (Tok.Finalized)
     return;
   Changes.push_back(Change(Tok, /*CreateReplacement=*/false,
                            Tok.WhitespaceRange, /*Spaces=*/0,
                            Tok.OriginalColumn, Tok.NewlinesBefore, "", "",
                            InPPDirective && !Tok.IsFirst,
                            /*IsInsideToken=*/false));
 }
 
+llvm::Error
+WhitespaceManager::addReplacement(const tooling::Replacement &Replacement) {
+  return Replaces.add(Replacement);
+}
+
 void WhitespaceManager::replaceWhitespaceInToken(
     const FormatToken &Tok, unsigned Offset, unsigned ReplaceChars,
     StringRef PreviousPostfix, StringRef CurrentPrefix, bool InPPDirective,
     unsigned Newlines, int Spaces) {
   if (Tok.Finalized)
     return;
   SourceLocation Start = Tok.getStartOfNonWhitespace().getLocWithOffset(Offset);
   Changes.push_back(

unittests/Format/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   Support
   )
 
 add_clang_unittest(FormatTests
   CleanupTest.cpp
   FormatTest.cpp
   FormatTestComments.cpp
   FormatTestJS.cpp
   FormatTestJava.cpp
   FormatTestObjC.cpp
   FormatTestProto.cpp
+  FormatTestRawStrings.cpp
   FormatTestSelective.cpp
   FormatTestTextProto.cpp
   NamespaceEndCommentsFixerTest.cpp
   SortImportsTestJS.cpp
   SortIncludesTest.cpp
   UsingDeclarationsSorterTest.cpp
   )
 
 target_link_libraries(FormatTests
   clangBasic
   clangFormat
   clangFrontend
   clangRewrite
   clangToolingCore
   )

unittests/Format/FormatTest.cpp

std::vector<FormatStyle::IncludeCategory> ExpectedCategories = {{"abc/.*", 2},
                                                                   {".*", 1}};
   CHECK_PARSE("IncludeCategories:\n"
               "  - Regex: abc/.*\n"
               "    Priority: 2\n"
               "  - Regex: .*\n"
               "    Priority: 1",
               IncludeCategories, ExpectedCategories);
   CHECK_PARSE("IncludeIsMainRegex: 'abc$'", IncludeIsMainRegex, "abc$");
+
+  Style.RawStringFormats.clear();
+  std::vector<FormatStyle::RawStringFormat> ExpectedRawStringFormats = {
+      {"pb", FormatStyle::LK_TextProto, "llvm"},
+      {"cpp", FormatStyle::LK_Cpp, "google"}};
+
+  CHECK_PARSE("RawStringFormats:\n"
+              "  - Delimiter: 'pb'\n"
+              "    Language: TextProto\n"
+              "    BasedOnStyle: llvm\n"
+              "  - Delimiter: 'cpp'\n"
+              "    Language: Cpp\n"
+              "    BasedOnStyle: google",
+              RawStringFormats, ExpectedRawStringFormats);
 }
 
 TEST_F(FormatTest, ParsesConfigurationWithLanguages) {
   FormatStyle Style = {};
   Style.Language = FormatStyle::LK_Cpp;
   CHECK_PARSE("Language: Cpp\n"
               "IndentWidth: 12",
               IndentWidth, 12u);

unittests/Format/FormatTestRawStrings.cpp

+//===- unittest/Format/FormatTestRawStrings.cpp - Formatting unit tests ---===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Format/Format.h"
+
+#include "../Tooling/ReplacementTest.h"
+#include "FormatTestUtils.h"
+
+#include "clang/Frontend/TextDiagnosticPrinter.h"
+#include "llvm/Support/Debug.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "gtest/gtest.h"
+
+#define DEBUG_TYPE "format-test"
+
+using clang::tooling::ReplacementTest;
+using clang::tooling::toReplacements;
+
+namespace clang {
+namespace format {
+namespace {
+
+class FormatTestRawStrings : public ::testing::Test {
+protected:
+  enum StatusCheck { SC_ExpectComplete, SC_ExpectIncomplete, SC_DoNotCheck };
+
+  std::string format(llvm::StringRef Code,
+                     const FormatStyle &Style = getLLVMStyle(),
+                     StatusCheck CheckComplete = SC_ExpectComplete) {
+    DEBUG(llvm::errs() << "---\n");
+    DEBUG(llvm::errs() << Code << "\n\n");
+    std::vector<tooling::Range> Ranges(1, tooling::Range(0, Code.size()));
+    FormattingAttemptStatus Status;
+    tooling::Replacements Replaces =
+        reformat(Style, Code, Ranges, "<stdin>", &Status);
+    if (CheckComplete != SC_DoNotCheck) {
+      bool ExpectedCompleteFormat = CheckComplete == SC_ExpectComplete;
+      EXPECT_EQ(ExpectedCompleteFormat, Status.FormatComplete)
+          << Code << "\n\n";
+    }
+    ReplacementCount = Replaces.size();
+    auto Result = applyAllReplacements(Code, Replaces);
+    EXPECT_TRUE(static_cast<bool>(Result));
+    DEBUG(llvm::errs() << "\n" << *Result << "\n\n");
+    return *Result;
+  }
+
+  FormatStyle getStyleWithColumns(FormatStyle Style, unsigned ColumnLimit) {
+    Style.ColumnLimit = ColumnLimit;
+    return Style;
+  }
+
+  FormatStyle getLLVMStyleWithColumns(unsigned ColumnLimit) {
+    return getStyleWithColumns(getLLVMStyle(), ColumnLimit);
+  }
+
+  int ReplacementCount;
+
+  FormatStyle getRawStringPbStyleWithColumns(unsigned ColumnLimit) {
+    FormatStyle Style = getLLVMStyle();
+    Style.ColumnLimit = ColumnLimit;
+    Style.RawStringFormats = {{/*Delimiter=*/"pb",
+                               /*Kind=*/FormatStyle::LK_TextProto,
+                               /*BasedOnStyle=*/"google"}};
+    return Style;
+  }
+
+  FormatStyle getRawStringLLVMCppStyleBasedOn(std::string BasedOnStyle) {
+    FormatStyle Style = getLLVMStyle();
+    Style.RawStringFormats = {{/*Delimiter=*/"cpp",
+                               /*Kind=*/FormatStyle::LK_Cpp, BasedOnStyle}};
+    return Style;
+  }
+
+  FormatStyle getRawStringGoogleCppStyleBasedOn(std::string BasedOnStyle) {
+    FormatStyle Style = getGoogleStyle(FormatStyle::LK_Cpp);
+    Style.RawStringFormats = {{/*Delimiter=*/"cpp",
+                               /*Kind=*/FormatStyle::LK_Cpp, BasedOnStyle}};
+    return Style;
+  }
+};
+
+TEST_F(FormatTestRawStrings, ReformatsAccordingToBaseStyle) {
+  // llvm style puts '*' on the right.
+  // google style puts '*' on the left.
+
+  // Use the llvm style if the raw string style has no BasedOnStyle.
+  EXPECT_EQ(R"test(int *i = R"cpp(int *p = nullptr;)cpp")test",
+            format(R"test(int * i = R"cpp(int * p = nullptr;)cpp")test",
+                   getRawStringLLVMCppStyleBasedOn("")));
+
+  // Use the google style if the raw string style has BasedOnStyle=google.
+  EXPECT_EQ(R"test(int *i = R"cpp(int* p = nullptr;)cpp")test",
+            format(R"test(int * i = R"cpp(int * p = nullptr;)cpp")test",
+                   getRawStringLLVMCppStyleBasedOn("google")));
+
+  // Use the llvm style if the raw string style has no BasedOnStyle=llvm.
+  EXPECT_EQ(R"test(int* i = R"cpp(int *p = nullptr;)cpp")test",
+            format(R"test(int * i = R"cpp(int * p = nullptr;)cpp")test",
+                   getRawStringGoogleCppStyleBasedOn("llvm")));
+}
+
+TEST_F(FormatTestRawStrings, MatchesDelimitersCaseSensitively) {
+  // Don't touch the 'PB' raw string, format the 'pb' raw string.
+  EXPECT_EQ(R"test(
+s = R"PB(item:1)PB";
+t = R"pb(item: 1)pb";)test",
+            format(R"test(
+s = R"PB(item:1)PB";
+t = R"pb(item:1)pb";)test", getRawStringPbStyleWithColumns(40)));
+
+  FormatStyle MixedStyle = getLLVMStyle();
+  MixedStyle.RawStringFormats = {
+      {/*Delimiter=*/"cpp", /*Kind=*/FormatStyle::LK_Cpp,
+       /*BasedOnStyle=*/"llvm"},
+      {/*Delimiter=*/"CPP", /*Kind=*/FormatStyle::LK_Cpp,
+       /*BasedOnStyle=*/"google"}};
+
+  // Format the 'cpp' raw string with '*' on the right.
+  // Format the 'CPP' raw string with '*' on the left.
+  // Do not format the 'Cpp' raw string.
+  // Do not format non-raw strings.
+  EXPECT_EQ(R"test(
+a = R"cpp(int *i = 0;)cpp";
+b = R"CPP(int* j = 0;)CPP";
+c = R"Cpp(int * k = 0;)Cpp";
+d = R"cpp(int * k = 0;)Cpp";)test",
+            format(R"test(
+a = R"cpp(int * i = 0;)cpp";
+b = R"CPP(int * j = 0;)CPP";
+c = R"Cpp(int * k = 0;)Cpp";
+d = R"cpp(int * k = 0;)Cpp";)test",
+                   MixedStyle));
+}
+
+TEST_F(FormatTestRawStrings, ReformatsShortRawStringsOnSingleLine) {
+  EXPECT_EQ(
+      R"test(P p = TP(R"pb()pb");)test",
+      format(
+          R"test(P p = TP(R"pb( )pb");)test",
+          getRawStringPbStyleWithColumns(40)));
+  EXPECT_EQ(
+      R"test(P p = TP(R"pb(item_1: 1)pb");)test",
+      format(
+          R"test(P p = TP(R"pb(item_1:1)pb");)test",
+          getRawStringPbStyleWithColumns(40)));
+  EXPECT_EQ(
+      R"test(P p = TP(R"pb(item_1: 1)pb");)test",
+      format(
+          R"test(P p = TP(R"pb(  item_1 :  1   )pb");)test",
+          getRawStringPbStyleWithColumns(40)));
+  EXPECT_EQ(
+      R"test(P p = TP(R"pb(item_1: 1 item_2: 2)pb");)test",
+      format(
+          R"test(P p = TP(R"pb(item_1:1 item_2:2)pb");)test",
+          getRawStringPbStyleWithColumns(40)));
+  EXPECT_EQ(
+      R"test(P p = TP(R"pb(item_1 <1> item_2: {2})pb");)test",
+      format(
+          R"test(P p = TP(R"pb(item_1<1> item_2:{2})pb");)test",
+          getRawStringPbStyleWithColumns(40)));
+}
+
+TEST_F(FormatTestRawStrings, BreaksRawStringsExceedingColumnLimit) {
+  EXPECT_EQ(R"test(
+P p = TPPPPPPPPPPPPPPP(
+    R"pb(item_1: 1, item_2: 2)pb");)test",
+            format(R"test(
+P p = TPPPPPPPPPPPPPPP(R"pb(item_1: 1, item_2: 2)pb");)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  EXPECT_EQ(R"test(
+P p =
+    TPPPPPPPPPPPPPPP(
+        R"pb(item_1: 1,
+             item_2: 2,
+             item_3: 3)pb");)test",
+            format(R"test(
+P p = TPPPPPPPPPPPPPPP(R"pb(item_1: 1, item_2: 2, item_3: 3)pb");)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  EXPECT_EQ(
+      R"test(P p = TP(R"pb(item_1 <1>
+    item_2: <2>
+    item_3 {})pb");)test",
+      format(
+          R"test(P p = TP(R"pb(item_1<1> item_2:<2> item_3{ })pb");)test",
+          getRawStringPbStyleWithColumns(40)));
+
+  EXPECT_EQ(
+      R"test(
+P p = TP(R"pb(item_1: 1,
+              item_2: 2,
+              item_3: 3,
+              item_4: 4)pb");)test",
+      format(
+          R"test(
+P p = TP(R"pb(item_1: 1, item_2: 2, item_3: 3, item_4: 4)pb");)test",
+          getRawStringPbStyleWithColumns(40)));
+
+  EXPECT_EQ(R"test(
+P p = TPPPPPPPPPPPPPPP(
+    R"pb(item_1 <1>,
+         item_2: {2},
+         item_3: <3>,
+         item_4: {4})pb");)test",
+            format(R"test(
+P p = TPPPPPPPPPPPPPPP(R"pb(item_1<1>, item_2: {2}, item_3: <3>, item_4:{4})pb");)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  // Breaks before a short raw string exceeding the column limit.
+  EXPECT_EQ(R"test(
+FFFFFFFFFFFFFFFFFFFFFFFFFFF(
+    R"pb(key: 1)pb");
+P p = TPPPPPPPPPPPPPPPPPPPP(
+    R"pb(key: 2)pb");
+auto TPPPPPPPPPPPPPPPPPPPP =
+    R"pb(key: 3)pb";
+P p = TPPPPPPPPPPPPPPPPPPPP(
+    R"pb(i: 1, j: 2)pb");
+
+int f(string s) {
+  FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF(
+      R"pb(key: 1)pb");
+  P p = TPPPPPPPPPPPPPPPPPPPP(
+      R"pb(key: 2)pb");
+  auto TPPPPPPPPPPPPPPPPPPPP =
+      R"pb(key: 3)pb";
+  if (s.empty())
+    P p = TPPPPPPPPPPPPPPPPPPPP(
+        R"pb(i: 1, j: 2)pb");
+}
+)test",
+            format(R"test(
+FFFFFFFFFFFFFFFFFFFFFFFFFFF(R"pb(key:1)pb");
+P p = TPPPPPPPPPPPPPPPPPPPP(R"pb(key:2)pb");
+auto TPPPPPPPPPPPPPPPPPPPP = R"pb(key:3)pb";
+P p = TPPPPPPPPPPPPPPPPPPPP(R"pb(i: 1, j:2)pb");
+
+int f(string s) {
+  FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF(R"pb(key:1)pb");
+  P p = TPPPPPPPPPPPPPPPPPPPP(R"pb(key:2)pb");
+  auto TPPPPPPPPPPPPPPPPPPPP = R"pb(key:3)pb";
+  if (s.empty())
+    P p = TPPPPPPPPPPPPPPPPPPPP(R"pb(i: 1, j:2)pb");
+}
+)test",
+                   getRawStringPbStyleWithColumns(40)));
+}
+
+TEST_F(FormatTestRawStrings, FormatsRawStringArguments) {
+  EXPECT_EQ(R"test(
+P p = TP(R"pb(key {1})pb", param_2);)test",
+            format(R"test(
+P p = TP(R"pb(key{1})pb",param_2);)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  EXPECT_EQ(R"test(
+PPPPPPPPPPPPP(R"pb(keykeyk)pb",
+              param_2);)test",
+            format(R"test(
+PPPPPPPPPPPPP(R"pb(keykeyk)pb", param_2);)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  EXPECT_EQ(R"test(
+P p = TP(R"pb(item: {i: 1, s: 's'}
+    item: {i: 2, s: 't'})pb");)test",
+            format(R"test(
+P p = TP(R"pb(item: {i: 1, s: 's'} item: {i: 2, s: 't'})pb");)test",
+                   getRawStringPbStyleWithColumns(40)));
+  EXPECT_EQ(R"test(
+FFFFFFFFFFFFFFFFFFF(
+    R"pb(key: "value")pb",
+    R"pb(key2: "value")pb");)test",
+            format(R"test(
+FFFFFFFFFFFFFFFFFFF(R"pb(key: "value")pb", R"pb(key2: "value")pb");)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  // Formats the first out of two arguments.
+  EXPECT_EQ(R"test(
+FFFFFFFF(R"pb(key: 1)pb", argument2);
+struct S {
+  const s =
+      f(R"pb(key: 1)pb", argument2);
+  void f() {
+    if (gol)
+      return g(R"pb(key: 1)pb",
+               132789237);
+    return g(R"pb(key: 1)pb", "172893");
+  }
+};)test",
+            format(R"test(
+FFFFFFFF(R"pb(key:1)pb", argument2);
+struct S {
+const s = f(R"pb(key:1)pb", argument2);
+void f() {
+  if (gol)
+    return g(R"pb(key:1)pb", 132789237);
+  return g(R"pb(key:1)pb", "172893");
+}
+};)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  // Formats the second out of two arguments.
+  EXPECT_EQ(R"test(
+FFFFFFFF(argument1, R"pb(key: 2)pb");
+struct S {
+  const s =
+      f(argument1, R"pb(key: 2)pb");
+  void f() {
+    if (gol)
+      return g(12784137,
+               R"pb(key: 2)pb");
+    return g(17283122, R"pb(key: 2)pb");
+  }
+};)test",
+            format(R"test(
+FFFFFFFF(argument1, R"pb(key:2)pb");
+struct S {
+const s = f(argument1, R"pb(key:2)pb");
+void f() {
+  if (gol)
+    return g(12784137, R"pb(key:2)pb");
+  return g(17283122, R"pb(key:2)pb");
+}
+};)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  // Formats two short raw string arguments.
+  EXPECT_EQ(R"test(
+FFFFF(R"pb(key: 1)pb", R"pb(key: 2)pb");)test",
+            format(R"test(
+FFFFF(R"pb(key:1)pb", R"pb(key:2)pb");)test",
+                   getRawStringPbStyleWithColumns(40)));
+  // TODO(krasimir): The original source code fits on one line, so the
+  // non-optimizing formatter is chosen. But after the formatting in protos is
+  // made, the code doesn't fit on one line anymore and further formatting
+  // splits it.
+  //
+  // Should we disable raw string formatting for the non-optimizing formatter?
+  EXPECT_EQ(R"test(
+FFFFFFF(R"pb(key: 1)pb", R"pb(key: 2)pb");)test",
+            format(R"test(
+FFFFFFF(R"pb(key:1)pb", R"pb(key:2)pb");)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  // Formats two short raw string arguments, puts second on newline.
+  EXPECT_EQ(R"test(
+FFFFFFFF(R"pb(key: 1)pb",
+         R"pb(key: 2)pb");)test",
+            format(R"test(
+FFFFFFFF(R"pb(key:1)pb", R"pb(key:2)pb");)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  // Formats both arguments.
+  EXPECT_EQ(R"test(
+FFFFFFFF(R"pb(key: 1)pb",
+         R"pb(key: 2)pb");
+struct S {
+  const s = f(R"pb(key: 1)pb",
+              R"pb(key: 2)pb");
+  void f() {
+    if (gol)
+      return g(R"pb(key: 1)pb",
+               R"pb(key: 2)pb");
+    return g(R"pb(k1)pb", R"pb(k2)pb");
+  }
+};)test",
+            format(R"test(
+FFFFFFFF(R"pb(key:1)pb", R"pb(key:2)pb");
+struct S {
+const s = f(R"pb(key:1)pb", R"pb(key:2)pb");
+void f() {
+  if (gol)
+    return g(R"pb(key:1)pb", R"pb(key:2)pb");
+  return g(R"pb( k1 )pb", R"pb( k2 )pb");
+}
+};)test",
+                   getRawStringPbStyleWithColumns(40)));
+}
+
+} // end namespace
+} // end namespace format
+} // end namespace clang