[clang-format] Format raw string literals

[krasimir]

Summary

This patch adds raw string literal formatting.

It's still work in progress, however feedback is welcome.

[klimek]

lib/Format/ContinuationIndenter.cpp

89 ↗

(On Diff #108460)

Why 19?

1259–1262 ↗

(On Diff #108460)

Why 3? Generally, when you use magic values other than 0, -1, 1, comment on what exactly they mean.

lib/Format/ContinuationIndenter.h

41 ↗

(On Diff #108460)

Why getGoogleStyle?

[krasimir]

• Fixed formatting

[krasimir]

• Add comments on magic

[krasimir]

• Add tests for delimiters and make delimiter search case-insensitive

[krasimir]

• Add tests for raw strings in arguments

[krasimir]

• Make reformatRawStringLiteral return the new state column
• Add FormattedRawString cache

[krasimir]

• Add BasedOnStyle to RawStringFormat

[krasimir]

lib/Format/ContinuationIndenter.h

41 ↗

(On Diff #108460)

Added BasedOnStyle to the RawStringStyle type, now it's not arbitrary anymore. Thank you!

[krasimir]

• Add tests for case-sensitive delimiter matching

[krasimir]

• Add tests for raw strings starting on own line

[krasimir]

• Add tests for breaks before raw strings

[krasimir]

• Add tests for raw string operands

[krasimir]

• Add more tests for raw string operands

[krasimir]

• Add LastStartColumn and prefix and postfix raw string alignment

[krasimir]

• Adjust the indent of raw string suffixes

[krasimir]

• Add a test demonstrating exceeding the column limit

[krasimir]

• Propagate estimated raw string penalty to the reflow optimization

[krasimir]

• Rebase with master

[krasimir]

I think this is ready for a round of reviews.

[klimek]

lib/Format/ContinuationIndenter.cpp

83 ↗

(On Diff #109554)

If it == 5, it''s also trivially empty. It can't really be 6 (as it needs to be repeated), so only 5 + 2n are valid sizes? So < 7 would work here?

95 ↗

(On Diff #109554)

After tokenization, can't we assume that's true?

138 ↗

(On Diff #109554)

Why do we need a new parameter to pass through instead of using FirstIndent?

1011 ↗

(On Diff #109554)

Add a comment why this happens (-> reading this, I assume the State.Column is adapted in the call to reformatRawStringLiteral).

1015 ↗

(On Diff #109554)

I'd have expected the raw string formatting to happen here, as breakProtrudingToken is very similar (semantically) to formatting raw strings.

1286–1289 ↗

(On Diff #109554)

Why do we need to estimate the penalty, as opposed to getting it from the recursive formatting step?

1325–1326 ↗

(On Diff #109554)

Why do we need an extra cache and can't let the State take care of this?

lib/Format/ContinuationIndenter.h

167 ↗

(On Diff #109554)

"A key in for" doesn't parse.

[krasimir]

lib/Format/ContinuationIndenter.cpp

83 ↗

(On Diff #109554)

The delimiter could be the empty string. Added a comment for that.

95 ↗

(On Diff #109554)

No. The tokenizer is pretty liberal and will recognize R"pb(text)" as a string literal, for example (this is because of some handling of the strings occurring as header names). Added a test for that.

[krasimir]

lib/Format/ContinuationIndenter.cpp

138 ↗

(On Diff #109554)

There are two parameters in State that are initialized: State.FirstIndent, which is always initialized as FirstIndent and Stare.Column, which is initialized appropriately.

The raw string formatting introduces the ability to format a piece of code as if it's first line started at FirstStartColumn, and this new parameter needs to be propagated appropriately where it matters.

1015 ↗

(On Diff #109554)

I explored this approach initially. The problem is that breakProtrudingToken internally assumes that the token has been put on the state (the protruding part) and reformatting conceptually takes place before that.

1286–1289 ↗

(On Diff #109554)

That would be the other option. I went with the simplest approach.
Here's some problems with getting the penalty from the recursive step:

1. This breaks the abstraction of reformat. Conceptually, the penalty stuff is an implementation detail of the Optimizing Formatter. For other implementations of Formatter-s, we don't have a meaningful penalty.
2. The penalty is computed in the context of a different FormatStyle, so it might be incompatible with the way the penalties are computed in the current context.

1325–1326 ↗

(On Diff #109554)

How can the State take care of this? We want to save the edits to the raw string text so that we don't call reformat again in non-dry-run-mode. Conceptually, there might be different State-s having the same FormattedRawStringCacheKey, where the edits could be reused.

[krasimir]

• Address review comments

[krasimir]

• Update comments

[klimek]

lib/Format/ContinuationIndenter.cpp

95 ↗

(On Diff #109554)

I mean, that's a string literal, just not a raw string literal :) I'm wondering why the tokanizer doesn't have to make that distinction. Anyway, if it can happen, of course this looks fine.

138 ↗

(On Diff #109554)

I thought that's what FirstIndent means? I'm still really confused :)

1015 ↗

(On Diff #109554)

Why? I guess my question is: Where do the side effects of breakProtrudingToken and reformatRawStringLiteral differ?

1286–1289 ↗

(On Diff #109554)

(2) is an interesting argument. I don't see why (1) holds, though, as we're already doing recursive formatting for child blocks within statements. I'd like to get Daniel's opinion on this point.

1325–1326 ↗

(On Diff #109554)

What's the performance difference there?

[djasper]

lib/Format/ContinuationIndenter.cpp

1286–1289 ↗

(On Diff #109554)

Ok, as for the function itself: It needs a better description of what the estimation does and why.

For the points:

1. The optimizing formatter can compute the penalty. If the NoLineBreakFormatter is used, then the penalty should be 0 (there should be no line break or column limit violation). The NoColumnLimitFormatter doesn't care about penalties at all, so we can also assume 0.

2. In what way can it be incompatible?

[krasimir]

lib/Format/ContinuationIndenter.cpp

138 ↗

(On Diff #109554)

FirstIndent is the expected indent of the first token of the unwrapped line, which gets propagated, for example, in line breaks. The formatter itself controls it.
FirstStartColumn is where this line starts. The formatter doesn't control it.

Consider:

void f() {
  auto s = R"pb(key1: value1,
      key2: value2,
      key3: value3)pb";
}

To reformat the raw string literal, we need to take into account both the column where key1 starts (that is FirstStartColumn is 16) and the indent we'd like the first unwrapped line to have (that is FirstIndent is 6).

Maybe we need to rename these and add comments about them?

1015 ↗

(On Diff #109554)

You're right. This belongs to breakProtrudingToken. I'll move this to an implementation of BreakableToken.

1286–1289 ↗

(On Diff #109554)

I have another point about switching to use the recursive formatting of the penalties:

3. The penalty for the last line of a multiline token is handled at a later stage. If we use the penalty recursively, that will include the penalty for the last line, so we'll need to somehow propagate to the later stage that we don't wanna handle the penalty for the last line there.

As an example for (2) penalty incompatibility, you could use a style inside the raw strings that has different PenaltyExcessCharacter, for example.

Also, consider another possible use-case of formatting SQL raw string literals, for example. clang-format doesn't understand how to format these and could possibly only touch the leading indent of the lines. In that case, the penalty estimation would work.

1325–1326 ↗

(On Diff #109554)

Without the cache, the edits to each raw string token are computed at least twice: once in DryRun mode, once in wet mode.
So, I can come up with an example where this cache makes formatting twice faster.

Switching the implementation to a BreakableToken would eliminate the need of the cache, since the edits will be stored in the BreakableToken itself.

[krasimir]

• Move reformatRawStringLiteral to breakProtrudingToken

[krasimir]

lib/Format/ContinuationIndenter.cpp

1325–1326 ↗

(On Diff #109554)

I was wrong. I moved the implementation to breakProtrudingToken, but that doesn't eliminate the need for the cache.

[krasimir]

• Document estimatedPenalty

[krasimir]

• Fix formatting around a newly added block

[krasimir]

lib/Format/ContinuationIndenter.cpp

1309 ↗

(On Diff #109885)

@djasper, @klimek: what's your opinion about this decision? It seems logical, but most of the examples I see in the wild are formatted more like State.FirstIndent + Style.IndentWidth.

1286–1289 ↗

(On Diff #109554)

Added a description for estimatedPenalty.

[klimek]

lib/Format/ContinuationIndenter.cpp

138 ↗

(On Diff #109554)

The same thing is true for blocks inside expressions, though - what's the difference?

1015 ↗

(On Diff #109554)

Wait wait :) I'm not arguing it needs to be using the same implementation. I'm just saying it should be at the same spot in the code.

1325–1326 ↗

(On Diff #109554)

What's the performance difference without the cache?

[krasimir]

lib/Format/ContinuationIndenter.cpp

138 ↗

(On Diff #109554)

I don't know how blocks inside expressions are handled. Could you please give me an example or point me to that code?

1325–1326 ↗

(On Diff #109554)

Without the cache, the edits to each raw string token are computed at least twice: once in DryRun mode, once in wet mode.
So, I can come up with an example where this cache makes formatting twice faster.

[klimek]

lib/Format/ContinuationIndenter.cpp

138 ↗

(On Diff #109554)

void f() {
  otherfunc([] {
    innerStatement();
    auto letsgo = [] {
      deeper();
    };
  });
}

[krasimir]

lib/Format/ContinuationIndenter.cpp

138 ↗

(On Diff #109554)

Thank you! Here's how formatting of blocks inside expressions works now:
Consider this formatted code (with column limit 20):

a = [] {
  bbbb = 1;
};

This is parsed as two unwrapped lines, one like a = [] {}; and a child line attached to the { token like bbbb = 1;. Since these are two independent lines, they have two different values of FirstIndent: 0 and 2, respectively. Also, since these are on two separate unwrapped lines, the formatting of the { is independent of the formatting inside (except for the first indent that gets propagated through).

The situation with raw string literals is a bit different. Consider the following formatted code (with column limit 20):

auto s = R"pb(key1: value1,
    key2: value2,
    key3: value3)pb";

This is parsed as a single unwrapped line, like auto s = R"pb(...)pb"; with a single value of FirstIndent: 0. We need to somehow propagate the information about the additional indent that a line break at line level 0 inside the raw string literal would get, and that's how we end up with an additional parameter. From the point of view of the surrounding formatting, the formatting of the raw string literal is a black box returning replacements (and possibly additional penalty). So the surrounding formatting code needs to inform the raw string formatting code about the context in which the formatting occurs, which is achieved by passing the newly introduced FirstStartColumn, NextStartColumn and LastStartColumn parameters.

[klimek]

lib/Format/ContinuationIndenter.cpp

138 ↗

(On Diff #109554)

I'm wondering whether the inner indent should come from the style of the inner formatter. I'm not sure yet. I think if it shouldn't, that's a good argument for passing it through (and we need to document that decision).

1286–1289 ↗

(On Diff #109554)

As long as the penalty is applied consistently for every different possibility, and just adds up, won't it still be correct (in selecting the best solution), even if we have totally different excess char penalties? Can you give an example where that's not true?

1325–1326 ↗

(On Diff #109554)

I understand that without the cache it's going to be evaluated twice. We do that in lots of cases, though. Which is why I'm curious how much difference in runtime this actually makes (we usually dry-run a bazillion of states and then apply only one, so the dry- vs. non-dry-run runtime usually doesn't matter).

[krasimir]

lib/Format/ContinuationIndenter.cpp

138 ↗

(On Diff #109554)

The inner indent already comes from the style of the inner formatter. The offset where the inner column 0 starts, however, is controlled by the outer formatter.
Consider this code:

string s = R"pb(item: 1,
    msg_field {
      key: value
    })pb";

void f() {
  string s = R"pb(item: 1,
      msg_field {
        key: value
      })pb";
}

The 4 column relative indent of msg_field is the ContinuationIndentWidth of the outer format style.
The additional 2 column relative indent of key comes from the IndentWidth of the inner format style.
The outer formatter sees this code as:

string s = R"pb(<...stuff that the inner formatter will take care of>
####<...stuff that the inner formatter will take care of>
####<...stuff that the inner formatter will take care of>
####<...stuff that the inner formatter will take care of>)pb";

void f() {
  string s = R"pb(<...stuff that the inner formatter will take care of>
######<...stuff that the inner formatter will take care of>
######<...stuff that the inner formatter will take care of>
######<...stuff that the inner formatter will take care of>
######})pb";
}

where # signifies the offset where column 0 starts in the raw strings.
I'll refine the information about this new option with a set of rules that roughly determine the indentation decisions taken.

1286–1289 ↗

(On Diff #109554)

Good point. Thank you!
I can't come up with an example.

1325–1326 ↗

(On Diff #109554)

Do you have any ideas on how to gather some meaningful measurements on the runtime impact of this patch? I'm thinking of running it over a bunch of real-world source code that does proto processing and noting the runtime.

[klimek]

lib/Format/ContinuationIndenter.cpp

1325–1326 ↗

(On Diff #109554)

Yep, that's exactly what I'd expect. Also try to find some "worst case" scenarios, where you think you have an example that would benefit the most from the caching and measure that - that often gives a good upper bound.

[krasimir]

lib/Format/ContinuationIndenter.cpp

1325–1326 ↗

(On Diff #109554)

The cache adds ~4% performance improvement on a bunch of real-world examples.

[krasimir]

• Get the penalty from the recursive invocation

[krasimir]

• Update the section in moveStateToNextToken to match the original

[krasimir]

• Updated indent to IndentWidth
• Revised the rules for formatting the prefix and suffix

[krasimir]

I think this is ready for another round of reviews.

[klimek]

lib/Format/ContinuationIndenter.cpp

138 ↗

(On Diff #109554)

Ok, so the problem is that if we want to start an inner formatter, FirstStartColumn will be the column at which we currently are at the state of the parent formatter?
(If that's the case, please document it :)

1325–1326 ↗

(On Diff #109554)

I'm not sure whether that's worth it, but I'm happy to let Daniel tip the scale.

[krasimir]

lib/Format/ContinuationIndenter.cpp

1325–1326 ↗

(On Diff #109554)

I believe that the value of the cache is that it shows explicitly and mechanically the exact state that is propagated to the inner formatter. As such, I would argue that the cache increases maintainability.

[krasimir]

• Rebase with master

[djasper]

lib/Format/ContinuationIndenter.cpp

1325–1326 ↗

(On Diff #109554)

I agree that the performance gain aren't worth the additional effort here. We would need a cache to prevent computing the same breaks in various paths of the combinatorial formatting, but I haven't studied the code enough to know whether we have this separately yet.

On maintainability:
We similarly don't do this in other places (breakProtrudingToken, formatChildren, etc.) and so I think doing this differently for raw string literals will make things less maintainable. Moreover, I see your point that caching makes it in some way easier to see that we are doing the exact same thing in the second phase. At the same time ensuring that caching is correct might be hard to maintain. E.g. finding bugs that are caused by us storing incorrect information in the cache or something might be extraordinarily hard to track down.

[krasimir]

• Fix a regression after preprocessor indentation

[krasimir]

• Remove raw string cache

[krasimir]

• Document FirstStateColumn @ getInitialState

[krasimir]

I think this is ready for another round of reviews.

[krasimir]

ping

[krasimir]

ping

[klimek]

Needs a bit more comments, otherwise LG.

include/clang/Format/Format.h

1730–1734 ↗

(On Diff #114555)

Needs a comment. Also, this should probably go into an internal namespace somewhere below all the public interfaces?

lib/Format/ContinuationIndenter.cpp

1324 ↗

(On Diff #114555)

No else after return.

lib/Format/TokenAnalyzer.h

75–79 ↗

(On Diff #114555)

I think you'll want to document these.

[krasimir]

• Address review comments

[krasimir]

include/clang/Format/Format.h

1730–1734 ↗

(On Diff #114555)

Good point. Moved to a new internal header FormatInternal.h and added documentation.

[krasimir]

I'll need a bit of refactoring on the test side of things, since old gcc doesn't like raw string literals in macros.

[klimek]

lib/Format/FormatInternal.h

27–28 ↗

(On Diff #120481)

"and at which column the fragment's last line should end".

I don't understand the "if it is a newline" part.

29–30 ↗

(On Diff #120481)

"specifies the column at which the first line of \p Code starts".

33–34 ↗

(On Diff #120481)

"the column at which the last line of \p Code should end".

Also, s/it/if/. Does "just a newline" mean "is empty"?

[krasimir]

• Improve comments

[krasimir]

• Refactor tests to use a function instead of a macro

[krasimir]

• Update example

cfe/trunk/include/clang/Format/Format.h

enum PointerAlignmentStyle {
     ///   int * a;
     /// \endcode
     PAS_Middle
   };
 
   /// \brief Pointer and reference alignment style.
   PointerAlignmentStyle PointerAlignment;
 
+  /// See documentation of ``RawStringFormats``.
+  struct RawStringFormat {
+    /// \brief The delimiter that this raw string format matches.
+    std::string Delimiter;
+    /// \brief The language of this raw string.
+    LanguageKind Language;
+    /// \brief The style name on which this raw string format is based on.
+    /// If not specified, the raw string format is based on the style that this
+    /// format is based on.
+    std::string BasedOnStyle;
+    bool operator==(const RawStringFormat &Other) const {
+      return Delimiter == Other.Delimiter && Language == Other.Language &&
+             BasedOnStyle == Other.BasedOnStyle;
+    }
+  };
+
+  /// \brief Raw string delimiters denoting that the raw string contents are
+  /// code in a particular language and can be reformatted.
+  ///
+  /// A raw string with a matching delimiter will be reformatted assuming the
+  /// specified language based on a predefined style given by 'BasedOnStyle'.
+  /// If 'BasedOnStyle' is not found, the formatting is based on llvm style.
+  ///
+  /// To configure this in the .clang-format file, use:
+  /// \code{.yaml}
+  ///   RawStringFormats:
+  ///     - Delimiter: 'pb'
+  ///       Language:  TextProto
+  ///       BasedOnStyle: llvm
+  ///     - Delimiter: 'proto'
+  ///       Language:  TextProto
+  ///       BasedOnStyle: google
+  /// \endcode
+  std::vector<RawStringFormat> RawStringFormats;
+
   /// \brief If ``true``, clang-format will attempt to re-flow comments.
   /// \code
   ///    false:
   ///    // veryVeryVeryVeryVeryVeryVeryVeryVeryVeryVeryLongComment with plenty of information
   ///    /* second veryVeryVeryVeryVeryVeryVeryVeryVeryVeryVeryLongComment with plenty of information */
   ///
   ///    true:
   ///    // veryVeryVeryVeryVeryVeryVeryVeryVeryVeryVeryLongComment with plenty of
return AccessModifierOffset == R.AccessModifierOffset &&
            PenaltyBreakBeforeFirstCallParameter ==
                R.PenaltyBreakBeforeFirstCallParameter &&
            PenaltyBreakComment == R.PenaltyBreakComment &&
            PenaltyBreakFirstLessLess == R.PenaltyBreakFirstLessLess &&
            PenaltyBreakString == R.PenaltyBreakString &&
            PenaltyExcessCharacter == R.PenaltyExcessCharacter &&
            PenaltyReturnTypeOnItsOwnLine == R.PenaltyReturnTypeOnItsOwnLine &&
            PointerAlignment == R.PointerAlignment &&
+           RawStringFormats == R.RawStringFormats &&
            SpaceAfterCStyleCast == R.SpaceAfterCStyleCast &&
            SpaceAfterTemplateKeyword == R.SpaceAfterTemplateKeyword &&
            SpaceBeforeAssignmentOperators == R.SpaceBeforeAssignmentOperators &&
            SpaceBeforeParens == R.SpaceBeforeParens &&
            SpaceInEmptyParentheses == R.SpaceInEmptyParentheses &&
            SpacesBeforeTrailingComments == R.SpacesBeforeTrailingComments &&
            SpacesInAngles == R.SpacesInAngles &&
            SpacesInContainerLiterals == R.SpacesInContainerLiterals &&

cfe/trunk/lib/Format/ContinuationIndenter.h

 
 #ifndef LLVM_CLANG_LIB_FORMAT_CONTINUATIONINDENTER_H
 #define LLVM_CLANG_LIB_FORMAT_CONTINUATIONINDENTER_H
 
 #include "Encoding.h"
 #include "FormatToken.h"
 #include "clang/Format/Format.h"
 #include "llvm/Support/Regex.h"
+#include <map>
+#include <tuple>
 
 namespace clang {
 class SourceManager;
 
 namespace format {
 
 class AnnotatedLine;
 struct FormatToken;
 struct LineState;
 struct ParenState;
+struct RawStringFormatStyleManager;
 class WhitespaceManager;
 
+struct RawStringFormatStyleManager {
+  llvm::StringMap<FormatStyle> DelimiterStyle;
+
+  RawStringFormatStyleManager(const FormatStyle &CodeStyle);
+
+  llvm::Optional<FormatStyle> get(StringRef Delimiter) const;
+};
+
 class ContinuationIndenter {
 public:
   /// \brief Constructs a \c ContinuationIndenter to format \p Line starting in
   /// column \p FirstIndent.
   ContinuationIndenter(const FormatStyle &Style,
                        const AdditionalKeywords &Keywords,
                        const SourceManager &SourceMgr,
                        WhitespaceManager &Whitespaces,
                        encoding::Encoding Encoding,
                        bool BinPackInconclusiveFunctions);
 
   /// \brief Get the initial state, i.e. the state after placing \p Line's
-  /// first token at \p FirstIndent.
-  LineState getInitialState(unsigned FirstIndent, const AnnotatedLine *Line,
-                            bool DryRun);
+  /// first token at \p FirstIndent. When reformatting a fragment of code, as in
+  /// the case of formatting inside raw string literals, \p FirstStartColumn is
+  /// the column at which the state of the parent formatter is.
+  LineState getInitialState(unsigned FirstIndent, unsigned FirstStartColumn,
+                            const AnnotatedLine *Line, bool DryRun);
 
   // FIXME: canBreak and mustBreak aren't strictly indentation-related. Find a
   // better home.
   /// \brief Returns \c true, if a line break after \p State is allowed.
   bool canBreak(const LineState &State);
 
   /// \brief Returns \c true, if a line break after \p State is mandatory.
   bool mustBreak(const LineState &State);
private:
 
   /// \brief Update 'State' according to the next token being one of "(<{[".
   void moveStatePastScopeOpener(LineState &State, bool Newline);
   /// \brief Update 'State' according to the next token being one of ")>}]".
   void moveStatePastScopeCloser(LineState &State);
   /// \brief Update 'State' with the next token opening a nested block.
   void moveStateToNewBlock(LineState &State);
 
+  /// \brief Reformats a raw string literal.
+  /// 
+  /// \returns An extra penalty induced by reformatting the token.
+  unsigned reformatRawStringLiteral(const FormatToken &Current,
+                                    unsigned StartColumn, LineState &State,
+                                    StringRef Delimiter,
+                                    const FormatStyle &RawStringStyle,
+                                    bool DryRun);
+
   /// \brief If the current token sticks out over the end of the line, break
   /// it if possible.
   ///
   /// \returns An extra penalty if a token was broken, otherwise 0.
   ///
-  /// The returned penalty will cover the cost of the additional line breaks and
-  /// column limit violation in all lines except for the last one. The penalty
-  /// for the column limit violation in the last line (and in single line
-  /// tokens) is handled in \c addNextStateToQueue.
+  /// The returned penalty will cover the cost of the additional line breaks
+  /// and column limit violation in all lines except for the last one. The
+  /// penalty for the column limit violation in the last line (and in single
+  /// line tokens) is handled in \c addNextStateToQueue.
   unsigned breakProtrudingToken(const FormatToken &Current, LineState &State,
                                 bool DryRun);
 
   /// \brief Appends the next token to \p State and updates information
   /// necessary for indentation.
   ///
   /// Puts the token on the current line.
   ///
private:
 
   FormatStyle Style;
   const AdditionalKeywords &Keywords;
   const SourceManager &SourceMgr;
   WhitespaceManager &Whitespaces;
   encoding::Encoding Encoding;
   bool BinPackInconclusiveFunctions;
   llvm::Regex CommentPragmasRegex;
+  const RawStringFormatStyleManager RawStringFormats;
 };
 
 struct ParenState {
   ParenState(unsigned Indent, unsigned LastSpace, bool AvoidBinPacking,
              bool NoLineBreak)
       : Indent(Indent), LastSpace(LastSpace), NestedBlockIndent(Indent),
         BreakBeforeClosingBrace(false), AvoidBinPacking(AvoidBinPacking),
         BreakBeforeParameter(false), NoLineBreak(NoLineBreak),

cfe/trunk/lib/Format/ContinuationIndenter.cpp

 //===--- ContinuationIndenter.cpp - Format C++ code -----------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 ///
 /// \file
 /// \brief This file implements the continuation indenter.
 ///
 //===----------------------------------------------------------------------===//
 
 #include "ContinuationIndenter.h"
 #include "BreakableToken.h"
+#include "FormatInternal.h"
 #include "WhitespaceManager.h"
 #include "clang/Basic/OperatorPrecedence.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Format/Format.h"
 #include "llvm/Support/Debug.h"
 
 #define DEBUG_TYPE "format-indenter"
 
static bool opensProtoMessageField(const FormatToken &LessTok,
   if (LessTok.isNot(tok::less))
     return false;
   return Style.Language == FormatStyle::LK_TextProto ||
          (Style.Language == FormatStyle::LK_Proto &&
           (LessTok.NestingLevel > 0 ||
            (LessTok.Previous && LessTok.Previous->is(tok::equal))));
 }
 
+// Returns the delimiter of a raw string literal, or None if TokenText is not
+// the text of a raw string literal. The delimiter could be the empty string.
+// For example, the delimiter of R"deli(cont)deli" is deli.
+static llvm::Optional<StringRef> getRawStringDelimiter(StringRef TokenText) {
+  if (TokenText.size() < 5 // The smallest raw string possible is 'R"()"'.
+      || !TokenText.startswith("R\"") || !TokenText.endswith("\""))
+    return None;
+
+  // A raw string starts with 'R"<delimiter>(' and delimiter is ascii and has
+  // size at most 16 by the standard, so the first '(' must be among the first
+  // 19 bytes.
+  size_t LParenPos = TokenText.substr(0, 19).find_first_of('(');
+  if (LParenPos == StringRef::npos)
+    return None;
+  StringRef Delimiter = TokenText.substr(2, LParenPos - 2);
+
+  // Check that the string ends in ')Delimiter"'.
+  size_t RParenPos = TokenText.size() - Delimiter.size() - 2;
+  if (TokenText[RParenPos] != ')')
+    return None;
+  if (!TokenText.substr(RParenPos + 1).startswith(Delimiter))
+    return None;
+  return Delimiter;
+}
+
+RawStringFormatStyleManager::RawStringFormatStyleManager(
+    const FormatStyle &CodeStyle) {
+  for (const auto &RawStringFormat : CodeStyle.RawStringFormats) {
+    FormatStyle Style;
+    if (!getPredefinedStyle(RawStringFormat.BasedOnStyle,
+                            RawStringFormat.Language, &Style)) {
+      Style = getLLVMStyle();
+      Style.Language = RawStringFormat.Language;
+    }
+    Style.ColumnLimit = CodeStyle.ColumnLimit;
+    DelimiterStyle.insert({RawStringFormat.Delimiter, Style});
+  }
+}
+
+llvm::Optional<FormatStyle>
+RawStringFormatStyleManager::get(StringRef Delimiter) const {
+  auto It = DelimiterStyle.find(Delimiter);
+  if (It == DelimiterStyle.end())
+    return None;
+  return It->second;
+}
+
 ContinuationIndenter::ContinuationIndenter(const FormatStyle &Style,
                                            const AdditionalKeywords &Keywords,
                                            const SourceManager &SourceMgr,
                                            WhitespaceManager &Whitespaces,
                                            encoding::Encoding Encoding,
                                            bool BinPackInconclusiveFunctions)
     : Style(Style), Keywords(Keywords), SourceMgr(SourceMgr),
       Whitespaces(Whitespaces), Encoding(Encoding),
       BinPackInconclusiveFunctions(BinPackInconclusiveFunctions),
-      CommentPragmasRegex(Style.CommentPragmas) {}
+      CommentPragmasRegex(Style.CommentPragmas), RawStringFormats(Style) {}
 
 LineState ContinuationIndenter::getInitialState(unsigned FirstIndent,
+                                                unsigned FirstStartColumn,
                                                 const AnnotatedLine *Line,
                                                 bool DryRun) {
   LineState State;
   State.FirstIndent = FirstIndent;
+  if (FirstStartColumn && Line->First->NewlinesBefore == 0)
+    State.Column = FirstStartColumn;
+  else
     State.Column = FirstIndent;
   // With preprocessor directive indentation, the line starts on column 0
   // since it's indented after the hash, but FirstIndent is set to the
   // preprocessor indent.
   if (Style.IndentPPDirectives == FormatStyle::PPDIS_AfterHash &&
       (Line->Type == LT_PreprocessorDirective ||
        Line->Type == LT_ImportStatement))
     State.Column = 0;
   State.Line = Line;
unsigned NewIndent =
                                : Style.IndentWidth);
   State.Stack.push_back(ParenState(NewIndent, State.Stack.back().LastSpace,
                                    /*AvoidBinPacking=*/true,
                                    /*NoLineBreak=*/false));
   State.Stack.back().NestedBlockIndent = NestedBlockIndent;
   State.Stack.back().BreakBeforeParameter = true;
 }
 
+static unsigned getLastLineEndColumn(StringRef Text, unsigned StartColumn,
+                                     unsigned TabWidth,
+                                     encoding::Encoding Encoding) {
+  size_t LastNewlinePos = Text.find_last_of("\n");
+  if (LastNewlinePos == StringRef::npos) {
+    return StartColumn +
+           encoding::columnWidthWithTabs(Text, StartColumn, TabWidth, Encoding);
+  } else {
+    return encoding::columnWidthWithTabs(Text.substr(LastNewlinePos),
+                                         /*StartColumn=*/0, TabWidth, Encoding);
+  }
+}
+
+unsigned ContinuationIndenter::reformatRawStringLiteral(
+    const FormatToken &Current, unsigned StartColumn, LineState &State,
+    StringRef Delimiter, const FormatStyle &RawStringStyle, bool DryRun) {
+  // The text of a raw string is between the leading 'R"delimiter(' and the
+  // trailing 'delimiter)"'.
+  unsigned PrefixSize = 3 + Delimiter.size();
+  unsigned SuffixSize = 2 + Delimiter.size();
+
+  // The first start column is the column the raw text starts.
+  unsigned FirstStartColumn = StartColumn + PrefixSize;
+
+  // The next start column is the intended indentation a line break inside
+  // the raw string at level 0. It is determined by the following rules:
+  //   - if the content starts on newline, it is one level more than the current
+  //     indent, and
+  //   - if the content does not start on a newline, it is the first start
+  //     column.
+  // These rules have the advantage that the formatted content both does not
+  // violate the rectangle rule and visually flows within the surrounding
+  // source.
+  bool ContentStartsOnNewline = Current.TokenText[PrefixSize] == '\n';
+  unsigned NextStartColumn = ContentStartsOnNewline
+                                 ? State.Stack.back().Indent + Style.IndentWidth
+                                 : FirstStartColumn;
+
+  // The last start column is the column the raw string suffix starts if it is
+  // put on a newline.
+  // The last start column is the intended indentation of the raw string postfix
+  // if it is put on a newline. It is determined by the following rules:
+  //   - if the raw string prefix starts on a newline, it is the column where
+  //     that raw string prefix starts, and
+  //   - if the raw string prefix does not start on a newline, it is the current
+  //     indent.
+  unsigned LastStartColumn = Current.NewlinesBefore
+                                 ? FirstStartColumn - PrefixSize
+                                 : State.Stack.back().Indent;
+
+  std::string RawText =
+      Current.TokenText.substr(PrefixSize).drop_back(SuffixSize);
+
+  std::pair<tooling::Replacements, unsigned> Fixes = internal::reformat(
+      RawStringStyle, RawText, {tooling::Range(0, RawText.size())},
+      FirstStartColumn, NextStartColumn, LastStartColumn, "<stdin>",
+      /*FormattingAttemptStatus=*/nullptr);
+
+  auto NewCode = applyAllReplacements(RawText, Fixes.first);
+  tooling::Replacements NoFixes;
+  if (!NewCode) {
+    State.Column += Current.ColumnWidth;
+    return 0;
+  }
+  if (!DryRun) {
+    SourceLocation OriginLoc =
+        Current.Tok.getLocation().getLocWithOffset(PrefixSize);
+    for (const tooling::Replacement &Fix : Fixes.first) {
+      auto Err = Whitespaces.addReplacement(tooling::Replacement(
+          SourceMgr, OriginLoc.getLocWithOffset(Fix.getOffset()),
+          Fix.getLength(), Fix.getReplacementText()));
+      if (Err) {
+        llvm::errs() << "Failed to reformat raw string: "
+                     << llvm::toString(std::move(Err)) << "\n";
+      }
+    }
+  }
+  unsigned RawLastLineEndColumn = getLastLineEndColumn(
+      *NewCode, FirstStartColumn, Style.TabWidth, Encoding);
+  State.Column = RawLastLineEndColumn + SuffixSize;
+  return Fixes.second;
+}
+
 unsigned ContinuationIndenter::addMultilineToken(const FormatToken &Current,
                                                  LineState &State) {
   if (!Current.IsMultiline)
     return 0;
 
   // Break before further function parameters on all levels.
   for (unsigned i = 0, e = State.Stack.size(); i != e; ++i)
     State.Stack[i].BreakBeforeParameter = true;
 
   unsigned ColumnsUsed = State.Column;
   // We can only affect layout of the first and the last line, so the penalty
   // for all other lines is constant, and we ignore it.
   State.Column = Current.LastLineColumnWidth;
 
   if (ColumnsUsed > getColumnLimit(State))
     return Style.PenaltyExcessCharacter * (ColumnsUsed - getColumnLimit(State));
   return 0;
 }
 
 unsigned ContinuationIndenter::breakProtrudingToken(const FormatToken &Current,
                                                     LineState &State,
                                                     bool DryRun) {
-  // Don't break multi-line tokens other than block comments. Instead, just
-  // update the state.
-  if (Current.isNot(TT_BlockComment) && Current.IsMultiline)
+  // Compute the raw string style to use in case this is a raw string literal
+  // that can be reformatted.
+  llvm::Optional<StringRef> Delimiter = None;
+  llvm::Optional<FormatStyle> RawStringStyle = None;
+  if (Current.isStringLiteral())
+    Delimiter = getRawStringDelimiter(Current.TokenText);
+  if (Delimiter)
+    RawStringStyle = RawStringFormats.get(*Delimiter);
+
+  // Don't break multi-line tokens other than block comments and raw string
+  // literals. Instead, just update the state.
+  if (Current.isNot(TT_BlockComment) && !RawStringStyle && Current.IsMultiline)
     return addMultilineToken(Current, State);
 
   // Don't break implicit string literals or import statements.
   if (Current.is(TT_ImplicitStringLiteral) ||
       State.Line->Type == LT_ImportStatement)
     return 0;
 
   if (!Current.isStringLiteral() && !Current.is(tok::comment))
if (Current.isStringLiteral()) {
     // preprocessor directives that contain long string literals.
     if (State.Line->Type == LT_PreprocessorDirective)
       return 0;
     // Exempts unterminated string literals from line breaking. The user will
     // likely want to terminate the string before any line breaking is done.
     if (Current.IsUnterminatedLiteral)
       return 0;
 
+    if (RawStringStyle) {
+      RawStringStyle->ColumnLimit = ColumnLimit;
+      return reformatRawStringLiteral(Current, StartColumn, State, *Delimiter,
+                                      *RawStringStyle, DryRun);
+    } 
     StringRef Text = Current.TokenText;
     StringRef Prefix;
     StringRef Postfix;
     // FIXME: Handle whitespace between '_T', '(', '"..."', and ')'.
     // FIXME: Store Prefix and Suffix (or PrefixLength and SuffixLength to
     // reduce the overhead) for each FormatToken, which is a string, so that we
     // don't run multiple checks here on the hot path.
     if ((Text.endswith(Postfix = "\"") &&

cfe/trunk/lib/Format/Format.cpp

 /// \brief This file implements functions declared in Format.h. This will be
 /// split into separate files as we go.
 ///
 //===----------------------------------------------------------------------===//
 
 #include "clang/Format/Format.h"
 #include "AffectedRangeManager.h"
 #include "ContinuationIndenter.h"
+#include "FormatInternal.h"
 #include "FormatTokenLexer.h"
 #include "NamespaceEndCommentsFixer.h"
 #include "SortJavaScriptImports.h"
 #include "TokenAnalyzer.h"
 #include "TokenAnnotator.h"
 #include "UnwrappedLineFormatter.h"
 #include "UnwrappedLineParser.h"
 #include "UsingDeclarationsSorter.h"

 #include <algorithm>
 #include <memory>
 #include <string>
 
 #define DEBUG_TYPE "format-formatter"
 
 using clang::format::FormatStyle;
 
-LLVM_YAML_IS_SEQUENCE_VECTOR(clang::format::FormatStyle::IncludeCategory)
+LLVM_YAML_IS_SEQUENCE_VECTOR(clang::format::FormatStyle::IncludeCategory);
+LLVM_YAML_IS_SEQUENCE_VECTOR(clang::format::FormatStyle::RawStringFormat);
 
 namespace llvm {
 namespace yaml {
 template <> struct ScalarEnumerationTraits<FormatStyle::LanguageKind> {
   static void enumeration(IO &IO, FormatStyle::LanguageKind &Value) {
     IO.enumCase(Value, "Cpp", FormatStyle::LK_Cpp);
     IO.enumCase(Value, "Java", FormatStyle::LK_Java);
     IO.enumCase(Value, "JavaScript", FormatStyle::LK_JavaScript);
static void mapping(IO &IO, FormatStyle &Style) {
     IO.mapOptional("PenaltyBreakComment", Style.PenaltyBreakComment);
     IO.mapOptional("PenaltyBreakFirstLessLess",
                    Style.PenaltyBreakFirstLessLess);
     IO.mapOptional("PenaltyBreakString", Style.PenaltyBreakString);
     IO.mapOptional("PenaltyExcessCharacter", Style.PenaltyExcessCharacter);
     IO.mapOptional("PenaltyReturnTypeOnItsOwnLine",
                    Style.PenaltyReturnTypeOnItsOwnLine);
     IO.mapOptional("PointerAlignment", Style.PointerAlignment);
+    IO.mapOptional("RawStringFormats", Style.RawStringFormats);
     IO.mapOptional("ReflowComments", Style.ReflowComments);
     IO.mapOptional("SortIncludes", Style.SortIncludes);
     IO.mapOptional("SortUsingDeclarations", Style.SortUsingDeclarations);
     IO.mapOptional("SpaceAfterCStyleCast", Style.SpaceAfterCStyleCast);
     IO.mapOptional("SpaceAfterTemplateKeyword",
                    Style.SpaceAfterTemplateKeyword);
     IO.mapOptional("SpaceBeforeAssignmentOperators",
                    Style.SpaceBeforeAssignmentOperators);

 
 template <> struct MappingTraits<FormatStyle::IncludeCategory> {
   static void mapping(IO &IO, FormatStyle::IncludeCategory &Category) {
     IO.mapOptional("Regex", Category.Regex);
     IO.mapOptional("Priority", Category.Priority);
   }
 };
 
+template <> struct MappingTraits<FormatStyle::RawStringFormat> {
+  static void mapping(IO &IO, FormatStyle::RawStringFormat &Format) {
+    IO.mapOptional("Delimiter", Format.Delimiter);
+    IO.mapOptional("Language", Format.Language);
+    IO.mapOptional("BasedOnStyle", Format.BasedOnStyle);
+  }
+};
+
 // Allows to read vector<FormatStyle> while keeping default values.
 // IO.getContext() should contain a pointer to the FormatStyle structure, that
 // will be used to get default values for missing keys.
 // If the first element has no Language specified, it will be treated as the
 // default one for the following elements.
 template <> struct DocumentListTraits<std::vector<FormatStyle>> {
   static size_t size(IO &IO, std::vector<FormatStyle> &Seq) {
     return Seq.size();
FormatStyle getLLVMStyle() {
   LLVMStyle.NamespaceIndentation = FormatStyle::NI_None;
   LLVMStyle.ObjCBlockIndentWidth = 2;
   LLVMStyle.ObjCSpaceAfterProperty = false;
   LLVMStyle.ObjCSpaceBeforeProtocolList = true;
   LLVMStyle.PointerAlignment = FormatStyle::PAS_Right;
   LLVMStyle.SpacesBeforeTrailingComments = 1;
   LLVMStyle.Standard = FormatStyle::LS_Cpp11;
   LLVMStyle.UseTab = FormatStyle::UT_Never;
+  LLVMStyle.RawStringFormats = {{"pb", FormatStyle::LK_TextProto, "google"}};
   LLVMStyle.ReflowComments = true;
   LLVMStyle.SpacesInParentheses = false;
   LLVMStyle.SpacesInSquareBrackets = false;
   LLVMStyle.SpaceInEmptyParentheses = false;
   LLVMStyle.SpacesInContainerLiterals = true;
   LLVMStyle.SpacesInCStyleCastParentheses = false;
   LLVMStyle.SpaceAfterCStyleCast = false;
   LLVMStyle.SpaceAfterTemplateKeyword = true;

 
 namespace {
 
 class JavaScriptRequoter : public TokenAnalyzer {
 public:
   JavaScriptRequoter(const Environment &Env, const FormatStyle &Style)
       : TokenAnalyzer(Env, Style) {}
 
-  tooling::Replacements
+  std::pair<tooling::Replacements, unsigned>
   analyze(TokenAnnotator &Annotator,
           SmallVectorImpl<AnnotatedLine *> &AnnotatedLines,
           FormatTokenLexer &Tokens) override {
     AffectedRangeMgr.computeAffectedLines(AnnotatedLines.begin(),
                                           AnnotatedLines.end());
     tooling::Replacements Result;
     requoteJSStringLiteral(AnnotatedLines, Result);
-    return Result;
+    return {Result, 0};
   }
 
 private:
   // Replaces double/single-quoted string literal as appropriate, re-escaping
   // the contents in the process.
   void requoteJSStringLiteral(SmallVectorImpl<AnnotatedLine *> &Lines,
                               tooling::Replacements &Result) {
     for (AnnotatedLine *Line : Lines) {

 };
 
 class Formatter : public TokenAnalyzer {
 public:
   Formatter(const Environment &Env, const FormatStyle &Style,
             FormattingAttemptStatus *Status)
       : TokenAnalyzer(Env, Style), Status(Status) {}
 
-  tooling::Replacements
+  std::pair<tooling::Replacements, unsigned>
   analyze(TokenAnnotator &Annotator,
           SmallVectorImpl<AnnotatedLine *> &AnnotatedLines,
           FormatTokenLexer &Tokens) override {
     tooling::Replacements Result;
     deriveLocalStyle(AnnotatedLines);
     AffectedRangeMgr.computeAffectedLines(AnnotatedLines.begin(),
                                           AnnotatedLines.end());
     for (unsigned i = 0, e = AnnotatedLines.size(); i != e; ++i) {
       Annotator.calculateFormattingInformation(*AnnotatedLines[i]);
     }
     Annotator.setCommentLineLevels(AnnotatedLines);
 
     WhitespaceManager Whitespaces(
         Env.getSourceManager(), Style,
         inputUsesCRLF(Env.getSourceManager().getBufferData(Env.getFileID())));
     ContinuationIndenter Indenter(Style, Tokens.getKeywords(),
                                   Env.getSourceManager(), Whitespaces, Encoding,
                                   BinPackInconclusiveFunctions);
-    UnwrappedLineFormatter(&Indenter, &Whitespaces, Style, Tokens.getKeywords(),
-                           Env.getSourceManager(), Status)
-        .format(AnnotatedLines);
+    unsigned Penalty =
+        UnwrappedLineFormatter(&Indenter, &Whitespaces, Style,
+                               Tokens.getKeywords(), Env.getSourceManager(),
+                               Status)
+            .format(AnnotatedLines, /*DryRun=*/false,
+                    /*AdditionalIndent=*/0,
+                    /*FixBadIndentation=*/false,
+                    /*FirstStartColumn=*/Env.getFirstStartColumn(),
+                    /*NextStartColumn=*/Env.getNextStartColumn(),
+                    /*LastStartColumn=*/Env.getLastStartColumn());
     for (const auto &R : Whitespaces.generateReplacements())
       if (Result.add(R))
-        return Result;
-    return Result;
+        return {Result, 0};
+    return {Result, Penalty};
   }
 
 private:
   static bool inputUsesCRLF(StringRef Text) {
     return Text.count('\r') * 2 > Text.count('\n');
   }
 
   bool

 // file.
 class Cleaner : public TokenAnalyzer {
 public:
   Cleaner(const Environment &Env, const FormatStyle &Style)
       : TokenAnalyzer(Env, Style),
         DeletedTokens(FormatTokenLess(Env.getSourceManager())) {}
 
   // FIXME: eliminate unused parameters.
-  tooling::Replacements
+  std::pair<tooling::Replacements, unsigned>
   analyze(TokenAnnotator &Annotator,
           SmallVectorImpl<AnnotatedLine *> &AnnotatedLines,
           FormatTokenLexer &Tokens) override {
     // FIXME: in the current implementation the granularity of affected range
     // is an annotated line. However, this is not sufficient. Furthermore,
     // redundant code introduced by replacements does not necessarily
     // intercept with ranges of replacements that result in the redundancy.
     // To determine if some redundant code is actually introduced by
for (auto &Line : AnnotatedLines) {
         cleanupRight(Line->First, tok::l_paren, tok::comma);
         cleanupLeft(Line->First, tok::comma, tok::r_paren);
         cleanupLeft(Line->First, TT_CtorInitializerComma, tok::l_brace);
         cleanupLeft(Line->First, TT_CtorInitializerColon, tok::l_brace);
         cleanupLeft(Line->First, TT_CtorInitializerColon, tok::equal);
       }
     }
 
-    return generateFixes();
+    return {generateFixes(), 0};
   }
 
 private:
   bool containsOnlyComments(const AnnotatedLine &Line) {
     for (FormatToken *Tok = Line.First; Tok != nullptr; Tok = Tok->Next) {
       if (Tok->isNot(tok::comment))
         return false;
     }
auto Cleanup = [](const FormatStyle &Style, StringRef Code,
     return cleanup(Style, Code, Ranges, FileName);
   };
   // Make header insertion replacements insert new headers into correct blocks.
   tooling::Replacements NewReplaces =
       fixCppIncludeInsertions(Code, Replaces, Style);
   return processReplacements(Cleanup, Code, NewReplaces, Style);
 }
 
-tooling::Replacements reformat(const FormatStyle &Style, StringRef Code,
-                               ArrayRef<tooling::Range> Ranges,
-                               StringRef FileName,
-                               FormattingAttemptStatus *Status) {
+namespace internal {
+std::pair<tooling::Replacements, unsigned>
+reformat(const FormatStyle &Style, StringRef Code,
+         ArrayRef<tooling::Range> Ranges, unsigned FirstStartColumn,
+         unsigned NextStartColumn, unsigned LastStartColumn, StringRef FileName,
   FormatStyle Expanded = expandPresets(Style);
   if (Expanded.DisableFormat)
-    return tooling::Replacements();
+    return {tooling::Replacements(), 0};
   if (isLikelyXml(Code))
-    return tooling::Replacements();
+    return {tooling::Replacements(), 0};
   if (Expanded.Language == FormatStyle::LK_JavaScript && isMpegTS(Code))
-    return tooling::Replacements();
+    return {tooling::Replacements(), 0};
 
-  typedef std::function<tooling::Replacements(const Environment &)>
+  typedef std::function<std::pair<tooling::Replacements, unsigned>(
+      const Environment &)>
       AnalyzerPass;
   SmallVector<AnalyzerPass, 4> Passes;
 
   if (Style.Language == FormatStyle::LK_Cpp) {
     if (Style.FixNamespaceComments)
       Passes.emplace_back([&](const Environment &Env) {
         return NamespaceEndCommentsFixer(Env, Expanded).process();
       });
if (Style.Language == FormatStyle::LK_JavaScript &&
     Passes.emplace_back([&](const Environment &Env) {
       return JavaScriptRequoter(Env, Expanded).process();
     });
 
   Passes.emplace_back([&](const Environment &Env) {
     return Formatter(Env, Expanded, Status).process();
   });
 
-  std::unique_ptr<Environment> Env =
-      Environment::CreateVirtualEnvironment(Code, FileName, Ranges);
+  std::unique_ptr<Environment> Env = Environment::CreateVirtualEnvironment(
+      Code, FileName, Ranges, FirstStartColumn, NextStartColumn,
+      LastStartColumn);
   llvm::Optional<std::string> CurrentCode = None;
   tooling::Replacements Fixes;
+  unsigned Penalty = 0;
   for (size_t I = 0, E = Passes.size(); I < E; ++I) {
-    tooling::Replacements PassFixes = Passes[I](*Env);
+    std::pair<tooling::Replacements, unsigned> PassFixes = Passes[I](*Env);
     auto NewCode = applyAllReplacements(
-        CurrentCode ? StringRef(*CurrentCode) : Code, PassFixes);
+        CurrentCode ? StringRef(*CurrentCode) : Code, PassFixes.first);
     if (NewCode) {
-      Fixes = Fixes.merge(PassFixes);
+      Fixes = Fixes.merge(PassFixes.first);
+      Penalty += PassFixes.second;
       if (I + 1 < E) {
         CurrentCode = std::move(*NewCode);
         Env = Environment::CreateVirtualEnvironment(
             *CurrentCode, FileName,
-            tooling::calculateRangesAfterReplacements(Fixes, Ranges));
+            tooling::calculateRangesAfterReplacements(Fixes, Ranges),
+            FirstStartColumn, NextStartColumn, LastStartColumn);
       }
     }
   }
 
-  return Fixes;
+  return {Fixes, Penalty};
+}
+} // namespace internal
+
+tooling::Replacements reformat(const FormatStyle &Style, StringRef Code,
+                               ArrayRef<tooling::Range> Ranges,
+                               StringRef FileName,
+                               FormattingAttemptStatus *Status) {
+  return internal::reformat(Style, Code, Ranges,
+                            /*FirstStartColumn=*/0,
+                            /*NextStartColumn=*/0,
+                            /*LastStartColumn=*/0, FileName, Status)
+      .first;
 }
 
 tooling::Replacements cleanup(const FormatStyle &Style, StringRef Code,
                               ArrayRef<tooling::Range> Ranges,
                               StringRef FileName) {
   // cleanups only apply to C++ (they mostly concern ctor commas etc.)
   if (Style.Language != FormatStyle::LK_Cpp)
     return tooling::Replacements();
   std::unique_ptr<Environment> Env =
       Environment::CreateVirtualEnvironment(Code, FileName, Ranges);
   Cleaner Clean(*Env, Style);
-  return Clean.process();
+  return Clean.process().first;
 }
 
 tooling::Replacements reformat(const FormatStyle &Style, StringRef Code,
                                ArrayRef<tooling::Range> Ranges,
                                StringRef FileName, bool *IncompleteFormat) {
   FormattingAttemptStatus Status;
   auto Result = reformat(Style, Code, Ranges, FileName, &Status);
   if (!Status.FormatComplete)
     *IncompleteFormat = true;
   return Result;
 }
 
 tooling::Replacements fixNamespaceEndComments(const FormatStyle &Style,
                                               StringRef Code,
                                               ArrayRef<tooling::Range> Ranges,
                                               StringRef FileName) {
   std::unique_ptr<Environment> Env =
       Environment::CreateVirtualEnvironment(Code, FileName, Ranges);
   NamespaceEndCommentsFixer Fix(*Env, Style);
-  return Fix.process();
+  return Fix.process().first;
 }
 
 tooling::Replacements sortUsingDeclarations(const FormatStyle &Style,
                                             StringRef Code,
                                             ArrayRef<tooling::Range> Ranges,
                                             StringRef FileName) {
   std::unique_ptr<Environment> Env =
       Environment::CreateVirtualEnvironment(Code, FileName, Ranges);
   UsingDeclarationsSorter Sorter(*Env, Style);
-  return Sorter.process();
+  return Sorter.process().first;
 }
 
 LangOptions getFormattingLangOpts(const FormatStyle &Style) {
   LangOptions LangOpts;
   LangOpts.CPlusPlus = 1;
   LangOpts.CPlusPlus11 = Style.Standard == FormatStyle::LS_Cpp03 ? 0 : 1;
   LangOpts.CPlusPlus14 = Style.Standard == FormatStyle::LS_Cpp03 ? 0 : 1;
   LangOpts.CPlusPlus1z = Style.Standard == FormatStyle::LS_Cpp03 ? 0 : 1;

cfe/trunk/lib/Format/FormatInternal.h

+//===--- FormatInternal.h - Format C++ code ---------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// \brief This file declares Format APIs to be used internally by the
+/// formatting library implementation.
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_LIB_FORMAT_FORMATINTERNAL_H
+#define LLVM_CLANG_LIB_FORMAT_FORMATINTERNAL_H
+
+namespace clang {
+namespace format {
+namespace internal {
+
+/// \brief Reformats the given \p Ranges in the code fragment \p Code.
+///
+/// A fragment of code could conceptually be surrounded by other code that might
+/// constrain how that fragment is laid out.
+/// For example, consider the fragment of code between 'R"(' and ')"',
+/// exclusive, in the following code:
+///
+/// void outer(int x) {
+///   string inner = R"(name: data
+///                     ^ FirstStartColumn
+///     value: {
+///       x: 1
+///     ^ NextStartColumn
+///     }
+///   )";
+///   ^ LastStartColumn
+/// }
+///
+/// The outer code can influence the inner fragment as follows:
+///   * \p FirstStartColumn specifies the column at which \p Code starts.
+///   * \p NextStartColumn specifies the additional indent dictated by the
+///     surrounding code. It is applied to the rest of the lines of \p Code.
+///   * \p LastStartColumn specifies the column at which the last line of
+///     \p Code should end, in case the last line is an empty line.
+///
+///     In the case where the last line of the fragment contains content,
+///     the fragment ends at the end of that content and \p LastStartColumn is
+///     not taken into account, for example in:
+///
+///     void block() {
+///       string inner = R"(name: value)";
+///     }
+///
+/// Each range is extended on either end to its next bigger logic unit, i.e.
+/// everything that might influence its formatting or might be influenced by its
+/// formatting.
+///
+/// Returns a pair P, where:
+///   * P.first are the ``Replacements`` necessary to make all \p Ranges comply
+///     with \p Style.
+///   * P.second is the penalty induced by formatting the fragment \p Code.
+///     If the formatting of the fragment doesn't have a notion of penalty,
+///     returns 0.
+///
+/// If ``Status`` is non-null, its value will be populated with the status of
+/// this formatting attempt. See \c FormattingAttemptStatus.
+std::pair<tooling::Replacements, unsigned>
+reformat(const FormatStyle &Style, StringRef Code,
+         ArrayRef<tooling::Range> Ranges, unsigned FirstStartColumn,
+         unsigned NextStartColumn, unsigned LastStartColumn, StringRef FileName,
+         FormattingAttemptStatus *Status);
+
+} // namespace internal
+} // namespace format
+} // namespace clang
+
+#endif

cfe/trunk/lib/Format/FormatTokenLexer.h

 enum LexerState {
   NORMAL,
   TEMPLATE_STRING,
   TOKEN_STASHED,
 };
 
 class FormatTokenLexer {
 public:
-  FormatTokenLexer(const SourceManager &SourceMgr, FileID ID,
+  FormatTokenLexer(const SourceManager &SourceMgr, FileID ID, unsigned Column,
                    const FormatStyle &Style, encoding::Encoding Encoding);
 
   ArrayRef<FormatToken *> lex();
 
   const AdditionalKeywords &getKeywords() { return Keywords; }
 
 private:
   void tryMergePreviousTokens();

cfe/trunk/lib/Format/FormatTokenLexer.cpp

 #include "clang/Basic/SourceManager.h"
 #include "clang/Format/Format.h"
 #include "llvm/Support/Regex.h"
 
 namespace clang {
 namespace format {
 
 FormatTokenLexer::FormatTokenLexer(const SourceManager &SourceMgr, FileID ID,
-                                   const FormatStyle &Style,
+                                   unsigned Column, const FormatStyle &Style,
                                    encoding::Encoding Encoding)
     : FormatTok(nullptr), IsFirstToken(true), StateStack({LexerState::NORMAL}),
-      Column(0), TrailingWhitespace(0), SourceMgr(SourceMgr), ID(ID),
+      Column(Column), TrailingWhitespace(0), SourceMgr(SourceMgr), ID(ID),
       Style(Style), IdentTable(getFormattingLangOpts(Style)),
       Keywords(IdentTable), Encoding(Encoding), FirstInLineIndex(0),
       FormattingDisabled(false), MacroBlockBeginRegex(Style.MacroBlockBegin),
       MacroBlockEndRegex(Style.MacroBlockEnd) {
   Lex.reset(new Lexer(ID, SourceMgr.getBuffer(ID), SourceMgr,
                       getFormattingLangOpts(Style)));
   Lex->SetKeepWhitespaceMode(true);
 

cfe/trunk/lib/Format/NamespaceEndCommentsFixer.h

 
 namespace clang {
 namespace format {
 
 class NamespaceEndCommentsFixer : public TokenAnalyzer {
 public:
   NamespaceEndCommentsFixer(const Environment &Env, const FormatStyle &Style);
 
-  tooling::Replacements
+  std::pair<tooling::Replacements, unsigned>
   analyze(TokenAnnotator &Annotator,
           SmallVectorImpl<AnnotatedLine *> &AnnotatedLines,
           FormatTokenLexer &Tokens) override;
 };
 
 } // end namespace format
 } // end namespace clang
 
 #endif

cfe/trunk/lib/Format/NamespaceEndCommentsFixer.cpp

getNamespaceToken(const AnnotatedLine *line,
   return NamespaceTok;
 }
 } // namespace
 
 NamespaceEndCommentsFixer::NamespaceEndCommentsFixer(const Environment &Env,
                                                      const FormatStyle &Style)
     : TokenAnalyzer(Env, Style) {}
 
-tooling::Replacements NamespaceEndCommentsFixer::analyze(
+std::pair<tooling::Replacements, unsigned> NamespaceEndCommentsFixer::analyze(
     TokenAnnotator &Annotator, SmallVectorImpl<AnnotatedLine *> &AnnotatedLines,
     FormatTokenLexer &Tokens) {
   const SourceManager &SourceMgr = Env.getSourceManager();
   AffectedRangeMgr.computeAffectedLines(AnnotatedLines.begin(),
                                         AnnotatedLines.end());
   tooling::Replacements Fixes;
   std::string AllNamespaceNames = "";
   size_t StartLineIndex = SIZE_MAX;
if (!hasEndComment(EndCommentPrevTok)) {
       bool isShort = I - StartLineIndex <= kShortNamespaceMaxLines + 1;
       if (!isShort)
         addEndComment(EndCommentPrevTok, EndCommentText, SourceMgr, &Fixes);
     } else if (!validEndComment(EndCommentPrevTok, NamespaceName)) {
       updateEndComment(EndCommentPrevTok, EndCommentText, SourceMgr, &Fixes);
     }
     StartLineIndex = SIZE_MAX;
   }
-  return Fixes;
+  return {Fixes, 0};
 }
 
 } // namespace format
 } // namespace clang

cfe/trunk/lib/Format/SortJavaScriptImports.cpp

 // implemented as a TokenAnalyzer because ES6 imports have substantial syntactic
 // structure, making it messy to sort them using regular expressions.
 class JavaScriptImportSorter : public TokenAnalyzer {
 public:
   JavaScriptImportSorter(const Environment &Env, const FormatStyle &Style)
       : TokenAnalyzer(Env, Style),
         FileContents(Env.getSourceManager().getBufferData(Env.getFileID())) {}
 
-  tooling::Replacements
+  std::pair<tooling::Replacements, unsigned>
   analyze(TokenAnnotator &Annotator,
           SmallVectorImpl<AnnotatedLine *> &AnnotatedLines,
           FormatTokenLexer &Tokens) override {
     tooling::Replacements Result;
     AffectedRangeMgr.computeAffectedLines(AnnotatedLines.begin(),
                                           AnnotatedLines.end());
 
     const AdditionalKeywords &Keywords = Tokens.getKeywords();
     SmallVector<JsModuleReference, 16> References;
     AnnotatedLine *FirstNonImportLine;
     std::tie(References, FirstNonImportLine) =
         parseModuleReferences(Keywords, AnnotatedLines);
 
     if (References.empty())
-      return Result;
+      return {Result, 0};
 
     SmallVector<unsigned, 16> Indices;
     for (unsigned i = 0, e = References.size(); i != e; ++i)
       Indices.push_back(i);
     std::stable_sort(Indices.begin(), Indices.end(),
                      [&](unsigned LHSI, unsigned RHSI) {
                        return References[LHSI] < References[RHSI];
                      });
for (unsigned i = 0, e = Indices.size(); i != e; ++i) {
         if (!Reference.IsExport &&
             (Reference.IsExport != References[Indices[i + 1]].IsExport ||
              Reference.Category != References[Indices[i + 1]].Category))
           ReferencesText += "\n";
       }
     }
 
     if (ReferencesInOrder && SymbolsInOrder)
-      return Result;
+      return {Result, 0};
 
     SourceRange InsertionPoint = References[0].Range;
     InsertionPoint.setEnd(References[References.size() - 1].Range.getEnd());
 
     // The loop above might collapse previously existing line breaks between
     // import blocks, and thus shrink the file. SortIncludes must not shrink
     // overall source length as there is currently no re-calculation of ranges
     // after applying source sorting.
auto Err = Result.add(tooling::Replacement(
         ReferencesText));
     // FIXME: better error handling. For now, just print error message and skip
     // the replacement for the release version.
     if (Err) {
       llvm::errs() << llvm::toString(std::move(Err)) << "\n";
       assert(false);
     }
 
-    return Result;
+    return {Result, 0};
   }
 
 private:
   FormatToken *Current;
   FormatToken *LineEnd;
 
   FormatToken invalidToken;
 

 tooling::Replacements sortJavaScriptImports(const FormatStyle &Style,
                                             StringRef Code,
                                             ArrayRef<tooling::Range> Ranges,
                                             StringRef FileName) {
   // FIXME: Cursor support.
   std::unique_ptr<Environment> Env =
       Environment::CreateVirtualEnvironment(Code, FileName, Ranges);
   JavaScriptImportSorter Sorter(*Env, Style);
-  return Sorter.process();
+  return Sorter.process().first;
 }
 
 } // end namespace format
 } // end namespace clang

cfe/trunk/lib/Format/TokenAnalyzer.h

 #include "llvm/Support/Debug.h"
 
 namespace clang {
 namespace format {
 
 class Environment {
 public:
   Environment(SourceManager &SM, FileID ID, ArrayRef<CharSourceRange> Ranges)
-      : ID(ID), CharRanges(Ranges.begin(), Ranges.end()), SM(SM) {}
+      : ID(ID), CharRanges(Ranges.begin(), Ranges.end()), SM(SM),
+      FirstStartColumn(0),
+      NextStartColumn(0),
+      LastStartColumn(0) {}
 
   Environment(FileID ID, std::unique_ptr<FileManager> FileMgr,
               std::unique_ptr<SourceManager> VirtualSM,
               std::unique_ptr<DiagnosticsEngine> Diagnostics,
-              const std::vector<CharSourceRange> &CharRanges)
+              const std::vector<CharSourceRange> &CharRanges,
+              unsigned FirstStartColumn,
+              unsigned NextStartColumn,
+              unsigned LastStartColumn)
       : ID(ID), CharRanges(CharRanges.begin(), CharRanges.end()),
-        SM(*VirtualSM), FileMgr(std::move(FileMgr)),
+        SM(*VirtualSM), 
+        FirstStartColumn(FirstStartColumn),
+        NextStartColumn(NextStartColumn),
+        LastStartColumn(LastStartColumn),
+        FileMgr(std::move(FileMgr)),
         VirtualSM(std::move(VirtualSM)), Diagnostics(std::move(Diagnostics)) {}
 
-  // This sets up an virtual file system with file \p FileName containing \p
-  // Code.
+  // This sets up an virtual file system with file \p FileName containing the
+  // fragment \p Code. Assumes that \p Code starts at \p FirstStartColumn,
+  // that the next lines of \p Code should start at \p NextStartColumn, and
+  // that \p Code should end at \p LastStartColumn if it ends in newline.
+  // See also the documentation of clang::format::internal::reformat.
   static std::unique_ptr<Environment>
   CreateVirtualEnvironment(StringRef Code, StringRef FileName,
-                           ArrayRef<tooling::Range> Ranges);
+                           ArrayRef<tooling::Range> Ranges,
+                           unsigned FirstStartColumn = 0,
+                           unsigned NextStartColumn = 0,
+                           unsigned LastStartColumn = 0);
 
   FileID getFileID() const { return ID; }
 
   ArrayRef<CharSourceRange> getCharRanges() const { return CharRanges; }
 
   const SourceManager &getSourceManager() const { return SM; }
 
+  // Returns the column at which the fragment of code managed by this
+  // environment starts.
+  unsigned getFirstStartColumn() const { return FirstStartColumn; }
+
+  // Returns the column at which subsequent lines of the fragment of code
+  // managed by this environment should start.
+  unsigned getNextStartColumn() const { return NextStartColumn; }
+
+  // Returns the column at which the fragment of code managed by this
+  // environment should end if it ends in a newline.
+  unsigned getLastStartColumn() const { return LastStartColumn; }
+
 private:
   FileID ID;
   SmallVector<CharSourceRange, 8> CharRanges;
   SourceManager &SM;
+  unsigned FirstStartColumn;
+  unsigned NextStartColumn;
+  unsigned LastStartColumn;
 
   // The order of these fields are important - they should be in the same order
   // as they are created in `CreateVirtualEnvironment` so that they can be
   // deleted in the reverse order as they are created.
   std::unique_ptr<FileManager> FileMgr;
   std::unique_ptr<SourceManager> VirtualSM;
   std::unique_ptr<DiagnosticsEngine> Diagnostics;
 };
 
 class TokenAnalyzer : public UnwrappedLineConsumer {
 public:
   TokenAnalyzer(const Environment &Env, const FormatStyle &Style);
 
-  tooling::Replacements process();
+  std::pair<tooling::Replacements, unsigned> process();
 
 protected:
-  virtual tooling::Replacements
+  virtual std::pair<tooling::Replacements, unsigned>
   analyze(TokenAnnotator &Annotator,
           SmallVectorImpl<AnnotatedLine *> &AnnotatedLines,
           FormatTokenLexer &Tokens) = 0;
 
   void consumeUnwrappedLine(const UnwrappedLine &TheLine) override;
 
   void finishRun() override;
 

cfe/trunk/lib/Format/TokenAnalyzer.cpp

 
 namespace clang {
 namespace format {
 
 // This sets up an virtual file system with file \p FileName containing \p
 // Code.
 std::unique_ptr<Environment>
 Environment::CreateVirtualEnvironment(StringRef Code, StringRef FileName,
-                                      ArrayRef<tooling::Range> Ranges) {
+                                      ArrayRef<tooling::Range> Ranges,
+                                      unsigned FirstStartColumn,
+                                      unsigned NextStartColumn,
+                                      unsigned LastStartColumn) {
   // This is referenced by `FileMgr` and will be released by `FileMgr` when it
   // is deleted.
   IntrusiveRefCntPtr<vfs::InMemoryFileSystem> InMemoryFileSystem(
       new vfs::InMemoryFileSystem);
   // This is passed to `SM` as reference, so the pointer has to be referenced
   // in `Environment` so that `FileMgr` can out-live this function scope.
   std::unique_ptr<FileManager> FileMgr(
       new FileManager(FileSystemOptions(), InMemoryFileSystem));
Environment::CreateVirtualEnvironment(StringRef Code, StringRef FileName,
   assert(ID.isValid());
   SourceLocation StartOfFile = VirtualSM->getLocForStartOfFile(ID);
   std::vector<CharSourceRange> CharRanges;
   for (const tooling::Range &Range : Ranges) {
     SourceLocation Start = StartOfFile.getLocWithOffset(Range.getOffset());
     SourceLocation End = Start.getLocWithOffset(Range.getLength());
     CharRanges.push_back(CharSourceRange::getCharRange(Start, End));
   }
-  return llvm::make_unique<Environment>(ID, std::move(FileMgr),
-                                        std::move(VirtualSM),
-                                        std::move(Diagnostics), CharRanges);
+  return llvm::make_unique<Environment>(
+      ID, std::move(FileMgr), std::move(VirtualSM), std::move(Diagnostics),
+      CharRanges, FirstStartColumn, NextStartColumn, LastStartColumn);
 }
 
 TokenAnalyzer::TokenAnalyzer(const Environment &Env, const FormatStyle &Style)
     : Style(Style), Env(Env),
       AffectedRangeMgr(Env.getSourceManager(), Env.getCharRanges()),
       UnwrappedLines(1),
       Encoding(encoding::detectEncoding(
           Env.getSourceManager().getBufferData(Env.getFileID()))) {
   DEBUG(
       llvm::dbgs() << "File encoding: "
                    << (Encoding == encoding::Encoding_UTF8 ? "UTF8" : "unknown")
                    << "\n");
   DEBUG(llvm::dbgs() << "Language: " << getLanguageName(Style.Language)
                      << "\n");
 }
 
-tooling::Replacements TokenAnalyzer::process() {
+std::pair<tooling::Replacements, unsigned> TokenAnalyzer::process() {
   tooling::Replacements Result;
-  FormatTokenLexer Tokens(Env.getSourceManager(), Env.getFileID(), Style,
-                          Encoding);
+  FormatTokenLexer Tokens(Env.getSourceManager(), Env.getFileID(),
+                          Env.getFirstStartColumn(), Style, Encoding);
 
-  UnwrappedLineParser Parser(Style, Tokens.getKeywords(), Tokens.lex(), *this);
+  UnwrappedLineParser Parser(Style, Tokens.getKeywords(),
+                             Env.getFirstStartColumn(), Tokens.lex(), *this);
   Parser.parse();
   assert(UnwrappedLines.rbegin()->empty());
+  unsigned Penalty = 0;
   for (unsigned Run = 0, RunE = UnwrappedLines.size(); Run + 1 != RunE; ++Run) {
     DEBUG(llvm::dbgs() << "Run " << Run << "...\n");
     SmallVector<AnnotatedLine *, 16> AnnotatedLines;
 
     TokenAnnotator Annotator(Style, Tokens.getKeywords());
     for (unsigned i = 0, e = UnwrappedLines[Run].size(); i != e; ++i) {
       AnnotatedLines.push_back(new AnnotatedLine(UnwrappedLines[Run][i]));
       Annotator.annotate(*AnnotatedLines.back());
     }
 
-    tooling::Replacements RunResult =
+    std::pair<tooling::Replacements, unsigned> RunResult =
         analyze(Annotator, AnnotatedLines, Tokens);
 
     DEBUG({
       llvm::dbgs() << "Replacements for run " << Run << ":\n";
-      for (tooling::Replacements::const_iterator I = RunResult.begin(),
-                                                 E = RunResult.end();
+      for (tooling::Replacements::const_iterator I = RunResult.first.begin(),
+                                                 E = RunResult.first.end();
            I != E; ++I) {
         llvm::dbgs() << I->toString() << "\n";
       }
     });
     for (unsigned i = 0, e = AnnotatedLines.size(); i != e; ++i) {
       delete AnnotatedLines[i];
     }
-    for (const auto &R : RunResult) {
+
+    Penalty += RunResult.second;
+    for (const auto &R : RunResult.first) {
       auto Err = Result.add(R);
       // FIXME: better error handling here. For now, simply return an empty
       // Replacements to indicate failure.
       if (Err) {
         llvm::errs() << llvm::toString(std::move(Err)) << "\n";
-        return tooling::Replacements();
+        return {tooling::Replacements(), 0};
       }
     }
   }
-  return Result;
+  return {Result, Penalty};
 }
 
 void TokenAnalyzer::consumeUnwrappedLine(const UnwrappedLine &TheLine) {
   assert(!UnwrappedLines.empty());
   UnwrappedLines.back().push_back(TheLine);
 }
 
 void TokenAnalyzer::finishRun() {
   UnwrappedLines.push_back(SmallVector<UnwrappedLine, 16>());
 }
 
 } // end namespace format
 } // end namespace clang

cfe/trunk/lib/Format/TokenAnnotator.h

 class AnnotatedLine {
 public:
   AnnotatedLine(const UnwrappedLine &Line)
       : First(Line.Tokens.front().Tok), Level(Line.Level),
         MatchingOpeningBlockLineIndex(Line.MatchingOpeningBlockLineIndex),
         InPPDirective(Line.InPPDirective),
         MustBeDeclaration(Line.MustBeDeclaration), MightBeFunctionDecl(false),
         IsMultiVariableDeclStmt(false), Affected(false),
-        LeadingEmptyLinesAffected(false), ChildrenAffected(false) {
+        LeadingEmptyLinesAffected(false), ChildrenAffected(false),
+        FirstStartColumn(Line.FirstStartColumn) {
     assert(!Line.Tokens.empty());
 
     // Calculate Next and Previous for all tokens. Note that we must overwrite
     // Next and Previous for every token, as previous formatting runs might have
     // left them in a different state.
     First->Previous = nullptr;
     FormatToken *Current = First;
     for (std::list<UnwrappedLineNode>::const_iterator I = ++Line.Tokens.begin(),
public:
 
   /// \c True if the leading empty lines of this line intersect with one of the
   /// input ranges.
   bool LeadingEmptyLinesAffected;
 
   /// \c True if one of this line's children intersects with an input range.
   bool ChildrenAffected;
 
+  unsigned FirstStartColumn;
+
 private:
   // Disallow copying.
   AnnotatedLine(const AnnotatedLine &) = delete;
   void operator=(const AnnotatedLine &) = delete;
 };
 
 /// \brief Determines extra information about the tokens comprising an
 /// \c UnwrappedLine.

cfe/trunk/lib/Format/TokenAnnotator.cpp

 void TokenAnnotator::calculateFormattingInformation(AnnotatedLine &Line) {
   for (SmallVectorImpl<AnnotatedLine *>::iterator I = Line.Children.begin(),
                                                   E = Line.Children.end();
        I != E; ++I) {
     calculateFormattingInformation(**I);
   }
 
   Line.First->TotalLength =
-      Line.First->IsMultiline ? Style.ColumnLimit : Line.First->ColumnWidth;
+      Line.First->IsMultiline ? Style.ColumnLimit
+                              : Line.FirstStartColumn + Line.First->ColumnWidth;
   FormatToken *Current = Line.First->Next;
   bool InFunctionDecl = Line.MightBeFunctionDecl;
   while (Current) {
     if (isFunctionDeclarationName(*Current, Line))
       Current->Type = TT_FunctionDeclarationName;
     if (Current->is(TT_LineComment)) {
       if (Current->Previous->BlockKind == BK_BracedInit &&
           Current->Previous->opensScope())

cfe/trunk/lib/Format/UnwrappedLineFormatter.h

UnwrappedLineFormatter(ContinuationIndenter *Indenter,
                          const SourceManager &SourceMgr,
                          FormattingAttemptStatus *Status)
       : Indenter(Indenter), Whitespaces(Whitespaces), Style(Style),
         Keywords(Keywords), SourceMgr(SourceMgr), Status(Status) {}
 
   /// \brief Format the current block and return the penalty.
   unsigned format(const SmallVectorImpl<AnnotatedLine *> &Lines,
                   bool DryRun = false, int AdditionalIndent = 0,
-                  bool FixBadIndentation = false);
+                  bool FixBadIndentation = false,
+                  unsigned FirstStartColumn = 0,
+                  unsigned NextStartColumn = 0,
+                  unsigned LastStartColumn = 0);
 
 private:
   /// \brief Add a new line and the required indent before the first Token
   /// of the \c UnwrappedLine if there was no structural parsing error.
   void formatFirstToken(const AnnotatedLine &Line,
-                        const AnnotatedLine *PreviousLine, unsigned Indent);
+                        const AnnotatedLine *PreviousLine, unsigned Indent,
+                        unsigned NewlineIndent);
 
   /// \brief Returns the column limit for a line, taking into account whether we
   /// need an escaped newline due to a continued preprocessor directive.
   unsigned getColumnLimit(bool InPPDirective,
                           const AnnotatedLine *NextLine) const;
 
   // Cache to store the penalty of formatting a vector of AnnotatedLines
   // starting from a specific additional offset. Improves performance if there

cfe/trunk/lib/Format/UnwrappedLineFormatter.cpp

LineFormatter(ContinuationIndenter *Indenter, WhitespaceManager *Whitespaces,
                 UnwrappedLineFormatter *BlockFormatter)
       : Indenter(Indenter), Whitespaces(Whitespaces), Style(Style),
         BlockFormatter(BlockFormatter) {}
   virtual ~LineFormatter() {}
 
   /// \brief Formats an \c AnnotatedLine and returns the penalty.
   ///
   /// If \p DryRun is \c false, directly applies the changes.
-  virtual unsigned formatLine(const AnnotatedLine &Line, unsigned FirstIndent,
+  virtual unsigned formatLine(const AnnotatedLine &Line,
+                              unsigned FirstIndent,
+                              unsigned FirstStartColumn,
                               bool DryRun) = 0;
 
 protected:
   /// \brief If the \p State's next token is an r_brace closing a nested block,
   /// format the nested block before it.
   ///
   /// Returns \c true if all children could be placed successfully and adapts
   /// \p Penalty as well as \p State. If \p DryRun is false, also directly
if (Style.ColumnLimit > 0 &&
         Child->Last->TotalLength + State.Column + 2 > Style.ColumnLimit)
       return false;
 
     if (!DryRun) {
       Whitespaces->replaceWhitespace(
           *Child->First, /*Newlines=*/0, /*Spaces=*/1,
           /*StartOfTokenColumn=*/State.Column, State.Line->InPPDirective);
     }
-    Penalty += formatLine(*Child, State.Column + 1, DryRun);
+    Penalty +=
+        formatLine(*Child, State.Column + 1, /*FirstStartColumn=*/0, DryRun);
 
     State.Column += 1 + Child->Last->TotalLength;
     return true;
   }
 
   ContinuationIndenter *Indenter;
 
 private:
NoColumnLimitLineFormatter(ContinuationIndenter *Indenter,
                              WhitespaceManager *Whitespaces,
                              const FormatStyle &Style,
                              UnwrappedLineFormatter *BlockFormatter)
       : LineFormatter(Indenter, Whitespaces, Style, BlockFormatter) {}
 
   /// \brief Formats the line, simply keeping all of the input's line breaking
   /// decisions.
   unsigned formatLine(const AnnotatedLine &Line, unsigned FirstIndent,
-                      bool DryRun) override {
+                      unsigned FirstStartColumn, bool DryRun) override {
     assert(!DryRun);
-    LineState State =
-        Indenter->getInitialState(FirstIndent, &Line, /*DryRun=*/false);
+    LineState State = Indenter->getInitialState(FirstIndent, FirstStartColumn,
+                                                &Line, /*DryRun=*/false);
     while (State.NextToken) {
       bool Newline =
           Indenter->mustBreak(State) ||
           (Indenter->canBreak(State) && State.NextToken->NewlinesBefore > 0);
       unsigned Penalty = 0;
       formatChildren(State, Newline, /*DryRun=*/false, Penalty);
       Indenter->addTokenToState(State, Newline, /*DryRun=*/false);
     }
     return 0;
   }
 };
 
 /// \brief Formatter that puts all tokens into a single line without breaks.
 class NoLineBreakFormatter : public LineFormatter {
 public:
   NoLineBreakFormatter(ContinuationIndenter *Indenter,
                        WhitespaceManager *Whitespaces, const FormatStyle &Style,
                        UnwrappedLineFormatter *BlockFormatter)
       : LineFormatter(Indenter, Whitespaces, Style, BlockFormatter) {}
 
   /// \brief Puts all tokens into a single line.
   unsigned formatLine(const AnnotatedLine &Line, unsigned FirstIndent,
-                      bool DryRun) override {
+                      unsigned FirstStartColumn, bool DryRun) override {
     unsigned Penalty = 0;
-    LineState State = Indenter->getInitialState(FirstIndent, &Line, DryRun);
+    LineState State =
+        Indenter->getInitialState(FirstIndent, FirstStartColumn, &Line, DryRun);
     while (State.NextToken) {
       formatChildren(State, /*Newline=*/false, DryRun, Penalty);
       Indenter->addTokenToState(
           State, /*Newline=*/State.NextToken->MustBreakBefore, DryRun);
     }
     return Penalty;
   }
 };
 
 /// \brief Finds the best way to break lines.
 class OptimizingLineFormatter : public LineFormatter {
 public:
   OptimizingLineFormatter(ContinuationIndenter *Indenter,
                           WhitespaceManager *Whitespaces,
                           const FormatStyle &Style,
                           UnwrappedLineFormatter *BlockFormatter)
       : LineFormatter(Indenter, Whitespaces, Style, BlockFormatter) {}
 
   /// \brief Formats the line by finding the best line breaks with line lengths
   /// below the column limit.
   unsigned formatLine(const AnnotatedLine &Line, unsigned FirstIndent,
-                      bool DryRun) override {
-    LineState State = Indenter->getInitialState(FirstIndent, &Line, DryRun);
+                      unsigned FirstStartColumn, bool DryRun) override {
+    LineState State =
+        Indenter->getInitialState(FirstIndent, FirstStartColumn, &Line, DryRun);
 
     // If the ObjC method declaration does not fit on a line, we should format
     // it with one arg per line.
     if (State.Line->Type == LT_ObjCMethodDecl)
       State.Stack.back().BreakBeforeParameter = true;
 
     // Find best solution in solution space.
     return analyzeSolutionSpace(State, DryRun);
private:
   llvm::SpecificBumpPtrAllocator<StateNode> Allocator;
 };
 
 } // anonymous namespace
 
 unsigned
 UnwrappedLineFormatter::format(const SmallVectorImpl<AnnotatedLine *> &Lines,
                                bool DryRun, int AdditionalIndent,
-                               bool FixBadIndentation) {
+                               bool FixBadIndentation,
+                               unsigned FirstStartColumn,
+                               unsigned NextStartColumn,
+                               unsigned LastStartColumn) {
   LineJoiner Joiner(Style, Keywords, Lines);
 
   // Try to look up already computed penalty in DryRun-mode.
   std::pair<const SmallVectorImpl<AnnotatedLine *> *, unsigned> CacheKey(
       &Lines, AdditionalIndent);
   auto CacheIt = PenaltyCache.find(CacheKey);
   if (DryRun && CacheIt != PenaltyCache.end())
     return CacheIt->second;
 
   assert(!Lines.empty());
   unsigned Penalty = 0;
   LevelIndentTracker IndentTracker(Style, Keywords, Lines[0]->Level,
                                    AdditionalIndent);
   const AnnotatedLine *PreviousLine = nullptr;
   const AnnotatedLine *NextLine = nullptr;
 
   // The minimum level of consecutive lines that have been formatted.
   unsigned RangeMinLevel = UINT_MAX;
 
+  bool FirstLine = true;
   for (const AnnotatedLine *Line =
            Joiner.getNextMergedLine(DryRun, IndentTracker);
-       Line; Line = NextLine) {
+       Line; Line = NextLine, FirstLine = false) {
     const AnnotatedLine &TheLine = *Line;
     unsigned Indent = IndentTracker.getIndent();
 
     // We continue formatting unchanged lines to adjust their indent, e.g. if a
     // scope was added. However, we need to carefully stop doing this when we
     // exit the scope of affected lines to prevent indenting a the entire
     // remaining file if it currently missing a closing brace.
     bool ContinueFormatting =
         TheLine.Level > RangeMinLevel ||
         (TheLine.Level == RangeMinLevel && !TheLine.startsWith(tok::r_brace));
 
     bool FixIndentation = (FixBadIndentation || ContinueFormatting) &&
                           Indent != TheLine.First->OriginalColumn;
     bool ShouldFormat = TheLine.Affected || FixIndentation;
     // We cannot format this line; if the reason is that the line had a
     // parsing error, remember that.
     if (ShouldFormat && TheLine.Type == LT_Invalid && Status) {
       Status->FormatComplete = false;
       Status->Line =
           SourceMgr.getSpellingLineNumber(TheLine.First->Tok.getLocation());
     }
 
     if (ShouldFormat && TheLine.Type != LT_Invalid) {
-      if (!DryRun)
-        formatFirstToken(TheLine, PreviousLine, Indent);
+      if (!DryRun) {
+        bool LastLine = Line->First->is(tok::eof);
+        formatFirstToken(TheLine, PreviousLine,
+                         Indent,
+                         LastLine ? LastStartColumn : NextStartColumn + Indent);
+      }
 
       NextLine = Joiner.getNextMergedLine(DryRun, IndentTracker);
       unsigned ColumnLimit = getColumnLimit(TheLine.InPPDirective, NextLine);
       bool FitsIntoOneLine =
           TheLine.Last->TotalLength + Indent <= ColumnLimit ||
           (TheLine.Type == LT_ImportStatement &&
            (Style.Language != FormatStyle::LK_JavaScript ||
             !Style.JavaScriptWrapImports));
-
       if (Style.ColumnLimit == 0)
         NoColumnLimitLineFormatter(Indenter, Whitespaces, Style, this)
-            .formatLine(TheLine, Indent, DryRun);
+            .formatLine(TheLine, NextStartColumn + Indent,
+                        FirstLine ? FirstStartColumn : 0, DryRun);
       else if (FitsIntoOneLine)
         Penalty += NoLineBreakFormatter(Indenter, Whitespaces, Style, this)
-                       .formatLine(TheLine, Indent, DryRun);
+                       .formatLine(TheLine, NextStartColumn + Indent,
+                                   FirstLine ? FirstStartColumn : 0, DryRun);
       else
         Penalty += OptimizingLineFormatter(Indenter, Whitespaces, Style, this)
-                       .formatLine(TheLine, Indent, DryRun);
+                       .formatLine(TheLine, NextStartColumn + Indent,
+                                   FirstLine ? FirstStartColumn : 0, DryRun);
       RangeMinLevel = std::min(RangeMinLevel, TheLine.Level);
     } else {
       // If no token in the current line is affected, we still need to format
       // affected children.
       if (TheLine.ChildrenAffected)
         for (const FormatToken *Tok = TheLine.First; Tok; Tok = Tok->Next)
           if (!Tok->Children.empty())
             format(Tok->Children, DryRun);
 
       // Adapt following lines on the current indent level to the same level
       // unless the current \c AnnotatedLine is not at the beginning of a line.
       bool StartsNewLine =
           TheLine.First->NewlinesBefore > 0 || TheLine.First->IsFirst;
       if (StartsNewLine)
         IndentTracker.adjustToUnmodifiedLine(TheLine);
       if (!DryRun) {
         bool ReformatLeadingWhitespace =
             StartsNewLine && ((PreviousLine && PreviousLine->Affected) ||
                               TheLine.LeadingEmptyLinesAffected);
         // Format the first token.
         if (ReformatLeadingWhitespace)
           formatFirstToken(TheLine, PreviousLine,
+                           TheLine.First->OriginalColumn,
                            TheLine.First->OriginalColumn);
         else
           Whitespaces->addUntouchableToken(*TheLine.First,
                                            TheLine.InPPDirective);
 
         // Notify the WhitespaceManager about the unchanged whitespace.
         for (FormatToken *Tok = TheLine.First->Next; Tok; Tok = Tok->Next)
           Whitespaces->addUntouchableToken(*Tok, TheLine.InPPDirective);
       }
       NextLine = Joiner.getNextMergedLine(DryRun, IndentTracker);
       RangeMinLevel = UINT_MAX;
     }
     if (!DryRun)
       markFinalized(TheLine.First);
     PreviousLine = &TheLine;
   }
   PenaltyCache[CacheKey] = Penalty;
   return Penalty;
 }
 
 void UnwrappedLineFormatter::formatFirstToken(const AnnotatedLine &Line,
                                               const AnnotatedLine *PreviousLine,
-                                              unsigned Indent) {
+                                              unsigned Indent,
+                                              unsigned NewlineIndent) {
   FormatToken &RootToken = *Line.First;
   if (RootToken.is(tok::eof)) {
     unsigned Newlines = std::min(RootToken.NewlinesBefore, 1u);
-    Whitespaces->replaceWhitespace(RootToken, Newlines, /*Spaces=*/0,
-                                   /*StartOfTokenColumn=*/0);
+    unsigned TokenIndent = Newlines ? NewlineIndent : 0;
+    Whitespaces->replaceWhitespace(RootToken, Newlines, TokenIndent,
+                                   TokenIndent);
     return;
   }
   unsigned Newlines =
       std::min(RootToken.NewlinesBefore, Style.MaxEmptyLinesToKeep + 1);
   // Remove empty lines before "}" where applicable.
   if (RootToken.is(tok::r_brace) &&
       (!RootToken.Next ||
        (RootToken.Next->is(tok::semi) && !RootToken.Next->Next)))
     Newlines = std::min(Newlines, 1u);
   if (Newlines == 0 && !RootToken.IsFirst)
     Newlines = 1;
   if (RootToken.IsFirst && !RootToken.HasUnescapedNewline)
     Newlines = 0;
 
-  // Preprocessor directives get indented after the hash, if indented.
-  if (Line.Type == LT_PreprocessorDirective || Line.Type == LT_ImportStatement)
-    Indent = 0;
-
   // Remove empty lines after "{".
   if (!Style.KeepEmptyLinesAtTheStartOfBlocks && PreviousLine &&
       PreviousLine->Last->is(tok::l_brace) &&
       PreviousLine->First->isNot(tok::kw_namespace) &&
       !startsExternCBlock(*PreviousLine))
     Newlines = 1;
 
   // Insert extra new line before access specifiers.
   if (PreviousLine && PreviousLine->Last->isOneOf(tok::semi, tok::r_brace) &&
       RootToken.isAccessSpecifier() && RootToken.NewlinesBefore == 1)
     ++Newlines;
 
   // Remove empty lines after access specifiers.
   if (PreviousLine && PreviousLine->First->isAccessSpecifier() &&
       (!PreviousLine->InPPDirective || !RootToken.HasUnescapedNewline))
     Newlines = std::min(1u, Newlines);
 
+  if (Newlines)
+    Indent = NewlineIndent;
+
+  // Preprocessor directives get indented after the hash, if indented.
+  if (Line.Type == LT_PreprocessorDirective || Line.Type == LT_ImportStatement)
+    Indent = 0;
+
   Whitespaces->replaceWhitespace(RootToken, Newlines, Indent, Indent,
                                  Line.InPPDirective &&
                                      !RootToken.HasUnescapedNewline);
 }
 
 unsigned
 UnwrappedLineFormatter::getColumnLimit(bool InPPDirective,
                                        const AnnotatedLine *NextLine) const {

cfe/trunk/lib/Format/UnwrappedLineParser.h

struct UnwrappedLine {
 
   /// \brief If this \c UnwrappedLine closes a block in a sequence of lines,
   /// \c MatchingOpeningBlockLineIndex stores the index of the corresponding
   /// opening line. Otherwise, \c MatchingOpeningBlockLineIndex must be
   /// \c kInvalidIndex.
   size_t MatchingOpeningBlockLineIndex;
 
   static const size_t kInvalidIndex = -1;
+
+  unsigned FirstStartColumn = 0;
 };
 
 class UnwrappedLineConsumer {
 public:
   virtual ~UnwrappedLineConsumer() {}
   virtual void consumeUnwrappedLine(const UnwrappedLine &Line) = 0;
   virtual void finishRun() = 0;
 };
 
 class FormatTokenSource;
 
 class UnwrappedLineParser {
 public:
   UnwrappedLineParser(const FormatStyle &Style,
                       const AdditionalKeywords &Keywords,
+                      unsigned FirstStartColumn,
                       ArrayRef<FormatToken *> Tokens,
                       UnwrappedLineConsumer &Callback);
 
   void parse();
 
 private:
   void reset();
   void parseFile();
private:
   // This is used to update PPLevelBranchCount at the end of a branch
   // sequence.
   std::stack<int> PPChainBranchIndex;
 
   // Contains the #ifndef condition for a potential include guard.
   FormatToken *IfNdefCondition;
   bool FoundIncludeGuardStart;
   bool IncludeGuardRejected;
+  // Contains the first start column where the source begins. This is zero for
+  // normal source code and may be nonzero when formatting a code fragment that
+  // does not start at the beginning of the file.
+  unsigned FirstStartColumn;
 
   friend class ScopedLineState;
   friend class CompoundStatementIndenter;
 };
 
 struct UnwrappedLineNode {
   UnwrappedLineNode() : Tok(nullptr) {}
   UnwrappedLineNode(FormatToken *Tok) : Tok(Tok) {}

cfe/trunk/lib/Format/UnwrappedLineParser.cpp

private:
   ArrayRef<FormatToken *> Tokens;
   int Position;
 };
 
 } // end anonymous namespace
 
 UnwrappedLineParser::UnwrappedLineParser(const FormatStyle &Style,
                                          const AdditionalKeywords &Keywords,
+                                         unsigned FirstStartColumn,
                                          ArrayRef<FormatToken *> Tokens,
                                          UnwrappedLineConsumer &Callback)
     : Line(new UnwrappedLine), MustBreakBeforeNextToken(false),
       CurrentLines(&Lines), Style(Style), Keywords(Keywords),
       CommentPragmasRegex(Style.CommentPragmas), Tokens(nullptr),
       Callback(Callback), AllTokens(Tokens), PPBranchLevel(-1),
       IfNdefCondition(nullptr), FoundIncludeGuardStart(false),
-      IncludeGuardRejected(false) {}
+      IncludeGuardRejected(false), FirstStartColumn(FirstStartColumn) {}
 
 void UnwrappedLineParser::reset() {
   PPBranchLevel = -1;
   IfNdefCondition = nullptr;
   FoundIncludeGuardStart = false;
   IncludeGuardRejected = false;
   Line.reset(new UnwrappedLine);
   CommentsBeforeNextToken.clear();
   FormatTok = nullptr;
   MustBreakBeforeNextToken = false;
   PreprocessorDirectives.clear();
   CurrentLines = &Lines;
   DeclarationScopeStack.clear();
   PPStack.clear();
+  Line->FirstStartColumn = FirstStartColumn;
 }
 
 void UnwrappedLineParser::parse() {
   IndexedTokenSource TokenSource(AllTokens);
+  Line->FirstStartColumn = FirstStartColumn;
   do {
     DEBUG(llvm::dbgs() << "----\n");
     reset();
     Tokens = &TokenSource;
     TokenSource.reset();
 
     readToken();
     parseFile();
while (!eof()) {
     } else {
       nextToken();
     }
   }
 }
 
 LLVM_ATTRIBUTE_UNUSED static void printDebugInfo(const UnwrappedLine &Line,
                                                  StringRef Prefix = "") {
-  llvm::dbgs() << Prefix << "Line(" << Line.Level << ")"
+  llvm::dbgs() << Prefix << "Line(" << Line.Level
+               << ", FSC=" << Line.FirstStartColumn << ")"
                << (Line.InPPDirective ? " MACRO" : "") << ": ";
   for (std::list<UnwrappedLineNode>::const_iterator I = Line.Tokens.begin(),
                                                     E = Line.Tokens.end();
        I != E; ++I) {
     llvm::dbgs() << I->Tok->Tok.getName() << "["
                  << "T=" << I->Tok->Type << ", OC=" << I->Tok->OriginalColumn
                  << "] ";
   }
if (Line->Tokens.empty())
     return;
   DEBUG({
     if (CurrentLines == &Lines)
       printDebugInfo(*Line);
   });
   CurrentLines->push_back(std::move(*Line));
   Line->Tokens.clear();
   Line->MatchingOpeningBlockLineIndex = UnwrappedLine::kInvalidIndex;
+  Line->FirstStartColumn = 0;
   if (CurrentLines == &Lines && !PreprocessorDirectives.empty()) {
     CurrentLines->append(
         std::make_move_iterator(PreprocessorDirectives.begin()),
         std::make_move_iterator(PreprocessorDirectives.end()));
     PreprocessorDirectives.clear();
   }
   // Disconnect the current token from the last token on the previous line.
   FormatTok->Previous = nullptr;

cfe/trunk/lib/Format/UsingDeclarationsSorter.h

 
 namespace clang {
 namespace format {
 
 class UsingDeclarationsSorter : public TokenAnalyzer {
 public:
   UsingDeclarationsSorter(const Environment &Env, const FormatStyle &Style);
 
-  tooling::Replacements
+  std::pair<tooling::Replacements, unsigned>
   analyze(TokenAnnotator &Annotator,
           SmallVectorImpl<AnnotatedLine *> &AnnotatedLines,
           FormatTokenLexer &Tokens) override;
 };
 
 } // end namespace format
 } // end namespace clang
 
 #endif

cfe/trunk/lib/Format/UsingDeclarationsSorter.cpp

 }
 
 } // namespace
 
 UsingDeclarationsSorter::UsingDeclarationsSorter(const Environment &Env,
                                                  const FormatStyle &Style)
     : TokenAnalyzer(Env, Style) {}
 
-tooling::Replacements UsingDeclarationsSorter::analyze(
+std::pair<tooling::Replacements, unsigned> UsingDeclarationsSorter::analyze(
     TokenAnnotator &Annotator, SmallVectorImpl<AnnotatedLine *> &AnnotatedLines,
     FormatTokenLexer &Tokens) {
   const SourceManager &SourceMgr = Env.getSourceManager();
   AffectedRangeMgr.computeAffectedLines(AnnotatedLines.begin(),
                                         AnnotatedLines.end());
   tooling::Replacements Fixes;
   SmallVector<UsingDeclaration, 4> UsingDeclarations;
   for (size_t I = 0, E = AnnotatedLines.size(); I != E; ++I) {
     if (AnnotatedLines[I]->InPPDirective ||
         !AnnotatedLines[I]->startsWith(tok::kw_using) ||
         AnnotatedLines[I]->First->Finalized) {
       endUsingDeclarationBlock(&UsingDeclarations, SourceMgr, &Fixes);
       continue;
     }
     if (AnnotatedLines[I]->First->NewlinesBefore > 1)
       endUsingDeclarationBlock(&UsingDeclarations, SourceMgr, &Fixes);
     std::string Label = computeUsingDeclarationLabel(AnnotatedLines[I]->First);
     if (Label.empty()) {
       endUsingDeclarationBlock(&UsingDeclarations, SourceMgr, &Fixes);
       continue;
     }
     UsingDeclarations.push_back(UsingDeclaration(AnnotatedLines[I], Label));
   }
   endUsingDeclarationBlock(&UsingDeclarations, SourceMgr, &Fixes);
-  return Fixes;
+  return {Fixes, 0};
 }
 
 } // namespace format
 } // namespace clang

cfe/trunk/lib/Format/WhitespaceManager.h

void replaceWhitespace(FormatToken &Tok, unsigned Newlines, unsigned Spaces,
                          bool InPPDirective = false);
 
   /// \brief Adds information about an unchangeable token's whitespace.
   ///
   /// Needs to be called for every token for which \c replaceWhitespace
   /// was not called.
   void addUntouchableToken(const FormatToken &Tok, bool InPPDirective);
 
+  llvm::Error addReplacement(const tooling::Replacement &Replacement);
+
   /// \brief Inserts or replaces whitespace in the middle of a token.
   ///
   /// Inserts \p PreviousPostfix, \p Newlines, \p Spaces and \p CurrentPrefix
   /// (in this order) at \p Offset inside \p Tok, replacing \p ReplaceChars
   /// characters.
   ///
   /// Note: \p Spaces can be negative to retain information about initial
   /// relative column offset between a line of a block comment and the start of

cfe/trunk/lib/Format/WhitespaceManager.cpp

if (Tok.Finalized)
     return;
   Changes.push_back(Change(Tok, /*CreateReplacement=*/false,
                            Tok.WhitespaceRange, /*Spaces=*/0,
                            Tok.OriginalColumn, Tok.NewlinesBefore, "", "",
                            InPPDirective && !Tok.IsFirst,
                            /*IsInsideToken=*/false));
 }
 
+llvm::Error
+WhitespaceManager::addReplacement(const tooling::Replacement &Replacement) {
+  return Replaces.add(Replacement);
+}
+
 void WhitespaceManager::replaceWhitespaceInToken(
     const FormatToken &Tok, unsigned Offset, unsigned ReplaceChars,
     StringRef PreviousPostfix, StringRef CurrentPrefix, bool InPPDirective,
     unsigned Newlines, int Spaces) {
   if (Tok.Finalized)
     return;
   SourceLocation Start = Tok.getStartOfNonWhitespace().getLocWithOffset(Offset);
   Changes.push_back(

cfe/trunk/unittests/Format/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   Support
   )
 
 add_clang_unittest(FormatTests
   CleanupTest.cpp
   FormatTest.cpp
   FormatTestComments.cpp
   FormatTestJS.cpp
   FormatTestJava.cpp
   FormatTestObjC.cpp
   FormatTestProto.cpp
+  FormatTestRawStrings.cpp
   FormatTestSelective.cpp
   FormatTestTextProto.cpp
   NamespaceEndCommentsFixerTest.cpp
   SortImportsTestJS.cpp
   SortIncludesTest.cpp
   UsingDeclarationsSorterTest.cpp
   )
 
 target_link_libraries(FormatTests
   clangBasic
   clangFormat
   clangFrontend
   clangRewrite
   clangToolingCore
   )

cfe/trunk/unittests/Format/FormatTest.cpp

std::vector<FormatStyle::IncludeCategory> ExpectedCategories = {{"abc/.*", 2},
                                                                   {".*", 1}};
   CHECK_PARSE("IncludeCategories:\n"
               "  - Regex: abc/.*\n"
               "    Priority: 2\n"
               "  - Regex: .*\n"
               "    Priority: 1",
               IncludeCategories, ExpectedCategories);
   CHECK_PARSE("IncludeIsMainRegex: 'abc$'", IncludeIsMainRegex, "abc$");
+
+  Style.RawStringFormats.clear();
+  std::vector<FormatStyle::RawStringFormat> ExpectedRawStringFormats = {
+      {"pb", FormatStyle::LK_TextProto, "llvm"},
+      {"cpp", FormatStyle::LK_Cpp, "google"}};
+
+  CHECK_PARSE("RawStringFormats:\n"
+              "  - Delimiter: 'pb'\n"
+              "    Language: TextProto\n"
+              "    BasedOnStyle: llvm\n"
+              "  - Delimiter: 'cpp'\n"
+              "    Language: Cpp\n"
+              "    BasedOnStyle: google",
+              RawStringFormats, ExpectedRawStringFormats);
 }
 
 TEST_F(FormatTest, ParsesConfigurationWithLanguages) {
   FormatStyle Style = {};
   Style.Language = FormatStyle::LK_Cpp;
   CHECK_PARSE("Language: Cpp\n"
               "IndentWidth: 12",
               IndentWidth, 12u);

cfe/trunk/unittests/Format/FormatTestRawStrings.cpp

+//===- unittest/Format/FormatTestRawStrings.cpp - Formatting unit tests ---===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Format/Format.h"
+
+#include "../Tooling/ReplacementTest.h"
+#include "FormatTestUtils.h"
+
+#include "clang/Frontend/TextDiagnosticPrinter.h"
+#include "llvm/Support/Debug.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "gtest/gtest.h"
+
+#define DEBUG_TYPE "format-test"
+
+using clang::tooling::ReplacementTest;
+using clang::tooling::toReplacements;
+
+namespace clang {
+namespace format {
+namespace {
+
+class FormatTestRawStrings : public ::testing::Test {
+protected:
+  enum StatusCheck { SC_ExpectComplete, SC_ExpectIncomplete, SC_DoNotCheck };
+
+  std::string format(llvm::StringRef Code,
+                     const FormatStyle &Style = getLLVMStyle(),
+                     StatusCheck CheckComplete = SC_ExpectComplete) {
+    DEBUG(llvm::errs() << "---\n");
+    DEBUG(llvm::errs() << Code << "\n\n");
+    std::vector<tooling::Range> Ranges(1, tooling::Range(0, Code.size()));
+    FormattingAttemptStatus Status;
+    tooling::Replacements Replaces =
+        reformat(Style, Code, Ranges, "<stdin>", &Status);
+    if (CheckComplete != SC_DoNotCheck) {
+      bool ExpectedCompleteFormat = CheckComplete == SC_ExpectComplete;
+      EXPECT_EQ(ExpectedCompleteFormat, Status.FormatComplete)
+          << Code << "\n\n";
+    }
+    ReplacementCount = Replaces.size();
+    auto Result = applyAllReplacements(Code, Replaces);
+    EXPECT_TRUE(static_cast<bool>(Result));
+    DEBUG(llvm::errs() << "\n" << *Result << "\n\n");
+    return *Result;
+  }
+
+  FormatStyle getStyleWithColumns(FormatStyle Style, unsigned ColumnLimit) {
+    Style.ColumnLimit = ColumnLimit;
+    return Style;
+  }
+
+  FormatStyle getLLVMStyleWithColumns(unsigned ColumnLimit) {
+    return getStyleWithColumns(getLLVMStyle(), ColumnLimit);
+  }
+
+  int ReplacementCount;
+
+  FormatStyle getRawStringPbStyleWithColumns(unsigned ColumnLimit) {
+    FormatStyle Style = getLLVMStyle();
+    Style.ColumnLimit = ColumnLimit;
+    Style.RawStringFormats = {{/*Delimiter=*/"pb",
+                               /*Kind=*/FormatStyle::LK_TextProto,
+                               /*BasedOnStyle=*/"google"}};
+    return Style;
+  }
+
+  FormatStyle getRawStringLLVMCppStyleBasedOn(std::string BasedOnStyle) {
+    FormatStyle Style = getLLVMStyle();
+    Style.RawStringFormats = {{/*Delimiter=*/"cpp",
+                               /*Kind=*/FormatStyle::LK_Cpp, BasedOnStyle}};
+    return Style;
+  }
+
+  FormatStyle getRawStringGoogleCppStyleBasedOn(std::string BasedOnStyle) {
+    FormatStyle Style = getGoogleStyle(FormatStyle::LK_Cpp);
+    Style.RawStringFormats = {{/*Delimiter=*/"cpp",
+                               /*Kind=*/FormatStyle::LK_Cpp, BasedOnStyle}};
+    return Style;
+  }
+
+  // Gcc 4.8 doesn't support raw string literals in macros, which breaks some
+  // build bots. We use this function instead.
+  void expect_eq(const std::string Expected, const std::string Actual) {
+    EXPECT_EQ(Expected, Actual);
+  }
+};
+
+TEST_F(FormatTestRawStrings, ReformatsAccordingToBaseStyle) {
+  // llvm style puts '*' on the right.
+  // google style puts '*' on the left.
+
+  // Use the llvm style if the raw string style has no BasedOnStyle.
+  expect_eq(R"test(int *i = R"cpp(int *p = nullptr;)cpp")test",
+            format(R"test(int * i = R"cpp(int * p = nullptr;)cpp")test",
+                   getRawStringLLVMCppStyleBasedOn("")));
+
+  // Use the google style if the raw string style has BasedOnStyle=google.
+  expect_eq(R"test(int *i = R"cpp(int* p = nullptr;)cpp")test",
+            format(R"test(int * i = R"cpp(int * p = nullptr;)cpp")test",
+                   getRawStringLLVMCppStyleBasedOn("google")));
+
+  // Use the llvm style if the raw string style has no BasedOnStyle=llvm.
+  expect_eq(R"test(int* i = R"cpp(int *p = nullptr;)cpp")test",
+            format(R"test(int * i = R"cpp(int * p = nullptr;)cpp")test",
+                   getRawStringGoogleCppStyleBasedOn("llvm")));
+}
+
+TEST_F(FormatTestRawStrings, MatchesDelimitersCaseSensitively) {
+  // Don't touch the 'PB' raw string, format the 'pb' raw string.
+  expect_eq(R"test(
+s = R"PB(item:1)PB";
+t = R"pb(item: 1)pb";)test",
+            format(R"test(
+s = R"PB(item:1)PB";
+t = R"pb(item:1)pb";)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  FormatStyle MixedStyle = getLLVMStyle();
+  MixedStyle.RawStringFormats = {
+      {/*Delimiter=*/"cpp", /*Kind=*/FormatStyle::LK_Cpp,
+       /*BasedOnStyle=*/"llvm"},
+      {/*Delimiter=*/"CPP", /*Kind=*/FormatStyle::LK_Cpp,
+       /*BasedOnStyle=*/"google"}};
+
+  // Format the 'cpp' raw string with '*' on the right.
+  // Format the 'CPP' raw string with '*' on the left.
+  // Do not format the 'Cpp' raw string.
+  // Do not format non-raw strings.
+  expect_eq(R"test(
+a = R"cpp(int *i = 0;)cpp";
+b = R"CPP(int* j = 0;)CPP";
+c = R"Cpp(int * k = 0;)Cpp";
+d = R"cpp(int * k = 0;)Cpp";)test",
+            format(R"test(
+a = R"cpp(int * i = 0;)cpp";
+b = R"CPP(int * j = 0;)CPP";
+c = R"Cpp(int * k = 0;)Cpp";
+d = R"cpp(int * k = 0;)Cpp";)test",
+                   MixedStyle));
+}
+
+TEST_F(FormatTestRawStrings, ReformatsShortRawStringsOnSingleLine) {
+  expect_eq(
+      R"test(P p = TP(R"pb()pb");)test",
+      format(
+          R"test(P p = TP(R"pb( )pb");)test",
+          getRawStringPbStyleWithColumns(40)));
+  expect_eq(
+      R"test(P p = TP(R"pb(item_1: 1)pb");)test",
+      format(
+          R"test(P p = TP(R"pb(item_1:1)pb");)test",
+          getRawStringPbStyleWithColumns(40)));
+  expect_eq(
+      R"test(P p = TP(R"pb(item_1: 1)pb");)test",
+      format(
+          R"test(P p = TP(R"pb(  item_1 :  1   )pb");)test",
+          getRawStringPbStyleWithColumns(40)));
+  expect_eq(
+      R"test(P p = TP(R"pb(item_1: 1 item_2: 2)pb");)test",
+      format(
+          R"test(P p = TP(R"pb(item_1:1 item_2:2)pb");)test",
+          getRawStringPbStyleWithColumns(40)));
+  expect_eq(
+      R"test(P p = TP(R"pb(item_1 <1> item_2: {2})pb");)test",
+      format(
+          R"test(P p = TP(R"pb(item_1<1> item_2:{2})pb");)test",
+          getRawStringPbStyleWithColumns(40)));
+
+  // Merge two short lines into one.
+  expect_eq(R"test(
+std::string s = R"pb(
+  item_1: 1 item_2: 2
+)pb";
+)test",
+            format(R"test(
+std::string s = R"pb(
+  item_1:1
+  item_2:2
+)pb";
+)test",
+                   getRawStringPbStyleWithColumns(40)));
+}
+
+TEST_F(FormatTestRawStrings, BreaksRawStringsExceedingColumnLimit) {
+  expect_eq(R"test(
+P p = TPPPPPPPPPPPPPPP(
+    R"pb(item_1: 1, item_2: 2)pb");)test",
+            format(R"test(
+P p = TPPPPPPPPPPPPPPP(R"pb(item_1: 1, item_2: 2)pb");)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  expect_eq(R"test(
+P p =
+    TPPPPPPPPPPPPPPP(
+        R"pb(item_1: 1,
+             item_2: 2,
+             item_3: 3)pb");)test",
+            format(R"test(
+P p = TPPPPPPPPPPPPPPP(R"pb(item_1: 1, item_2: 2, item_3: 3)pb");)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  expect_eq(R"test(
+P p = TP(R"pb(item_1 <1>
+              item_2: <2>
+              item_3 {})pb");)test",
+      format(R"test(
+P p = TP(R"pb(item_1<1> item_2:<2> item_3{ })pb");)test",
+          getRawStringPbStyleWithColumns(40)));
+
+  expect_eq(
+      R"test(
+P p = TP(R"pb(item_1: 1,
+              item_2: 2,
+              item_3: 3,
+              item_4: 4)pb");)test",
+      format(
+          R"test(
+P p = TP(R"pb(item_1: 1, item_2: 2, item_3: 3, item_4: 4)pb");)test",
+          getRawStringPbStyleWithColumns(40)));
+
+  expect_eq(R"test(
+P p = TPPPPPPPPPPPPPPP(
+    R"pb(item_1 <1>,
+         item_2: {2},
+         item_3: <3>,
+         item_4: {4})pb");)test",
+            format(R"test(
+P p = TPPPPPPPPPPPPPPP(R"pb(item_1<1>, item_2: {2}, item_3: <3>, item_4:{4})pb");)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  // Breaks before a short raw string exceeding the column limit.
+  expect_eq(R"test(
+FFFFFFFFFFFFFFFFFFFFFFFFFFF(
+    R"pb(key: 1)pb");
+P p = TPPPPPPPPPPPPPPPPPPPP(
+    R"pb(key: 2)pb");
+auto TPPPPPPPPPPPPPPPPPPPP =
+    R"pb(key: 3)pb";
+P p = TPPPPPPPPPPPPPPPPPPPP(
+    R"pb(i: 1, j: 2)pb");
+
+int f(string s) {
+  FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF(
+      R"pb(key: 1)pb");
+  P p = TPPPPPPPPPPPPPPPPPPPP(
+      R"pb(key: 2)pb");
+  auto TPPPPPPPPPPPPPPPPPPPP =
+      R"pb(key: 3)pb";
+  if (s.empty())
+    P p = TPPPPPPPPPPPPPPPPPPPP(
+        R"pb(i: 1, j: 2)pb");
+}
+)test",
+            format(R"test(
+FFFFFFFFFFFFFFFFFFFFFFFFFFF(R"pb(key:1)pb");
+P p = TPPPPPPPPPPPPPPPPPPPP(R"pb(key:2)pb");
+auto TPPPPPPPPPPPPPPPPPPPP = R"pb(key:3)pb";
+P p = TPPPPPPPPPPPPPPPPPPPP(R"pb(i: 1, j:2)pb");
+
+int f(string s) {
+  FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF(R"pb(key:1)pb");
+  P p = TPPPPPPPPPPPPPPPPPPPP(R"pb(key:2)pb");
+  auto TPPPPPPPPPPPPPPPPPPPP = R"pb(key:3)pb";
+  if (s.empty())
+    P p = TPPPPPPPPPPPPPPPPPPPP(R"pb(i: 1, j:2)pb");
+}
+)test",
+                   getRawStringPbStyleWithColumns(40)));
+}
+
+TEST_F(FormatTestRawStrings, FormatsRawStringArguments) {
+  expect_eq(R"test(
+P p = TP(R"pb(key {1})pb", param_2);)test",
+            format(R"test(
+P p = TP(R"pb(key{1})pb",param_2);)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  expect_eq(R"test(
+PPPPPPPPPPPPP(R"pb(keykeyk)pb",
+              param_2);)test",
+            format(R"test(
+PPPPPPPPPPPPP(R"pb(keykeyk)pb", param_2);)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  expect_eq(R"test(
+P p =
+    TP(R"pb(item: {i: 1, s: 's'}
+            item: {i: 2, s: 't'})pb");)test",
+            format(R"test(
+P p = TP(R"pb(item: {i: 1, s: 's'} item: {i: 2, s: 't'})pb");)test",
+                   getRawStringPbStyleWithColumns(40)));
+  expect_eq(R"test(
+FFFFFFFFFFFFFFFFFFF(
+    R"pb(key: "value")pb",
+    R"pb(key2: "value")pb");)test",
+            format(R"test(
+FFFFFFFFFFFFFFFFFFF(R"pb(key: "value")pb", R"pb(key2: "value")pb");)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  // Formats the first out of two arguments.
+  expect_eq(R"test(
+FFFFFFFF(R"pb(key: 1)pb", argument2);
+struct S {
+  const s =
+      f(R"pb(key: 1)pb", argument2);
+  void f() {
+    if (gol)
+      return g(R"pb(key: 1)pb",
+               132789237);
+    return g(R"pb(key: 1)pb", "172893");
+  }
+};)test",
+            format(R"test(
+FFFFFFFF(R"pb(key:1)pb", argument2);
+struct S {
+const s = f(R"pb(key:1)pb", argument2);
+void f() {
+  if (gol)
+    return g(R"pb(key:1)pb", 132789237);
+  return g(R"pb(key:1)pb", "172893");
+}
+};)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  // Formats the second out of two arguments.
+  expect_eq(R"test(
+FFFFFFFF(argument1, R"pb(key: 2)pb");
+struct S {
+  const s =
+      f(argument1, R"pb(key: 2)pb");
+  void f() {
+    if (gol)
+      return g(12784137,
+               R"pb(key: 2)pb");
+    return g(17283122, R"pb(key: 2)pb");
+  }
+};)test",
+            format(R"test(
+FFFFFFFF(argument1, R"pb(key:2)pb");
+struct S {
+const s = f(argument1, R"pb(key:2)pb");
+void f() {
+  if (gol)
+    return g(12784137, R"pb(key:2)pb");
+  return g(17283122, R"pb(key:2)pb");
+}
+};)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  // Formats two short raw string arguments.
+  expect_eq(R"test(
+FFFFF(R"pb(key: 1)pb", R"pb(key: 2)pb");)test",
+            format(R"test(
+FFFFF(R"pb(key:1)pb", R"pb(key:2)pb");)test",
+                   getRawStringPbStyleWithColumns(40)));
+  // TODO(krasimir): The original source code fits on one line, so the
+  // non-optimizing formatter is chosen. But after the formatting in protos is
+  // made, the code doesn't fit on one line anymore and further formatting
+  // splits it.
+  //
+  // Should we disable raw string formatting for the non-optimizing formatter?
+  expect_eq(R"test(
+FFFFFFF(R"pb(key: 1)pb", R"pb(key: 2)pb");)test",
+            format(R"test(
+FFFFFFF(R"pb(key:1)pb", R"pb(key:2)pb");)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  // Formats two short raw string arguments, puts second on newline.
+  expect_eq(R"test(
+FFFFFFFF(R"pb(key: 1)pb",
+         R"pb(key: 2)pb");)test",
+            format(R"test(
+FFFFFFFF(R"pb(key:1)pb", R"pb(key:2)pb");)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  // Formats both arguments.
+  expect_eq(R"test(
+FFFFFFFF(R"pb(key: 1)pb",
+         R"pb(key: 2)pb");
+struct S {
+  const s = f(R"pb(key: 1)pb",
+              R"pb(key: 2)pb");
+  void f() {
+    if (gol)
+      return g(R"pb(key: 1)pb",
+               R"pb(key: 2)pb");
+    return g(R"pb(k1)pb", R"pb(k2)pb");
+  }
+};)test",
+            format(R"test(
+FFFFFFFF(R"pb(key:1)pb", R"pb(key:2)pb");
+struct S {
+const s = f(R"pb(key:1)pb", R"pb(key:2)pb");
+void f() {
+  if (gol)
+    return g(R"pb(key:1)pb", R"pb(key:2)pb");
+  return g(R"pb( k1 )pb", R"pb( k2 )pb");
+}
+};)test",
+                   getRawStringPbStyleWithColumns(40)));
+}
+
+TEST_F(FormatTestRawStrings, RawStringStartingWithNewlines) {
+  expect_eq(R"test(
+std::string s = R"pb(
+  item_1: 1
+)pb";
+)test",
+            format(R"test(
+std::string s = R"pb(
+    item_1:1
+)pb";
+)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  expect_eq(R"test(
+std::string s = R"pb(
+
+  item_1: 1
+)pb";
+)test",
+            format(R"test(
+std::string s = R"pb(
+
+    item_1:1
+)pb";
+)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  expect_eq(R"test(
+std::string s = R"pb(
+  item_1: 1
+)pb";
+)test",
+            format(R"test(
+std::string s = R"pb(
+    item_1:1
+
+)pb";
+)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  expect_eq(R"test(
+std::string s = R"pb(
+  item_1: 1,
+  item_2: 2
+)pb";
+)test",
+            format(R"test(
+std::string s = R"pb(
+  item_1:1, item_2:2
+)pb";
+)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  expect_eq(R"test(
+std::string s = R"pb(
+  book {
+    title: "Alice's Adventures"
+    author: "Lewis Caroll"
+  }
+  book {
+    title: "Peter Pan"
+    author: "J. M. Barrie"
+  }
+)pb";
+)test",
+            format(R"test(
+std::string s = R"pb(
+    book { title: "Alice's Adventures" author: "Lewis Caroll" }
+    book { title: "Peter Pan" author: "J. M. Barrie" }
+)pb";
+)test",
+                   getRawStringPbStyleWithColumns(40)));
+}
+
+TEST_F(FormatTestRawStrings, BreaksBeforeRawStrings) {
+  expect_eq(R"test(
+ASSERT_TRUE(
+    ParseFromString(R"pb(item_1: 1)pb"),
+    ptr);)test",
+            format(R"test(
+ASSERT_TRUE(ParseFromString(R"pb(item_1: 1)pb"), ptr);)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  expect_eq(R"test(
+ASSERT_TRUE(toolong::ParseFromString(
+                R"pb(item_1: 1)pb"),
+            ptr);)test",
+            format(R"test(
+ASSERT_TRUE(toolong::ParseFromString(R"pb(item_1: 1)pb"), ptr);)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  expect_eq(R"test(
+ASSERT_TRUE(ParseFromString(
+                R"pb(item_1: 1,
+                     item_2: 2)pb"),
+            ptr);)test",
+            format(R"test(
+ASSERT_TRUE(ParseFromString(R"pb(item_1: 1, item_2: 2)pb"), ptr);)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  expect_eq(R"test(
+ASSERT_TRUE(
+    ParseFromString(
+        R"pb(item_1: 1 item_2: 2)pb"),
+    ptr);)test",
+            format(R"test(
+ASSERT_TRUE(ParseFromString(R"pb(item_1: 1 item_2: 2)pb"), ptr);)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+}
+
+TEST_F(FormatTestRawStrings, RawStringsInOperands) {
+  // Formats the raw string first operand of a binary operator expression.
+  expect_eq(R"test(auto S = R"pb(item_1: 1)pb" + rest;)test",
+            format(R"test(auto S = R"pb(item_1:1)pb" + rest;)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  expect_eq(R"test(
+auto S = R"pb(item_1: 1, item_2: 2)pb" +
+         rest;)test",
+            format(R"test(
+auto S = R"pb(item_1:1,item_2:2)pb"+rest;)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  expect_eq(R"test(
+auto S =
+    R"pb(item_1: 1 item_2: 2)pb" + rest;)test",
+            format(R"test(
+auto S = R"pb(item_1:1 item_2:2)pb"+rest;)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  expect_eq(R"test(
+auto S = R"pb(item_1: 1,
+              item_2: 2,
+              item_3: 3)pb" + rest;)test",
+            format(R"test(
+auto S = R"pb(item_1:1,item_2:2,item_3:3)pb"+rest;)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  expect_eq(R"test(
+auto S = R"pb(item_1: 1,
+              item_2: 2,
+              item_3: 3)pb" +
+         longlongrest;)test",
+            format(R"test(
+auto S = R"pb(item_1:1,item_2:2,item_3:3)pb"+longlongrest;)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  // Formats the raw string second operand of a binary operator expression.
+  expect_eq(R"test(auto S = first + R"pb(item_1: 1)pb";)test",
+            format(R"test(auto S = first + R"pb(item_1:1)pb";)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  expect_eq(R"test(
+auto S = first + R"pb(item_1: 1,
+                      item_2: 2)pb";)test",
+            format(R"test(
+auto S = first+R"pb(item_1:1,item_2:2)pb";)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  expect_eq(R"test(
+auto S = first + R"pb(item_1: 1
+                      item_2: 2)pb";)test",
+            format(R"test(
+auto S = first+R"pb(item_1:1 item_2:2)pb";)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  expect_eq(R"test(
+auto S = R"pb(item_1: 1,
+              item_2: 2,
+              item_3: 3)pb" + rest;)test",
+            format(R"test(
+auto S = R"pb(item_1:1,item_2:2,item_3:3)pb"+rest;)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  expect_eq(R"test(
+auto S = R"pb(item_1: 1,
+              item_2: 2,
+              item_3: 3)pb" +
+         longlongrest;)test",
+            format(R"test(
+auto S = R"pb(item_1:1,item_2:2,item_3:3)pb"+longlongrest;)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  // Formats the raw string operands in expressions.
+  expect_eq(R"test(
+auto S = R"pb(item_1: 1)pb" +
+         R"pb(item_2: 2)pb";
+)test",
+            format(R"test(
+auto S=R"pb(item_1:1)pb"+R"pb(item_2:2)pb";
+)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  expect_eq(R"test(
+auto S = R"pb(item_1: 1)pb" +
+         R"pb(item_2: 2)pb" +
+         R"pb(item_3: 3)pb";
+)test",
+            format(R"test(
+auto S=R"pb(item_1:1)pb"+R"pb(item_2:2)pb"+R"pb(item_3:3)pb";
+)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  expect_eq(R"test(
+auto S = (count < 3)
+             ? R"pb(item_1: 1)pb"
+             : R"pb(item_2: 2)pb";
+)test",
+            format(R"test(
+auto S=(count<3)?R"pb(item_1:1)pb":R"pb(item_2:2)pb";
+)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  expect_eq(R"test(
+auto S =
+    (count < 3)
+        ? R"pb(item_1: 1, item_2: 2)pb"
+        : R"pb(item_3: 3)pb";
+)test",
+            format(R"test(
+auto S=(count<3)?R"pb(item_1:1,item_2:2)pb":R"pb(item_3:3)pb";
+)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+  expect_eq(R"test(
+auto S =
+    (count < 3)
+        ? R"pb(item_1: 1)pb"
+        : R"pb(item_2: 2, item_3: 3)pb";
+)test",
+            format(R"test(
+auto S=(count<3)?R"pb(item_1:1)pb":R"pb(item_2:2,item_3:3)pb";
+)test",
+                   getRawStringPbStyleWithColumns(40)));
+
+}
+
+TEST_F(FormatTestRawStrings, PrefixAndSuffixAlignment) {
+  // Keep the suffix at the end of line if not on newline.
+  expect_eq(R"test(
+int s() {
+  auto S = PTP(
+      R"pb(
+        item_1: 1,
+        item_2: 2)pb");
+})test",
+            format(R"test(
+int s() {
+  auto S = PTP(
+      R"pb(
+      item_1: 1,
+      item_2: 2)pb");
+})test",
+                   getRawStringPbStyleWithColumns(20)));
+
+  // Align the suffix with the surrounding FirstIndent if the prefix is not on
+  // a line of its own.
+  expect_eq(R"test(
+int s() {
+  auto S = PTP(
+      R"pb(
+        item_1: 1,
+        item_2: 2
+      )pb");
+})test",
+            format(R"test(
+int s() {
+  auto S = PTP(R"pb(
+      item_1: 1,
+      item_2: 2
+      )pb");
+})test",
+                   getRawStringPbStyleWithColumns(20)));
+
+  // Align the prefix with the suffix if both the prefix and suffix are on a
+  // line of their own.
+  expect_eq(R"test(
+int s() {
+  auto S = PTP(
+      R"pb(
+        item_1: 1,
+        item_2: 2,
+      )pb");
+})test",
+            format(R"test(
+int s() {
+  auto S = PTP(
+      R"pb(
+      item_1: 1,
+      item_2: 2,
+      )pb");
+})test",
+                   getRawStringPbStyleWithColumns(20)));
+}
+
+TEST_F(FormatTestRawStrings, EstimatesPenalty) {
+  // The penalty for characters exceeding the column limit in the raw string
+  // forces 'hh' to be put on a newline.
+  expect_eq(R"test(
+ff(gggggg,
+   hh(R"pb(key {
+             i1: k1
+             i2: k2
+           })pb"));
+)test",
+            format(R"test(
+ff(gggggg, hh(R"pb(key {
+    i1: k1
+    i2: k2
+    })pb"));
+)test",
+                   getRawStringPbStyleWithColumns(20)));
+}
+
+TEST_F(FormatTestRawStrings, DontFormatNonRawStrings) {
+  expect_eq(R"test(a = R"pb(key:value)";)test",
+            format(R"test(a = R"pb(key:value)";)test",
+                   getRawStringPbStyleWithColumns(20)));
+}
+
+} // end namespace
+} // end namespace format
+} // end namespace clang