Refactor clang-format's formatter.

[klimek]

Summary

a) Pull out a class LevelIndentTracker whose responsibility is to keep track

of the indent of levels across multiple annotated lines.

b) Put all responsibility for merging lines into the LineJoiner; make the

LineJoiner iterate over the lines so we never operate on a line that might
be merged later; this makes the interface safer to use.

c) Move formatting of the end-of-file whitespace into formatFirstToken.

Fix bugs that became obvious after the refactoring:

1. We would not format lines with offsets correctly inside nested blocks if only the outer expression was affected: int x = s({ clang-format only this line class X { public: ^ this starts at the non-modified indnent level; previously we would // not fix this, now we correctly outdent it. void f(); }; });
2. We would incorrectly align comments across lines that do not have comments for lines with nested blocks: int expression; with comment int x = s({ int y; comment int z; we would incorrectly align this comment with the comment on 'expression' });

[djasper]

I like it.

lib/Format/UnwrappedLineFormatter.cpp

28

I think "Tracks the indent level of \c AnnotatedLines across levels" is slightly clearer as this tracks the indent for multiple lines.

29

I think this should be a class, not a struct.

34

Just use:

for (unsigned i = 0; i != StartLevel; ++i) ...

StartLevel already is a POD type (unsigned even) and we don't get anything.

39–43

Ignore.. Phab doesn't let me delete this comment (Eclipse-like behavior).

62

Isn't this missing an

Offset = getIndentOffset(*Line.First);

106–110

As obvious from my comment above where I probably show that I am not understanding something, it is unclear how this needs to be a class variable instead of a variable calculated per line. Please comment.

132–144

Remove all the braces.

440–441

Can you change the order? My OCD finds it weird to have Next after End ;-).

815–819

Both, the comment and the variable name are a bit confusing. Lets discuss this offline.

829

This assert seems a bit odd ..

833

The actions for "Format" and "!Format" seem pretty interleaved here. Couldn't we group them more?

843

No braces.. Here and below.

895

Merge this line with the next (there already is a clang-format bug for that).

lib/Format/UnwrappedLineFormatter.h

53

If the implementation isn't in the header anymore, this might deserve a small comment.

[klimek]

Implement review comments.

lib/Format/UnwrappedLineFormatter.cpp

28

Done.

29

Yep.

34

Done.

39–43

Ignored</irony>

62

Nope. Added class comment and local method comment.

106–110

I hope the additional comments on the methods clarify...

132–144

Done. ALL.

440–441

The problem is that my OCD finds it weird to have a non-const member in between the const ones. Here, I added an empty line, perhaps that helps?

815–819

Resolved via that: moved into the !Format part and rewrote comment.

829

Resolved via chat: moved around and duplicated the condition.

833

Resolved via chat: moved some of the code around.

843

Done.

895

Done.

lib/Format/UnwrappedLineFormatter.h

53

Yep.

[djasper]

Looks very nice now.

lib/Format/UnwrappedLineFormatter.cpp

30

This should be stronger, i.e. updateIndentForLine must be called...

34

This should somehow include that updateLevelIndentFromUnmodifiedLine needs to be called *in addition* to updateIndentForLine.

51

I'd probably just call this nextLine.

71

maybe adjustToUnmodifiedLine

[klimek]

lib/Format/UnwrappedLineFormatter.cpp

30

Done.

34

Done.

51

Done.

71

Done.

[klimek]

Commited as r237104

lib/Format/UnwrappedLineFormatter.h

UnwrappedLineFormatter(ContinuationIndenter *Indenter,
         Keywords(Keywords), IncompleteFormat(IncompleteFormat) {}
 
   /// \brief Format the current block and return the penalty.
   unsigned format(const SmallVectorImpl<AnnotatedLine *> &Lines,
                   bool DryRun = false, int AdditionalIndent = 0,
                   bool FixBadIndentation = false);
 
 private:
-  /// \brief Get the offset of the line relatively to the level.
-  ///
-  /// For example, 'public:' labels in classes are offset by 1 or 2
-  /// characters to the left from their level.
-  int getIndentOffset(const FormatToken &RootToken) {
-    if (Style.Language == FormatStyle::LK_Java ||
-        Style.Language == FormatStyle::LK_JavaScript)
-      return 0;
-    if (RootToken.isAccessSpecifier(false) ||
-        RootToken.isObjCAccessSpecifier() ||
-        (RootToken.is(Keywords.kw_signals) && RootToken.Next &&
-         RootToken.Next->is(tok::colon)))
-      return Style.AccessModifierOffset;
-    return 0;
-  }
-
   /// \brief Add a new line and the required indent before the first Token
   /// of the \c UnwrappedLine if there was no structural parsing error.
   void formatFirstToken(FormatToken &RootToken,
                         const AnnotatedLine *PreviousLine, unsigned IndentLevel,
                         unsigned Indent, bool InPPDirective);
 
-  /// \brief Get the indent of \p Level from \p IndentForLevel.
-  ///
-  /// \p IndentForLevel must contain the indent for the level \c l
-  /// at \p IndentForLevel[l], or a value < 0 if the indent for
-  /// that level is unknown.
-  unsigned getIndent(ArrayRef<int> IndentForLevel, unsigned Level);
-
-  void join(AnnotatedLine &A, const AnnotatedLine &B);
-
-  unsigned getColumnLimit(bool InPPDirective) const {
-    // In preprocessor directives reserve two chars for trailing " \"
-    return Style.ColumnLimit - (InPPDirective ? 2 : 0);
-  }
+  unsigned getColumnLimit(bool InPPDirective, const AnnotatedLine *NextLine) const;
 
   // Cache to store the penalty of formatting a vector of AnnotatedLines
   // starting from a specific additional offset. Improves performance if there
   // are many nested blocks.
   std::map<std::pair<const SmallVectorImpl<AnnotatedLine *> *, unsigned>,
            unsigned> PenaltyCache;
 
   ContinuationIndenter *Indenter;

lib/Format/UnwrappedLineFormatter.cpp

 
 bool startsExternCBlock(const AnnotatedLine &Line) {
   const FormatToken *Next = Line.First->getNextNonComment();
   const FormatToken *NextNext = Next ? Next->getNextNonComment() : nullptr;
   return Line.First->is(tok::kw_extern) && Next && Next->isStringLiteral() &&
          NextNext && NextNext->is(tok::l_brace);
 }
 
+/// \brief Tracks the indent level for an \c AnnotatedLine across levels.
+struct LevelIndentTracker {
+  LevelIndentTracker(const FormatStyle &Style,
+                     const AdditionalKeywords &Keywords, unsigned StartLevel,
+                     int AdditionalIndent)
+      : Style(Style), Keywords(Keywords) {
+    for (unsigned i = 0, e = StartLevel; i != e; ++i)
+      IndentForLevel.push_back(Style.IndentWidth * i + AdditionalIndent);
+  }
+
+  /// \brief Returns the indent for the current line.
+  unsigned getIndent() const { return Indent; }
+
+  /// \brief Update the indent state given that \p Line is going to be formatted
+  /// next.
+  void updateIndentForLine(const AnnotatedLine &Line) {
+    Offset = getIndentOffset(*Line.First);
+    if (Line.InPPDirective) {
+      Indent = Line.Level * Style.IndentWidth;
+    } else {
+      while (IndentForLevel.size() <= Line.Level)
+        IndentForLevel.push_back(-1);
+      IndentForLevel.resize(Line.Level + 1);
+      Indent = getIndent(IndentForLevel, Line.Level);
+    }
+    if (static_cast<int>(Indent) + Offset >= 0)
+      Indent += Offset;
+  }
+
+  /// \brief Update the level indent to adapt to the given \p Line.
+  ///
+  /// When a line is not formatted, we move the subsequent lines on the same
+  /// level to the same indent.
+  void updateLevelIndentFromUnmodifiedLine(const AnnotatedLine &Line) {
+    unsigned LevelIndent = Line.First->OriginalColumn;
+    if (static_cast<int>(LevelIndent) - Offset >= 0)
+      LevelIndent -= Offset;
+    if ((Line.First->isNot(tok::comment) || IndentForLevel[Line.Level] == -1) &&
+        !Line.InPPDirective)
+      IndentForLevel[Line.Level] = LevelIndent;
+  }
+
+private:
+  /// \brief Get the offset of the line relatively to the level.
+  ///
+  /// For example, 'public:' labels in classes are offset by 1 or 2
+  /// characters to the left from their level.
+  int getIndentOffset(const FormatToken &RootToken) {
+    if (Style.Language == FormatStyle::LK_Java ||
+        Style.Language == FormatStyle::LK_JavaScript)
+      return 0;
+    if (RootToken.isAccessSpecifier(false) ||
+        RootToken.isObjCAccessSpecifier() ||
+        (RootToken.is(Keywords.kw_signals) && RootToken.Next &&
+         RootToken.Next->is(tok::colon)))
+      return Style.AccessModifierOffset;
+    return 0;
+  }
+
+  /// \brief Get the indent of \p Level from \p IndentForLevel.
+  ///
+  /// \p IndentForLevel must contain the indent for the level \c l
+  /// at \p IndentForLevel[l], or a value < 0 if the indent for
+  /// that level is unknown.
+  unsigned getIndent(ArrayRef<int> IndentForLevel, unsigned Level) {
+    if (IndentForLevel[Level] != -1)
+      return IndentForLevel[Level];
+    if (Level == 0)
+      return 0;
+    return getIndent(IndentForLevel, Level - 1) + Style.IndentWidth;
+  }
+
+  const FormatStyle &Style;
+  const AdditionalKeywords &Keywords;
+
+  /// \brief The indent in characters for each level.
+  std::vector<int> IndentForLevel;
+
+  /// \brief Offset of the current line relative to the indent level.
+  ///
+  /// For example, the 'public' keywords is often indented with a negative
+  /// offset.
+  int Offset = 0;
+
+  /// \brief The current line's indent.
+  unsigned Indent = 0;
+};
+
 class LineJoiner {
 public:
-  LineJoiner(const FormatStyle &Style, const AdditionalKeywords &Keywords)
-      : Style(Style), Keywords(Keywords) {}
+  LineJoiner(const FormatStyle &Style, const AdditionalKeywords &Keywords,
+             const SmallVectorImpl<AnnotatedLine *> &Lines)
+      : Style(Style), Keywords(Keywords), End(Lines.end()),
+        Next(Lines.begin()) {}
+
+  /// \brief Returns the next line, merging multiple lines into one if possible.
+  const AnnotatedLine *getNextMergedLine(bool DryRun,
+                                         LevelIndentTracker &IndentTracker) {
+    if (Next == End)
+      return nullptr;
+    const AnnotatedLine *Current = *Next;
+    IndentTracker.updateIndentForLine(*Current);
+    unsigned MergedLines =
+        tryFitMultipleLinesInOne(IndentTracker.getIndent(), Next, End);
+    if (MergedLines > 0 && Style.ColumnLimit == 0) {
+      // Disallow line merging if there is a break at the start of one of the
+      // input lines.
+      for (unsigned i = 0; i < MergedLines; ++i) {
+        if (Next[i + 1]->First->NewlinesBefore > 0)
+          MergedLines = 0;
+      }
+    }
+    if (!DryRun) {
+      for (unsigned i = 0; i < MergedLines; ++i) {
+        join(*Next[i], *Next[i + 1]);
+      }
+    }
+    Next = Next + MergedLines + 1;
+    return Current;
+  }
 
+private:
   /// \brief Calculates how many lines can be merged into 1 starting at \p I.
   unsigned
   tryFitMultipleLinesInOne(unsigned Indent,
                            SmallVectorImpl<AnnotatedLine *>::const_iterator I,
                            SmallVectorImpl<AnnotatedLine *>::const_iterator E) {
     // Can't join the last line with anything.
     if (I + 1 == E)
       return 0;
tryFitMultipleLinesInOne(unsigned Indent,
     }
     if (TheLine->InPPDirective &&
         (TheLine->First->HasUnescapedNewline || TheLine->First->IsFirst)) {
       return tryMergeSimplePPDirective(I, E, Limit);
     }
     return 0;
   }
 
-private:
   unsigned
   tryMergeSimplePPDirective(SmallVectorImpl<AnnotatedLine *>::const_iterator I,
                             SmallVectorImpl<AnnotatedLine *>::const_iterator E,
                             unsigned Limit) {
     if (Limit == 0)
       return 0;
     if (I + 2 != E && I[2]->InPPDirective && !I[2]->First->HasUnescapedNewline)
       return 0;
private:
   bool containsMustBreak(const AnnotatedLine *Line) {
     for (const FormatToken *Tok = Line->First; Tok; Tok = Tok->Next) {
       if (Tok->MustBreakBefore)
         return true;
     }
     return false;
   }
 
+  void join(AnnotatedLine &A, const AnnotatedLine &B) {
+    assert(!A.Last->Next);
+    assert(!B.First->Previous);
+    if (B.Affected)
+      A.Affected = true;
+    A.Last->Next = B.First;
+    B.First->Previous = A.Last;
+    B.First->CanBreakBefore = true;
+    unsigned LengthA = A.Last->TotalLength + B.First->SpacesRequiredBefore;
+    for (FormatToken *Tok = B.First; Tok; Tok = Tok->Next) {
+      Tok->TotalLength += LengthA;
+      A.Last = Tok;
+    }
+  }
+
   const FormatStyle &Style;
   const AdditionalKeywords &Keywords;
+  const SmallVectorImpl<AnnotatedLine*>::const_iterator End;
+  SmallVectorImpl<AnnotatedLine*>::const_iterator Next;
 };
 
 static void markFinalized(FormatToken *Tok) {
   for (; Tok; Tok = Tok->Next) {
     Tok->Finalized = true;
     for (AnnotatedLine *Child : Tok->Children)
       markFinalized(Child->First);
   }

 };
 
 } // namespace
 
 unsigned
 UnwrappedLineFormatter::format(const SmallVectorImpl<AnnotatedLine *> &Lines,
                                bool DryRun, int AdditionalIndent,
                                bool FixBadIndentation) {
-  LineJoiner Joiner(Style, Keywords);
+  LineJoiner Joiner(Style, Keywords, Lines);
 
   // Try to look up already computed penalty in DryRun-mode.
   std::pair<const SmallVectorImpl<AnnotatedLine *> *, unsigned> CacheKey(
       &Lines, AdditionalIndent);
   auto CacheIt = PenaltyCache.find(CacheKey);
   if (DryRun && CacheIt != PenaltyCache.end())
     return CacheIt->second;
 
   assert(!Lines.empty());
   unsigned Penalty = 0;
-  std::vector<int> IndentForLevel;
-  for (unsigned i = 0, e = Lines[0]->Level; i != e; ++i)
-    IndentForLevel.push_back(Style.IndentWidth * i + AdditionalIndent);
+  LevelIndentTracker IndentTracker(Style, Keywords, Lines[0]->Level,
+                                   AdditionalIndent);
   const AnnotatedLine *PreviousLine = nullptr;
-  for (SmallVectorImpl<AnnotatedLine *>::const_iterator I = Lines.begin(),
-                                                        E = Lines.end();
-       I != E; ++I) {
-    const AnnotatedLine &TheLine = **I;
-    const FormatToken *FirstTok = TheLine.First;
-    int Offset = getIndentOffset(*FirstTok);
-
-    // Determine indent and try to merge multiple unwrapped lines.
-    unsigned Indent;
-    if (TheLine.InPPDirective) {
-      Indent = TheLine.Level * Style.IndentWidth;
-    } else {
-      while (IndentForLevel.size() <= TheLine.Level)
-        IndentForLevel.push_back(-1);
-      IndentForLevel.resize(TheLine.Level + 1);
-      Indent = getIndent(IndentForLevel, TheLine.Level);
-    }
-    unsigned LevelIndent = Indent;
-    if (static_cast<int>(Indent) + Offset >= 0)
-      Indent += Offset;
-
-    // Merge multiple lines if possible.
-    unsigned MergedLines = Joiner.tryFitMultipleLinesInOne(Indent, I, E);
-    if (MergedLines > 0 && Style.ColumnLimit == 0) {
-      // Disallow line merging if there is a break at the start of one of the
-      // input lines.
-      for (unsigned i = 0; i < MergedLines; ++i) {
-        if (I[i + 1]->First->NewlinesBefore > 0)
-          MergedLines = 0;
-      }
-    }
-    if (!DryRun) {
-      for (unsigned i = 0; i < MergedLines; ++i) {
-        join(*I[i], *I[i + 1]);
-      }
-    }
-    I += MergedLines;
-
+  const AnnotatedLine *NextLine = nullptr;
+  for (const AnnotatedLine *Line =
+           Joiner.getNextMergedLine(DryRun, IndentTracker);
+       Line; Line = NextLine) {
+    const AnnotatedLine &TheLine = *Line;
+    unsigned Indent = IndentTracker.getIndent();
     bool FixIndentation =
-        FixBadIndentation && (LevelIndent != FirstTok->OriginalColumn);
+        FixBadIndentation && (Indent != TheLine.First->OriginalColumn);
     bool ShouldFormat = TheLine.Affected || FixIndentation;
-    if (TheLine.First->is(tok::eof)) {
-      if (PreviousLine && PreviousLine->Affected && !DryRun) {
-        // Remove the file's trailing whitespace.
-        unsigned Newlines = std::min(FirstTok->NewlinesBefore, 1u);
-        Whitespaces->replaceWhitespace(*TheLine.First, Newlines,
-                                       /*IndentLevel=*/0, /*Spaces=*/0,
-                                       /*TargetColumn=*/0);
-      }
-    } else if (TheLine.Type != LT_Invalid && ShouldFormat) {
-      if (FirstTok->WhitespaceRange.isValid()) {
-        if (!DryRun)
-          formatFirstToken(*TheLine.First, PreviousLine, TheLine.Level, Indent,
-                           TheLine.InPPDirective);
-      } else {
-        Indent = LevelIndent = FirstTok->OriginalColumn;
-      }
+    bool Format = TheLine.Type != LT_Invalid && ShouldFormat;
+    // If we format a line, each AnnotatedLine will start a new line (otherwise
+    // we merge them into the same AnnotatedLine).
+    // If we do not format a line, this does not hold true - in that case an
+    // arbitrary number of annotated lines can be in one line.
+    bool AnnotatedLineStartsNewLine =
+        TheLine.First->NewlinesBefore > 0 || TheLine.First->IsFirst;
+    if (!Format && AnnotatedLineStartsNewLine) {
+      // Adapt following lines on the current indent level to the same level.
+      IndentTracker.updateLevelIndentFromUnmodifiedLine(TheLine);
+    }
+    NextLine = Joiner.getNextMergedLine(DryRun, IndentTracker);
+    // We cannot format this line; if the reason is that the line had a
+    // parsing error, remember that.
+    if (!Format && ShouldFormat && IncompleteFormat) {
+      assert(TheLine.Type == LT_Invalid);
+      *IncompleteFormat = true;
+
+    if (Format) {
 
-      // If everything fits on a single line, just put it there.
-      unsigned ColumnLimit = Style.ColumnLimit;
-      if (I + 1 != E) {
-        AnnotatedLine *NextLine = I[1];
-        if (NextLine->InPPDirective && !NextLine->First->HasUnescapedNewline)
-          ColumnLimit = getColumnLimit(TheLine.InPPDirective);
-      }
+      unsigned ColumnLimit = getColumnLimit(TheLine.InPPDirective, NextLine);
+      bool FitsIntoOneLine =
+          TheLine.Last->TotalLength + Indent <= ColumnLimit ||
+          TheLine.Type == LT_ImportStatement;
 
-      if (TheLine.Last->TotalLength + Indent <= ColumnLimit ||
-          TheLine.Type == LT_ImportStatement) {
-        Penalty += NoLineBreakFormatter(Indenter, Whitespaces, Style, this)
-                       .formatLine(TheLine, Indent, DryRun);
-      } else if (Style.ColumnLimit == 0) {
+      if (Style.ColumnLimit == 0) {
         NoColumnLimitLineFormatter(Indenter, Whitespaces, Style, this)
             .formatLine(TheLine, Indent, DryRun);
+      } else if (FitsIntoOneLine) {
+        Penalty += NoLineBreakFormatter(Indenter, Whitespaces, Style, this)
+                       .formatLine(TheLine, Indent, DryRun);
       } else {
         Penalty += OptimizingLineFormatter(Indenter, Whitespaces, Style, this)
                        .formatLine(TheLine, Indent, DryRun);
       }
-
-      if (!TheLine.InPPDirective)
-        IndentForLevel[TheLine.Level] = LevelIndent;
-    } else if (TheLine.ChildrenAffected) {
-      format(TheLine.Children, DryRun);
     } else {
-      // Format the first token if necessary, and notify the WhitespaceManager
-      // about the unchanged whitespace.
-      for (FormatToken *Tok = TheLine.First; Tok; Tok = Tok->Next) {
-        if (Tok == TheLine.First && (Tok->NewlinesBefore > 0 || Tok->IsFirst)) {
-          unsigned LevelIndent = Tok->OriginalColumn;
-          if (!DryRun) {
-            // Remove trailing whitespace of the previous line.
-            if ((PreviousLine && PreviousLine->Affected) ||
-                TheLine.LeadingEmptyLinesAffected) {
-              formatFirstToken(*Tok, PreviousLine, TheLine.Level, LevelIndent,
-                               TheLine.InPPDirective);
-            } else {
-              Whitespaces->addUntouchableToken(*Tok, TheLine.InPPDirective);
-            }
-          }
-
-          if (static_cast<int>(LevelIndent) - Offset >= 0)
-            LevelIndent -= Offset;
-          if ((Tok->isNot(tok::comment) ||
-               IndentForLevel[TheLine.Level] == -1) &&
-              !TheLine.InPPDirective)
-            IndentForLevel[TheLine.Level] = LevelIndent;
-        } else if (!DryRun) {
+      // If no token in the current line is affected, we still need to format
+      // affected children.
+      if (TheLine.ChildrenAffected) {
+        format(TheLine.Children, DryRun);
+      }
+        bool ReformatLeadingWhitespace =
+            AnnotatedLineStartsNewLine &&
+            ((PreviousLine && PreviousLine->Affected) ||
+             TheLine.LeadingEmptyLinesAffected);
+        // Format the first token.
+        if (ReformatLeadingWhitespace) {
+          formatFirstToken(*TheLine.First, PreviousLine, TheLine.Level,
+                           TheLine.First->OriginalColumn,
+          Whitespaces->addUntouchableToken(*TheLine.First,
+                                           TheLine.InPPDirective);
+        // Notify the WhitespaceManager about the unchanged whitespace.
+        for (FormatToken *Tok = TheLine.First->Next; Tok; Tok = Tok->Next) {
           Whitespaces->addUntouchableToken(*Tok, TheLine.InPPDirective);
         }
       }
     }
-    if (TheLine.Type == LT_Invalid && ShouldFormat && IncompleteFormat)
-      *IncompleteFormat = true;
     if (!DryRun)
       markFinalized(TheLine.First);
-    PreviousLine = *I;
+    PreviousLine = &TheLine;
   }
   PenaltyCache[CacheKey] = Penalty;
   return Penalty;
 }
 
 void UnwrappedLineFormatter::formatFirstToken(FormatToken &RootToken,
                                               const AnnotatedLine *PreviousLine,
                                               unsigned IndentLevel,
                                               unsigned Indent,
                                               bool InPPDirective) {
+  if (RootToken.is(tok::eof)) {
+    unsigned Newlines = std::min(RootToken.NewlinesBefore, 1u);
+    Whitespaces->replaceWhitespace(RootToken, Newlines, /*IndentLevel=*/0,
+                                   /*Spaces=*/0,
+                                   /*TargetColumn=*/0);
+    return;
+  }
   unsigned Newlines =
       std::min(RootToken.NewlinesBefore, Style.MaxEmptyLinesToKeep + 1);
   // Remove empty lines before "}" where applicable.
   if (RootToken.is(tok::r_brace) &&
       (!RootToken.Next ||
        (RootToken.Next->is(tok::semi) && !RootToken.Next->Next)))
     Newlines = std::min(Newlines, 1u);
   if (Newlines == 0 && !RootToken.IsFirst)
if (PreviousLine && PreviousLine->First->isAccessSpecifier() &&
       (!PreviousLine->InPPDirective || !RootToken.HasUnescapedNewline))
     Newlines = std::min(1u, Newlines);
 
   Whitespaces->replaceWhitespace(RootToken, Newlines, IndentLevel, Indent,
                                  Indent, InPPDirective &&
                                              !RootToken.HasUnescapedNewline);
 }
 
-/// \brief Get the indent of \p Level from \p IndentForLevel.
-///
-/// \p IndentForLevel must contain the indent for the level \c l
-/// at \p IndentForLevel[l], or a value < 0 if the indent for
-/// that level is unknown.
-unsigned UnwrappedLineFormatter::getIndent(ArrayRef<int> IndentForLevel,
-                                           unsigned Level) {
-  if (IndentForLevel[Level] != -1)
-    return IndentForLevel[Level];
-  if (Level == 0)
-    return 0;
-  return getIndent(IndentForLevel, Level - 1) + Style.IndentWidth;
-}
-
-void UnwrappedLineFormatter::join(AnnotatedLine &A, const AnnotatedLine &B) {
-  assert(!A.Last->Next);
-  assert(!B.First->Previous);
-  if (B.Affected)
-    A.Affected = true;
-  A.Last->Next = B.First;
-  B.First->Previous = A.Last;
-  B.First->CanBreakBefore = true;
-  unsigned LengthA = A.Last->TotalLength + B.First->SpacesRequiredBefore;
-  for (FormatToken *Tok = B.First; Tok; Tok = Tok->Next) {
-    Tok->TotalLength += LengthA;
-    A.Last = Tok;
-  }
+unsigned
+UnwrappedLineFormatter::getColumnLimit(bool InPPDirective,
+                                       const AnnotatedLine *NextLine) const {
+  // In preprocessor directives reserve two chars for trailing " \" if the
+  // next line continues the preprocessor directive.
+  bool ContinuesPPDirective =
+      InPPDirective && NextLine && NextLine->InPPDirective &&
+      // If there is an unescaped newline between this line and the next, the
+      // next line starts a new preprocessor directive.
+      !NextLine->First->HasUnescapedNewline;
+  return Style.ColumnLimit - (ContinuesPPDirective ? 2 : 0);
 }
 
 } // namespace format
 } // namespace clang

unittests/Format/FormatTest.cpp

TEST_F(FormatTest, MacroDefinitionsWithIncompleteCode) {
   verifyIncompleteFormat("#define STR(x) #x\n"
                          "f(STR(this_is_a_string_literal{));");
   verifyFormat("#pragma omp threadprivate( \\\n"
                "    y)), // expected-warning",
                getLLVMStyleWithColumns(28));
   verifyFormat("#d, = };");
   verifyFormat("#if \"a");
   verifyIncompleteFormat("({\n"
-                         "#define b }\\\n"
+                         "#define b     \\\n"
+                         "  }           \\\n"
                          "  a\n"
-                         "a");
+                         "a", getLLVMStyleWithColumns(15));
   verifyFormat("#define A     \\\n"
                "  {           \\\n"
                "    {\n"
                "#define B     \\\n"
                "  }           \\\n"
                "  }",
                getLLVMStyleWithColumns(15));
   verifyNoCrash("#if a\na(\n#else\n#endif\n{a");
verifyFormat(
       "      int i; // break\n"
       "    },\n"
       "    aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa(bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb,\n"
       "                                     ccccccccccccccccc));");
   verifyFormat("DEBUG({\n"
                "  if (a)\n"
                "    f();\n"
                "});");
+  EXPECT_EQ("int longlongname; // comment\n"
+            "int x = f({\n"
+            "  int x; // comment\n"
+            "  int y; // comment\n"
+            "});",
+            format("int longlongname; // comment\n"
+                   "int x = f({\n"
+                   "  int x; // comment\n"
+                   "  int y; // comment\n"
+                   "});",
+                   65, 0, getLLVMStyle()));
+  EXPECT_EQ("int s = f({\n"
+            "  class X {\n"
+            "  public:\n"
+            "    void f();\n"
+            "  };\n"
+            "});",
+            format("int s = f({\n"
+                   "  class X {\n"
+                   "    public:\n"
+                   "    void f();\n"
+                   "  };\n"
+                   "});",
+                   0, 0, getLLVMStyle()));
 }
 
 TEST_F(FormatTest, LayoutBlockInsideStatement) {
   EXPECT_EQ("SOME_MACRO { int i; }\n"
             "int i;",
             format("  SOME_MACRO  {int i;}  int i;"));
 }