[clang-format] Refactor WhitespaceManager and friends

[djasper]

Summary

The main motivation behind this is to cleanup the WhitespaceManager and make it more extensible for future alignment etc. features. Specifically, WhitespaceManager has started to copy more and more code that is already present in FormatToken. Instead, I think it makes more sense to actually store a reference to each FormatToken for each change.

This has as a consequence led to a change in the calculation of indent levels. Now, we actually compute them for each Token ahead of time, which should be more efficient as it removes an unsigned value for the ParenState, which is used during the combinatorial exploration of the solution space.

No functional changes intended.

[klimek]

Generally looks like the right direction, minus that I'm not sure yet what the plan for things broken in BreakableToken are.

lib/Format/TokenAnnotator.cpp

1843

Perhaps add an assert that we don't underflow.

lib/Format/UnwrappedLineFormatter.cpp

904–908

I'm not sure I understand the change in the function signature. Given that we really only need InPPDirective and FirstToken, it seems unnecessary to hand in the whole line? (in the spirit of minimal interfaces)

lib/Format/WhitespaceManager.h

109–110

What's the proposed fix?

[djasper]

lib/Format/UnwrappedLineFormatter.cpp

904–908

I think this interface makes more sense:

• I don't think it's significantly "larger" as we already have access to the whole previous line.
• Having a function that's called formatFirstToken, but can actually be called with an arbitrary token seems weird.

lib/Format/WhitespaceManager.h

109–110

Removed InToken (added that at first, but found the other one later).

I don't have a proposed fix. I just moved this comment from the original line 122.

[klimek]

lg

[djasper]

Added assert. Removed unused InToken.

[djasper]

Submitted as r293616.

[krasimir]

lib/Format/WhitespaceManager.cpp

178

What happened to the tok::unknown case?

[krasimir]

looks good!

[djasper]

lib/Format/WhitespaceManager.cpp

178

We actually don't set the token back to unknown in replaceWhitespaceInToken anymore if this is a comment. So I moved this logic back into the if above.

lib/Format/BreakableToken.h

virtual void replaceWhitespaceBefore(unsigned LineIndex,
                                        WhitespaceManager &Whitespaces) {}
 
   /// \brief Updates the next token of \p State to the next token after this
   /// one. This can be used when this token manages a set of underlying tokens
   /// as a unit and is responsible for the formatting of the them.
   virtual void updateNextToken(LineState &State) const {}
 
 protected:
-  BreakableToken(const FormatToken &Tok, unsigned IndentLevel,
-                 bool InPPDirective, encoding::Encoding Encoding,
-                 const FormatStyle &Style)
-      : Tok(Tok), IndentLevel(IndentLevel), InPPDirective(InPPDirective),
-        Encoding(Encoding), Style(Style) {}
+  BreakableToken(const FormatToken &Tok, bool InPPDirective,
+                 encoding::Encoding Encoding, const FormatStyle &Style)
+      : Tok(Tok), InPPDirective(InPPDirective), Encoding(Encoding),
+        Style(Style) {}
 
   const FormatToken &Tok;
-  const unsigned IndentLevel;
   const bool InPPDirective;
   const encoding::Encoding Encoding;
   const FormatStyle &Style;
 };
 
 /// \brief Base class for single line tokens that can be broken.
 ///
 /// \c getSplit() needs to be implemented by child classes.
 class BreakableSingleLineToken : public BreakableToken {
 public:
   unsigned getLineCount() const override;
   unsigned getLineLengthAfterSplit(unsigned LineIndex, unsigned TailOffset,
                                    StringRef::size_type Length) const override;
 
 protected:
-  BreakableSingleLineToken(const FormatToken &Tok, unsigned IndentLevel,
-                           unsigned StartColumn, StringRef Prefix,
-                           StringRef Postfix, bool InPPDirective,
-                           encoding::Encoding Encoding,
+  BreakableSingleLineToken(const FormatToken &Tok, unsigned StartColumn,
+                           StringRef Prefix, StringRef Postfix,
+                           bool InPPDirective, encoding::Encoding Encoding,
                            const FormatStyle &Style);
 
   // The column in which the token starts.
   unsigned StartColumn;
   // The prefix a line needs after a break in the token.
   StringRef Prefix;
   // The postfix a line needs before introducing a break.
   StringRef Postfix;
   // The token text excluding the prefix and postfix.
   StringRef Line;
 };
 
 class BreakableStringLiteral : public BreakableSingleLineToken {
 public:
   /// \brief Creates a breakable token for a single line string literal.
   ///
   /// \p StartColumn specifies the column in which the token will start
   /// after formatting.
-  BreakableStringLiteral(const FormatToken &Tok, unsigned IndentLevel,
-                         unsigned StartColumn, StringRef Prefix,
-                         StringRef Postfix, bool InPPDirective,
-                         encoding::Encoding Encoding, const FormatStyle &Style);
+  BreakableStringLiteral(const FormatToken &Tok, unsigned StartColumn,
+                         StringRef Prefix, StringRef Postfix,
+                         bool InPPDirective, encoding::Encoding Encoding,
+                         const FormatStyle &Style);
 
   Split getSplit(unsigned LineIndex, unsigned TailOffset,
                  unsigned ColumnLimit) const override;
   void insertBreak(unsigned LineIndex, unsigned TailOffset, Split Split,
                    WhitespaceManager &Whitespaces) override;
   void compressWhitespace(unsigned LineIndex, unsigned TailOffset, Split Split,
                           WhitespaceManager &Whitespaces) override {}
 };
 
 class BreakableComment : public BreakableToken {
 protected:
   /// \brief Creates a breakable token for a comment.
   ///
   /// \p StartColumn specifies the column in which the comment will start
   /// after formatting, while \p OriginalStartColumn specifies in which
   /// column the comment started before formatting.
   /// If the comment starts a line after formatting, set \p FirstInLine to true.
-  BreakableComment(const FormatToken &Token, unsigned IndentLevel,
-                   unsigned StartColumn, unsigned OriginalStartColumn,
-                   bool FirstInLine, bool InPPDirective,
-                   encoding::Encoding Encoding, const FormatStyle &Style);
+  BreakableComment(const FormatToken &Token, unsigned StartColumn,
+                   unsigned OriginalStartColumn, bool FirstInLine,
+                   bool InPPDirective, encoding::Encoding Encoding,
+                   const FormatStyle &Style);
 
 public:
   unsigned getLineCount() const override;
   Split getSplit(unsigned LineIndex, unsigned TailOffset,
                  unsigned ColumnLimit) const override;
   void compressWhitespace(unsigned LineIndex, unsigned TailOffset, Split Split,
                           WhitespaceManager &Whitespaces) override;
 
protected:
   // // comment 1 comment 2
   // and not:
   // // comment 1comment 2
   StringRef ReflowPrefix = " ";
 };
 
 class BreakableBlockComment : public BreakableComment {
 public:
-  BreakableBlockComment(const FormatToken &Token, unsigned IndentLevel,
-                        unsigned StartColumn, unsigned OriginalStartColumn,
-                        bool FirstInLine, bool InPPDirective,
-                        encoding::Encoding Encoding, const FormatStyle &Style);
+  BreakableBlockComment(const FormatToken &Token, unsigned StartColumn,
+                        unsigned OriginalStartColumn, bool FirstInLine,
+                        bool InPPDirective, encoding::Encoding Encoding,
+                        const FormatStyle &Style);
 
   unsigned getLineLengthAfterSplit(unsigned LineIndex,
                                    unsigned TailOffset,
                                    StringRef::size_type Length) const override;
   void insertBreak(unsigned LineIndex, unsigned TailOffset, Split Split,
                    WhitespaceManager &Whitespaces) override;
   Split getSplitBefore(unsigned LineIndex, unsigned PreviousEndColumn,
                        unsigned ColumnLimit) const override;
private:
   bool LastLineNeedsDecoration;
 
   // Either "* " if all lines begin with a "*", or empty.
   StringRef Decoration;
 };
 
 class BreakableLineCommentSection : public BreakableComment {
 public:
-  BreakableLineCommentSection(const FormatToken &Token, unsigned IndentLevel,
-                              unsigned StartColumn,
+  BreakableLineCommentSection(const FormatToken &Token, unsigned StartColumn,
                               unsigned OriginalStartColumn, bool FirstInLine,
                               bool InPPDirective, encoding::Encoding Encoding,
                               const FormatStyle &Style);
 
   unsigned getLineLengthAfterSplit(unsigned LineIndex,
                                    unsigned TailOffset,
                                    StringRef::size_type Length) const override;
   void insertBreak(unsigned LineIndex, unsigned TailOffset, Split Split,

lib/Format/BreakableToken.cpp

unsigned BreakableSingleLineToken::getLineLengthAfterSplit(
     StringRef::size_type Length) const {
   return StartColumn + Prefix.size() + Postfix.size() +
          encoding::columnWidthWithTabs(Line.substr(TailOffset, Length),
                                        StartColumn + Prefix.size(),
                                        Style.TabWidth, Encoding);
 }
 
 BreakableSingleLineToken::BreakableSingleLineToken(
-    const FormatToken &Tok, unsigned IndentLevel, unsigned StartColumn,
-    StringRef Prefix, StringRef Postfix, bool InPPDirective,
-    encoding::Encoding Encoding, const FormatStyle &Style)
-    : BreakableToken(Tok, IndentLevel, InPPDirective, Encoding, Style),
+    const FormatToken &Tok, unsigned StartColumn, StringRef Prefix,
+    StringRef Postfix, bool InPPDirective, encoding::Encoding Encoding,
+    const FormatStyle &Style)
+    : BreakableToken(Tok, InPPDirective, Encoding, Style),
       StartColumn(StartColumn), Prefix(Prefix), Postfix(Postfix) {
   assert(Tok.TokenText.endswith(Postfix));
   Line = Tok.TokenText.substr(
       Prefix.size(), Tok.TokenText.size() - Prefix.size() - Postfix.size());
 }
 
 BreakableStringLiteral::BreakableStringLiteral(
-    const FormatToken &Tok, unsigned IndentLevel, unsigned StartColumn,
-    StringRef Prefix, StringRef Postfix, bool InPPDirective,
-    encoding::Encoding Encoding, const FormatStyle &Style)
-    : BreakableSingleLineToken(Tok, IndentLevel, StartColumn, Prefix, Postfix,
-                               InPPDirective, Encoding, Style) {}
+    const FormatToken &Tok, unsigned StartColumn, StringRef Prefix,
+    StringRef Postfix, bool InPPDirective, encoding::Encoding Encoding,
+    const FormatStyle &Style)
+    : BreakableSingleLineToken(Tok, StartColumn, Prefix, Postfix, InPPDirective,
+                               Encoding, Style) {}
 
 BreakableToken::Split
 BreakableStringLiteral::getSplit(unsigned LineIndex, unsigned TailOffset,
                                  unsigned ColumnLimit) const {
   return getStringSplit(Line.substr(TailOffset),
                         StartColumn + Prefix.size() + Postfix.size(),
                         ColumnLimit, Style.TabWidth, Encoding);
 }
 
 void BreakableStringLiteral::insertBreak(unsigned LineIndex,
                                          unsigned TailOffset, Split Split,
                                          WhitespaceManager &Whitespaces) {
   unsigned LeadingSpaces = StartColumn;
   // The '@' of an ObjC string literal (@"Test") does not become part of the
   // string token.
   // FIXME: It might be a cleaner solution to merge the tokens as a
   // precomputation step.
   if (Prefix.startswith("@"))
     --LeadingSpaces;
   Whitespaces.replaceWhitespaceInToken(
       Tok, Prefix.size() + TailOffset + Split.first, Split.second, Postfix,
-      Prefix, InPPDirective, 1, IndentLevel, LeadingSpaces);
+      Prefix, InPPDirective, 1, LeadingSpaces);
 }
 
 BreakableComment::BreakableComment(const FormatToken &Token,
-                                   unsigned IndentLevel, unsigned StartColumn,
+                                   unsigned StartColumn,
                                    unsigned OriginalStartColumn,
                                    bool FirstInLine, bool InPPDirective,
                                    encoding::Encoding Encoding,
                                    const FormatStyle &Style)
-    : BreakableToken(Token, IndentLevel, InPPDirective, Encoding, Style),
+    : BreakableToken(Token, InPPDirective, Encoding, Style),
       StartColumn(StartColumn), OriginalStartColumn(OriginalStartColumn),
       FirstInLine(FirstInLine) {}
 
 unsigned BreakableComment::getLineCount() const { return Lines.size(); }
 
 BreakableToken::Split BreakableComment::getSplit(unsigned LineIndex,
                                                  unsigned TailOffset,
                                                  unsigned ColumnLimit) const {
void BreakableComment::compressWhitespace(unsigned LineIndex,
   // the start of the corresponding token, so compute the start of the Split
   // that needs to be compressed into a single space relative to the start of
   // its token.
   unsigned BreakOffsetInToken =
       Text.data() - tokenAt(LineIndex).TokenText.data() + Split.first;
   unsigned CharsToRemove = Split.second;
   Whitespaces.replaceWhitespaceInToken(
       tokenAt(LineIndex), BreakOffsetInToken, CharsToRemove, "", "",
-      /*InPPDirective=*/false,
-      /*Newlines=*/0, /*IndentLevel=*/0, /*Spaces=*/1);
+      /*InPPDirective=*/false, /*Newlines=*/0, /*Spaces=*/1);
 }
 
 BreakableToken::Split
 BreakableComment::getReflowSplit(StringRef Text, StringRef ReflowPrefix,
                                  unsigned PreviousEndColumn,
                                  unsigned ColumnLimit) const {
   unsigned ReflowStartColumn = PreviousEndColumn + ReflowPrefix.size();
   StringRef TrimmedText = Text.rtrim(Blanks);

 bool BreakableComment::mayReflow(unsigned LineIndex) const {
   return LineIndex > 0 && mayReflowContent(Content[LineIndex]) &&
          !Tok.Finalized && !switchesFormatting(tokenAt(LineIndex)) &&
          (!Tok.is(TT_LineComment) ||
           OriginalPrefix[LineIndex] == OriginalPrefix[LineIndex - 1]);
 }
 
 BreakableBlockComment::BreakableBlockComment(
-    const FormatToken &Token, unsigned IndentLevel, unsigned StartColumn,
+    const FormatToken &Token, unsigned StartColumn,
     unsigned OriginalStartColumn, bool FirstInLine, bool InPPDirective,
     encoding::Encoding Encoding, const FormatStyle &Style)
-    : BreakableComment(Token, IndentLevel, StartColumn, OriginalStartColumn,
-                       FirstInLine, InPPDirective, Encoding, Style) {
+    : BreakableComment(Token, StartColumn, OriginalStartColumn, FirstInLine,
+                       InPPDirective, Encoding, Style) {
   assert(Tok.is(TT_BlockComment) &&
          "block comment section must start with a block comment");
 
   StringRef TokenText(Tok.TokenText);
   assert(TokenText.startswith("/*") && TokenText.endswith("*/"));
   TokenText.substr(2, TokenText.size() - 4).split(Lines, "\n");
 
   int IndentDelta = StartColumn - OriginalStartColumn;
void BreakableBlockComment::insertBreak(unsigned LineIndex, unsigned TailOffset,
   // The split offset is from the beginning of the line. Convert it to an offset
   // from the beginning of the token text.
   unsigned BreakOffsetInToken =
       Text.data() - tokenAt(LineIndex).TokenText.data() + Split.first;
   unsigned CharsToRemove = Split.second;
   assert(LocalIndentAtLineBreak >= Prefix.size());
   Whitespaces.replaceWhitespaceInToken(
       tokenAt(LineIndex), BreakOffsetInToken, CharsToRemove, "", Prefix,
-      InPPDirective, /*Newlines=*/1, IndentLevel,
+      InPPDirective, /*Newlines=*/1,
       /*Spaces=*/LocalIndentAtLineBreak - Prefix.size());
 }
 
 BreakableToken::Split BreakableBlockComment::getSplitBefore(
     unsigned LineIndex,
     unsigned PreviousEndColumn,
     unsigned ColumnLimit) const {
   if (!mayReflow(LineIndex))
if (SplitBefore.first != StringRef::npos) {
     unsigned WhitespaceOffsetInToken = Content[LineIndex - 1].data() +
         Content[LineIndex - 1].size() -
         tokenAt(LineIndex).TokenText.data();
     unsigned WhitespaceLength = TrimmedContent.data() -
         tokenAt(LineIndex).TokenText.data() -
         WhitespaceOffsetInToken;
     Whitespaces.replaceWhitespaceInToken(
         tokenAt(LineIndex), WhitespaceOffsetInToken,
-        /*ReplaceChars=*/WhitespaceLength,
-        /*PreviousPostfix=*/"",
-        /*CurrentPrefix=*/ReflowPrefix, InPPDirective,
-        /*Newlines=*/0, IndentLevel, /*Spaces=*/0);
+        /*ReplaceChars=*/WhitespaceLength, /*PreviousPostfix=*/"",
+        /*CurrentPrefix=*/ReflowPrefix, InPPDirective, /*Newlines=*/0,
+        /*Spaces=*/0);
     // Check if we need to also insert a break at the whitespace range.
     // For this we first adapt the reflow split relative to the beginning of the
     // content.
     // Note that we don't need a penalty for this break, since it doesn't change
     // the total number of lines.
     Split BreakSplit = SplitBefore;
     BreakSplit.first += TrimmedContent.data() - Content[LineIndex].data();
     unsigned ReflownColumn =
void BreakableBlockComment::replaceWhitespaceBefore(
   // token text in the token.
   unsigned WhitespaceOffsetInToken = Content[LineIndex - 1].data() +
                                      Content[LineIndex - 1].size() -
                                      tokenAt(LineIndex).TokenText.data();
   unsigned WhitespaceLength = Content[LineIndex].data() -
                               tokenAt(LineIndex).TokenText.data() -
                               WhitespaceOffsetInToken;
   Whitespaces.replaceWhitespaceInToken(
-      tokenAt(LineIndex), WhitespaceOffsetInToken, WhitespaceLength, "",
-      Prefix, InPPDirective, /*Newlines=*/1, IndentLevel,
-      ContentColumn[LineIndex] - Prefix.size());
+      tokenAt(LineIndex), WhitespaceOffsetInToken, WhitespaceLength, "", Prefix,
+      InPPDirective, /*Newlines=*/1, ContentColumn[LineIndex] - Prefix.size());
 }
 
 unsigned
 BreakableBlockComment::getContentStartColumn(unsigned LineIndex,
                                              unsigned TailOffset) const {
   // If we break, we always break at the predefined indent.
   if (TailOffset != 0)
     return IndentAtLineBreak;
   return std::max(0, ContentColumn[LineIndex]);
 }
 
 BreakableLineCommentSection::BreakableLineCommentSection(
-    const FormatToken &Token, unsigned IndentLevel, unsigned StartColumn,
+    const FormatToken &Token, unsigned StartColumn,
     unsigned OriginalStartColumn, bool FirstInLine, bool InPPDirective,
     encoding::Encoding Encoding, const FormatStyle &Style)
-    : BreakableComment(Token, IndentLevel, StartColumn, OriginalStartColumn,
-                       FirstInLine, InPPDirective, Encoding, Style) {
+    : BreakableComment(Token, StartColumn, OriginalStartColumn, FirstInLine,
+                       InPPDirective, Encoding, Style) {
   assert(Tok.is(TT_LineComment) &&
          "line comment section must start with a line comment");
   FormatToken *LineTok = nullptr;
   for (const FormatToken *CurrentTok = &Tok;
        CurrentTok && CurrentTok->is(TT_LineComment);
        CurrentTok = CurrentTok->Next) {
     LastLineTok = LineTok;
     StringRef TokenText(CurrentTok->TokenText);
void BreakableLineCommentSection::insertBreak(unsigned LineIndex,
   // Compute the size of the new indent, including the size of the new prefix of
   // the newly broken line.
   unsigned IndentAtLineBreak = OriginalContentColumn[LineIndex] +
                                Prefix[LineIndex].size() -
                                OriginalPrefix[LineIndex].size();
   assert(IndentAtLineBreak >= Prefix[LineIndex].size());
   Whitespaces.replaceWhitespaceInToken(
       tokenAt(LineIndex), BreakOffsetInToken, CharsToRemove, "",
-      Prefix[LineIndex], InPPDirective, /*Newlines=*/1, IndentLevel,
+      Prefix[LineIndex], InPPDirective, /*Newlines=*/1,
       /*Spaces=*/IndentAtLineBreak - Prefix[LineIndex].size());
 }
 
 BreakableComment::Split BreakableLineCommentSection::getSplitBefore(
     unsigned LineIndex,
     unsigned PreviousEndColumn,
     unsigned ColumnLimit) const {
   if (!mayReflow(LineIndex)) return Split(StringRef::npos, 0);
void BreakableLineCommentSection::replaceWhitespaceBefore(
   // untouchable token.
   // This happens for instance here:
   // // line 1 \
   // // line 2
   if (LineIndex > 0 && Tokens[LineIndex] != Tokens[LineIndex - 1]) {
     if (SplitBefore.first != StringRef::npos) {
       // Reflow happens between tokens. Replace the whitespace between the
       // tokens by the empty string.
-      Whitespaces.replaceWhitespace(*Tokens[LineIndex],
-                                    /*Newlines=*/0,
-                                    /*IndentLevel=*/IndentLevel,
-                                    /*Spaces=*/0,
-                                    /*StartOfTokenColumn=*/StartColumn,
-                                    /*InPPDirective=*/false);
+      Whitespaces.replaceWhitespace(
+          *Tokens[LineIndex], /*Newlines=*/0, /*Spaces=*/0,
+          /*StartOfTokenColumn=*/StartColumn, /*InPPDirective=*/false);
       // Replace the indent and prefix of the token with the reflow prefix.
       unsigned WhitespaceLength =
           Content[LineIndex].data() - tokenAt(LineIndex).TokenText.data();
       Whitespaces.replaceWhitespaceInToken(*Tokens[LineIndex],
                                            /*Offset=*/0,
                                            /*ReplaceChars=*/WhitespaceLength,
                                            /*PreviousPostfix=*/"",
                                            /*CurrentPrefix=*/ReflowPrefix,
                                            /*InPPDirective=*/false,
                                            /*Newlines=*/0,
-                                           /*IndentLevel=*/IndentLevel,
                                            /*Spaces=*/0);
     } else {
       // This is the first line for the current token, but no reflow with the
       // previous token is necessary. However, we still may need to adjust the
       // start column.
       unsigned LineColumn =
           ContentColumn[LineIndex] -
           (Content[LineIndex].data() - Lines[LineIndex].data());
       if (tokenAt(LineIndex).OriginalColumn != LineColumn) {
         Whitespaces.replaceWhitespace(*Tokens[LineIndex],
                                       /*Newlines=*/1,
-                                      /*IndentLevel=*/IndentLevel,
                                       /*Spaces=*/LineColumn,
                                       /*StartOfTokenColumn=*/LineColumn,
                                       /*InPPDirective=*/false);
       } else {
         // The whitespace preceding the first line of this token does not need
         // to be touched.
         Whitespaces.addUntouchableToken(tokenAt(LineIndex),
                                         /*InPPDirective=*/false);
       }
     }
   } else if (OriginalPrefix[LineIndex] != Prefix[LineIndex]) {
     // This is not the first line of the token. Adjust the prefix if necessary.
 
     // Take care of the space possibly introduced after a decoration.
     assert(Prefix[LineIndex] == (OriginalPrefix[LineIndex] + " ").str() &&
            "Expecting a block comment decoration to differ from original by "
            "at most a space");
     Whitespaces.replaceWhitespaceInToken(
         tokenAt(LineIndex), OriginalPrefix[LineIndex].size(), 0, "", "",
-        /*InPPDirective=*/false,
-        /*Newlines=*/0, /*IndentLevel=*/0,
-        /*Spaces=*/1);
+        /*InPPDirective=*/false, /*Newlines=*/0, /*Spaces=*/1);
   }
   // Add a break after a reflow split has been introduced, if necessary.
   // Note that this break doesn't need to be penalized, since it doesn't change
   // the number of lines.
   if (SplitBefore.first != StringRef::npos &&
       SplitBefore.first + SplitBefore.second < Content[LineIndex].size()) {
     insertBreak(LineIndex, 0, SplitBefore, Whitespaces);
   }

lib/Format/ContinuationIndenter.h

private:
   const SourceManager &SourceMgr;
   WhitespaceManager &Whitespaces;
   encoding::Encoding Encoding;
   bool BinPackInconclusiveFunctions;
   llvm::Regex CommentPragmasRegex;
 };
 
 struct ParenState {
-  ParenState(unsigned Indent, unsigned IndentLevel, unsigned LastSpace,
-             bool AvoidBinPacking, bool NoLineBreak)
-      : Indent(Indent), IndentLevel(IndentLevel), LastSpace(LastSpace),
-        NestedBlockIndent(Indent), BreakBeforeClosingBrace(false),
-        AvoidBinPacking(AvoidBinPacking), BreakBeforeParameter(false),
-        NoLineBreak(NoLineBreak), NoLineBreakInOperand(false),
-        LastOperatorWrapped(true), ContainsLineBreak(false),
-        ContainsUnwrappedBuilder(false), AlignColons(true),
-        ObjCSelectorNameFound(false), HasMultipleNestedBlocks(false),
-        NestedBlockInlined(false) {}
+  ParenState(unsigned Indent, unsigned LastSpace, bool AvoidBinPacking,
+             bool NoLineBreak)
+      : Indent(Indent), LastSpace(LastSpace), NestedBlockIndent(Indent),
+        BreakBeforeClosingBrace(false), AvoidBinPacking(AvoidBinPacking),
+        BreakBeforeParameter(false), NoLineBreak(NoLineBreak),
+        NoLineBreakInOperand(false), LastOperatorWrapped(true),
+        ContainsLineBreak(false), ContainsUnwrappedBuilder(false),
+        AlignColons(true), ObjCSelectorNameFound(false),
+        HasMultipleNestedBlocks(false), NestedBlockInlined(false) {}
 
   /// \brief The position to which a specific parenthesis level needs to be
   /// indented.
   unsigned Indent;
 
-  /// \brief The number of indentation levels of the block.
-  unsigned IndentLevel;
-
   /// \brief The position of the last space on each level.
   ///
   /// Used e.g. to break like:
   /// functionCall(Parameter, otherCall(
   ///                             OtherParameter));
   unsigned LastSpace;
 
   /// \brief If a block relative to this parenthesis level gets wrapped, indent

lib/Format/ContinuationIndenter.cpp

 LineState ContinuationIndenter::getInitialState(unsigned FirstIndent,
                                                 const AnnotatedLine *Line,
                                                 bool DryRun) {
   LineState State;
   State.FirstIndent = FirstIndent;
   State.Column = FirstIndent;
   State.Line = Line;
   State.NextToken = Line->First;
-  State.Stack.push_back(ParenState(FirstIndent, Line->Level, FirstIndent,
+  State.Stack.push_back(ParenState(FirstIndent, FirstIndent,
                                    /*AvoidBinPacking=*/false,
                                    /*NoLineBreak=*/false));
   State.LineContainsContinuedForLoopSection = false;
   State.StartOfStringLiteral = 0;
   State.StartOfLineLevel = 0;
   State.LowestLevelOnLine = 0;
   State.IgnoreStackForComparison = false;
 
if (Current.is(tok::equal) &&
     }
     if (Previous.PartOfMultiVariableDeclStmt)
       State.Stack.back().LastSpace = State.Stack.back().VariablePos;
   }
 
   unsigned Spaces = Current.SpacesRequiredBefore + ExtraSpaces;
 
   if (!DryRun)
-    Whitespaces.replaceWhitespace(Current, /*Newlines=*/0, /*IndentLevel=*/0,
-                                  Spaces, State.Column + Spaces);
+    Whitespaces.replaceWhitespace(Current, /*Newlines=*/0, Spaces,
+                                  State.Column + Spaces);
 
   if (Current.is(TT_SelectorName) &&
       !State.Stack.back().ObjCSelectorNameFound) {
     unsigned MinIndent =
         std::max(State.FirstIndent + Style.ContinuationIndentWidth,
                  State.Stack.back().Indent);
     unsigned FirstColonPos = State.Column + Spaces + Current.ColumnWidth;
     if (Current.LongestObjCSelectorName == 0)
if (NextNonComment->is(tok::question) ||
       (PreviousNonComment && PreviousNonComment->is(tok::question)))
     State.Stack.back().BreakBeforeParameter = true;
   if (Current.is(TT_BinaryOperator) && Current.CanBreakBefore)
     State.Stack.back().BreakBeforeParameter = false;
 
   if (!DryRun) {
     unsigned Newlines = std::max(
         1u, std::min(Current.NewlinesBefore, Style.MaxEmptyLinesToKeep + 1));
-    Whitespaces.replaceWhitespace(Current, Newlines,
-                                  State.Stack.back().IndentLevel, State.Column,
-                                  State.Column, State.Line->InPPDirective);
+    Whitespaces.replaceWhitespace(Current, Newlines, State.Column, State.Column,
+                                  State.Line->InPPDirective);
   }
 
   if (!Current.isTrailingComment())
     State.Stack.back().LastSpace = State.Column;
   if (Current.is(tok::lessless))
     // If we are breaking before a "<<", we always want to indent relative to
     // RHS. This is necessary only for "<<", as we special-case it and don't
     // always indent relative to the RHS.
if (!Current.opensScope())
     return;
 
   if (Current.MatchingParen && Current.BlockKind == BK_Block) {
     moveStateToNewBlock(State);
     return;
   }
 
   unsigned NewIndent;
-  unsigned NewIndentLevel = State.Stack.back().IndentLevel;
   unsigned LastSpace = State.Stack.back().LastSpace;
   bool AvoidBinPacking;
   bool BreakBeforeParameter = false;
   unsigned NestedBlockIndent = std::max(State.Stack.back().StartOfFunctionCall,
                                         State.Stack.back().NestedBlockIndent);
   if (Current.isOneOf(tok::l_brace, TT_ArrayInitializerLSquare)) {
     if (Current.opensBlockOrBlockTypeList(Style)) {
       NewIndent = State.Stack.back().NestedBlockIndent + Style.IndentWidth;
       NewIndent = std::min(State.Column + 2, NewIndent);
-      ++NewIndentLevel;
     } else {
       NewIndent = State.Stack.back().LastSpace + Style.ContinuationIndentWidth;
     }
     const FormatToken *NextNoComment = Current.getNextNonComment();
     bool EndsInComma = Current.MatchingParen &&
                        Current.MatchingParen->Previous &&
                        Current.MatchingParen->Previous->is(tok::comma);
     AvoidBinPacking =
void ContinuationIndenter::moveStatePastScopeOpener(LineState &State,
   // array literals as these follow different indentation rules.
   bool NoLineBreak =
       Current.Children.empty() &&
       !Current.isOneOf(TT_DictLiteral, TT_ArrayInitializerLSquare) &&
       (State.Stack.back().NoLineBreak ||
        State.Stack.back().NoLineBreakInOperand ||
        (Current.is(TT_TemplateOpener) &&
         State.Stack.back().ContainsUnwrappedBuilder));
-  State.Stack.push_back(ParenState(NewIndent, NewIndentLevel, LastSpace,
-                                   AvoidBinPacking, NoLineBreak));
+  State.Stack.push_back(
+      ParenState(NewIndent, LastSpace, AvoidBinPacking, NoLineBreak));
   State.Stack.back().NestedBlockIndent = NestedBlockIndent;
   State.Stack.back().BreakBeforeParameter = BreakBeforeParameter;
   State.Stack.back().HasMultipleNestedBlocks = Current.BlockParameterCount > 1;
 }
 
 void ContinuationIndenter::moveStatePastScopeCloser(LineState &State) {
   const FormatToken &Current = *State.NextToken;
   if (!Current.closesScope())

 
 void ContinuationIndenter::moveStateToNewBlock(LineState &State) {
   unsigned NestedBlockIndent = State.Stack.back().NestedBlockIndent;
   // ObjC block sometimes follow special indentation rules.
   unsigned NewIndent =
       NestedBlockIndent + (State.NextToken->is(TT_ObjCBlockLBrace)
                                ? Style.ObjCBlockIndentWidth
                                : Style.IndentWidth);
-  State.Stack.push_back(ParenState(
-      NewIndent, /*NewIndentLevel=*/State.Stack.back().IndentLevel + 1,
-      State.Stack.back().LastSpace, /*AvoidBinPacking=*/true,
+  State.Stack.push_back(ParenState(NewIndent, State.Stack.back().LastSpace,
+                                   /*AvoidBinPacking=*/true,
                                    /*NoLineBreak=*/false));
   State.Stack.back().NestedBlockIndent = NestedBlockIndent;
   State.Stack.back().BreakBeforeParameter = true;
 }
 
 unsigned ContinuationIndenter::addMultilineToken(const FormatToken &Current,
                                                  LineState &State) {
   if (!Current.IsMultiline)
     return 0;
if (Text.startswith("\"") && Current.Previous &&
       Prefix = "@\"";
     }
     if ((Text.endswith(Postfix = "\"") &&
          (IsNSStringLiteral || Text.startswith(Prefix = "\"") ||
           Text.startswith(Prefix = "u\"") || Text.startswith(Prefix = "U\"") ||
           Text.startswith(Prefix = "u8\"") ||
           Text.startswith(Prefix = "L\""))) ||
         (Text.startswith(Prefix = "_T(\"") && Text.endswith(Postfix = "\")"))) {
-      Token.reset(new BreakableStringLiteral(
-          Current, State.Line->Level, StartColumn, Prefix, Postfix,
-          State.Line->InPPDirective, Encoding, Style));
+      Token.reset(new BreakableStringLiteral(Current, StartColumn, Prefix,
+                                             Postfix, State.Line->InPPDirective,
+                                             Encoding, Style));
     } else {
       return 0;
     }
   } else if (Current.is(TT_BlockComment)) {
     if (!Current.isTrailingComment() || !Style.ReflowComments ||
         CommentPragmasRegex.match(Current.TokenText.substr(2)) ||
         // If a comment token switches formatting, like
         // /* clang-format on */, we don't want to break it further,
         // but we may still want to adjust its indentation.
         switchesFormatting(Current))
       return addMultilineToken(Current, State);
     Token.reset(new BreakableBlockComment(
-        Current, State.Line->Level, StartColumn, Current.OriginalColumn,
-        !Current.Previous, State.Line->InPPDirective, Encoding, Style));
+        Current, StartColumn, Current.OriginalColumn, !Current.Previous,
+        State.Line->InPPDirective, Encoding, Style));
   } else if (Current.is(TT_LineComment) &&
              (Current.Previous == nullptr ||
               Current.Previous->isNot(TT_ImplicitStringLiteral))) {
     if (!Style.ReflowComments ||
         CommentPragmasRegex.match(Current.TokenText.substr(2)) ||
         switchesFormatting(Current))
       return 0;
     Token.reset(new BreakableLineCommentSection(
-        Current, State.Line->Level, StartColumn, Current.OriginalColumn,
-        !Current.Previous,
+        Current, StartColumn, Current.OriginalColumn, !Current.Previous,
         /*InPPDirective=*/false, Encoding, Style));
     // We don't insert backslashes when breaking line comments.
     ColumnLimit = Style.ColumnLimit;
   } else {
     return 0;
   }
   if (Current.UnbreakableTailLength >= ColumnLimit)
     return 0;

lib/Format/FormatToken.h

struct FormatToken {
   /// \brief The binding strength of a token. This is a combined value of
   /// operator precedence, parenthesis nesting, etc.
   unsigned BindingStrength = 0;
 
   /// \brief The nesting level of this token, i.e. the number of surrounding (),
   /// [], {} or <>.
   unsigned NestingLevel = 0;
 
+  /// \brief The indent level of this token. Copied from the surrounding line.
+  unsigned IndentLevel = 0;
+
   /// \brief Penalty for inserting a line break before this token.
   unsigned SplitPenalty = 0;
 
   /// \brief If this is the first ObjC selector name in an ObjC method
   /// definition or call, this contains the length of the longest name.
   ///
   /// This being set to 0 means that the selectors should not be colon-aligned,
   /// e.g. because several of them are block-type.

lib/Format/TokenAnnotator.cpp

void TokenAnnotator::calculateFormattingInformation(AnnotatedLine &Line) {
   for (SmallVectorImpl<AnnotatedLine *>::iterator I = Line.Children.begin(),
                                                   E = Line.Children.end();
        I != E; ++I) {
     calculateFormattingInformation(**I);
   }
 
   Line.First->TotalLength =
       Line.First->IsMultiline ? Style.ColumnLimit : Line.First->ColumnWidth;
-  if (!Line.First->Next)
-    return;
   FormatToken *Current = Line.First->Next;
   bool InFunctionDecl = Line.MightBeFunctionDecl;
   while (Current) {
     if (isFunctionDeclarationName(*Current, Line))
       Current->Type = TT_FunctionDeclarationName;
     if (Current->is(TT_LineComment)) {
       if (Current->Previous->BlockKind == BK_BracedInit &&
           Current->Previous->opensScope())
while (Current) {
     // FIXME: Move magic numbers to a better place.
     Current->SplitPenalty = 20 * Current->BindingStrength +
                             splitPenalty(Line, *Current, InFunctionDecl);
 
     Current = Current->Next;
   }
 
   calculateUnbreakableTailLengths(Line);
+  unsigned IndentLevel = Line.Level;
   for (Current = Line.First; Current != nullptr; Current = Current->Next) {
     if (Current->Role)
       Current->Role->precomputeFormattingInfos(Current);
+    if (Current->MatchingParen &&
+        Current->MatchingParen->opensBlockOrBlockTypeList(Style)) {
+      assert(IndentLevel > 0);
+      --IndentLevel;
+    }
+    Current->IndentLevel = IndentLevel;
+    if (Current->opensBlockOrBlockTypeList(Style))
+      ++IndentLevel;
   }
 
   DEBUG({ printDebugInfo(Line); });
 }
 
 void TokenAnnotator::calculateUnbreakableTailLengths(AnnotatedLine &Line) {
   unsigned UnbreakableTailLength = 0;
   FormatToken *Current = Line.Last;

lib/Format/UnwrappedLineFormatter.h

public:
   /// \brief Format the current block and return the penalty.
   unsigned format(const SmallVectorImpl<AnnotatedLine *> &Lines,
                   bool DryRun = false, int AdditionalIndent = 0,
                   bool FixBadIndentation = false);
 
 private:
   /// \brief Add a new line and the required indent before the first Token
   /// of the \c UnwrappedLine if there was no structural parsing error.
-  void formatFirstToken(FormatToken &RootToken,
-                        const AnnotatedLine *PreviousLine, unsigned IndentLevel,
-                        unsigned Indent, bool InPPDirective);
+  void formatFirstToken(const AnnotatedLine &Line,
+                        const AnnotatedLine *PreviousLine, unsigned Indent);
 
   /// \brief Returns the column limit for a line, taking into account whether we
   /// need an escaped newline due to a continued preprocessor directive.
   unsigned getColumnLimit(bool InPPDirective,
                           const AnnotatedLine *NextLine) const;
 
   // Cache to store the penalty of formatting a vector of AnnotatedLines
   // starting from a specific additional offset. Improves performance if there
   // are many nested blocks.
   std::map<std::pair<const SmallVectorImpl<AnnotatedLine *> *, unsigned>,
-           unsigned> PenaltyCache;
+           unsigned>
+      PenaltyCache;
 
   ContinuationIndenter *Indenter;
   WhitespaceManager *Whitespaces;
   const FormatStyle &Style;
   const AdditionalKeywords &Keywords;
   bool *IncompleteFormat;
 };
 } // end namespace format
 } // end namespace clang
 
 #endif // LLVM_CLANG_LIB_FORMAT_UNWRAPPEDLINEFORMATTER_H

lib/Format/UnwrappedLineFormatter.cpp

if (NewLine) {
           BlockFormatter->format(Previous.Children, DryRun, AdditionalIndent,
                                  /*FixBadIndentation=*/true);
       return true;
     }
 
     if (Previous.Children[0]->First->MustBreakBefore)
       return false;
 
-    // Cannot merge multiple statements into a single line.
-    if (Previous.Children.size() > 1)
-      return false;
-
     // Cannot merge into one line if this line ends on a comment.
     if (Previous.is(tok::comment))
       return false;
 
+    // Cannot merge multiple statements into a single line.
+    if (Previous.Children.size() > 1)
+      return false;
+
+    const AnnotatedLine *Child = Previous.Children[0];
     // We can't put the closing "}" on a line with a trailing comment.
-    if (Previous.Children[0]->Last->isTrailingComment())
+    if (Child->Last->isTrailingComment())
       return false;
 
     // If the child line exceeds the column limit, we wouldn't want to merge it.
     // We add +2 for the trailing " }".
     if (Style.ColumnLimit > 0 &&
-        Previous.Children[0]->Last->TotalLength + State.Column + 2 >
-            Style.ColumnLimit)
+        Child->Last->TotalLength + State.Column + 2 > Style.ColumnLimit)
       return false;
 
     if (!DryRun) {
       Whitespaces->replaceWhitespace(
-          *Previous.Children[0]->First,
-          /*Newlines=*/0, /*IndentLevel=*/0, /*Spaces=*/1,
+          *Child->First, /*Newlines=*/0, /*Spaces=*/1,
           /*StartOfTokenColumn=*/State.Column, State.Line->InPPDirective);
     }
-    Penalty += formatLine(*Previous.Children[0], State.Column + 1, DryRun);
+    Penalty += formatLine(*Child, State.Column + 1, DryRun);
 
-    State.Column += 1 + Previous.Children[0]->Last->TotalLength;
+    State.Column += 1 + Child->Last->TotalLength;
     return true;
   }
 
   ContinuationIndenter *Indenter;
 
 private:
   WhitespaceManager *Whitespaces;
   const FormatStyle &Style;
for (const AnnotatedLine *Line =
     bool ShouldFormat = TheLine.Affected || FixIndentation;
     // We cannot format this line; if the reason is that the line had a
     // parsing error, remember that.
     if (ShouldFormat && TheLine.Type == LT_Invalid && IncompleteFormat)
       *IncompleteFormat = true;
 
     if (ShouldFormat && TheLine.Type != LT_Invalid) {
       if (!DryRun)
-        formatFirstToken(*TheLine.First, PreviousLine, TheLine.Level, Indent,
-                         TheLine.InPPDirective);
+        formatFirstToken(TheLine, PreviousLine, Indent);
 
       NextLine = Joiner.getNextMergedLine(DryRun, IndentTracker);
       unsigned ColumnLimit = getColumnLimit(TheLine.InPPDirective, NextLine);
       bool FitsIntoOneLine =
           TheLine.Last->TotalLength + Indent <= ColumnLimit ||
           (TheLine.Type == LT_ImportStatement &&
            (Style.Language != FormatStyle::LK_JavaScript ||
             !Style.JavaScriptWrapImports));
if (ShouldFormat && TheLine.Type != LT_Invalid) {
       if (StartsNewLine)
         IndentTracker.adjustToUnmodifiedLine(TheLine);
       if (!DryRun) {
         bool ReformatLeadingWhitespace =
             StartsNewLine && ((PreviousLine && PreviousLine->Affected) ||
                               TheLine.LeadingEmptyLinesAffected);
         // Format the first token.
         if (ReformatLeadingWhitespace)
-          formatFirstToken(*TheLine.First, PreviousLine, TheLine.Level,
-                           TheLine.First->OriginalColumn,
-                           TheLine.InPPDirective);
+          formatFirstToken(TheLine, PreviousLine,
+                           TheLine.First->OriginalColumn);
         else
           Whitespaces->addUntouchableToken(*TheLine.First,
                                            TheLine.InPPDirective);
 
         // Notify the WhitespaceManager about the unchanged whitespace.
         for (FormatToken *Tok = TheLine.First->Next; Tok; Tok = Tok->Next)
           Whitespaces->addUntouchableToken(*Tok, TheLine.InPPDirective);
       }
       NextLine = Joiner.getNextMergedLine(DryRun, IndentTracker);
       RangeMinLevel = UINT_MAX;
     }
     if (!DryRun)
       markFinalized(TheLine.First);
     PreviousLine = &TheLine;
   }
   PenaltyCache[CacheKey] = Penalty;
   return Penalty;
 }
 
-void UnwrappedLineFormatter::formatFirstToken(FormatToken &RootToken,
+void UnwrappedLineFormatter::formatFirstToken(const AnnotatedLine &Line,
                                               const AnnotatedLine *PreviousLine,
-                                              unsigned IndentLevel,
-                                              unsigned Indent,
-                                              bool InPPDirective) {
+                                              unsigned Indent) {
+  FormatToken& RootToken = *Line.First;
   if (RootToken.is(tok::eof)) {
     unsigned Newlines = std::min(RootToken.NewlinesBefore, 1u);
-    Whitespaces->replaceWhitespace(RootToken, Newlines, /*IndentLevel=*/0,
-                                   /*Spaces=*/0, /*TargetColumn=*/0);
+    Whitespaces->replaceWhitespace(RootToken, Newlines, /*Spaces=*/0,
+                                   /*TargetColumn=*/0);
     return;
   }
   unsigned Newlines =
       std::min(RootToken.NewlinesBefore, Style.MaxEmptyLinesToKeep + 1);
   // Remove empty lines before "}" where applicable.
   if (RootToken.is(tok::r_brace) &&
       (!RootToken.Next ||
        (RootToken.Next->is(tok::semi) && !RootToken.Next->Next)))
if (PreviousLine && PreviousLine->Last->isOneOf(tok::semi, tok::r_brace) &&
       RootToken.isAccessSpecifier() && RootToken.NewlinesBefore == 1)
     ++Newlines;
 
   // Remove empty lines after access specifiers.
   if (PreviousLine && PreviousLine->First->isAccessSpecifier() &&
       (!PreviousLine->InPPDirective || !RootToken.HasUnescapedNewline))
     Newlines = std::min(1u, Newlines);
 
-  Whitespaces->replaceWhitespace(RootToken, Newlines, IndentLevel, Indent,
-                                 Indent, InPPDirective &&
-                                             !RootToken.HasUnescapedNewline);
+  Whitespaces->replaceWhitespace(RootToken, Newlines, Indent, Indent,
+                                 Line.InPPDirective &&
 }
 
 unsigned
 UnwrappedLineFormatter::getColumnLimit(bool InPPDirective,
                                        const AnnotatedLine *NextLine) const {
   // In preprocessor directives reserve two chars for trailing " \" if the
   // next line continues the preprocessor directive.
   bool ContinuesPPDirective =

lib/Format/WhitespaceManager.h

 class WhitespaceManager {
 public:
   WhitespaceManager(const SourceManager &SourceMgr, const FormatStyle &Style,
                     bool UseCRLF)
       : SourceMgr(SourceMgr), Style(Style), UseCRLF(UseCRLF) {}
 
   /// \brief Replaces the whitespace in front of \p Tok. Only call once for
   /// each \c AnnotatedToken.
-  void replaceWhitespace(FormatToken &Tok, unsigned Newlines,
-                         unsigned IndentLevel, unsigned Spaces,
+  void replaceWhitespace(FormatToken &Tok, unsigned Newlines, unsigned Spaces,
                          unsigned StartOfTokenColumn,
                          bool InPPDirective = false);
 
   /// \brief Adds information about an unchangeable token's whitespace.
   ///
   /// Needs to be called for every token for which \c replaceWhitespace
   /// was not called.
   void addUntouchableToken(const FormatToken &Tok, bool InPPDirective);
public:
   /// 0.
   ///
   /// When \p InPPDirective is true, escaped newlines are inserted. \p Spaces is
   /// used to align backslashes correctly.
   void replaceWhitespaceInToken(const FormatToken &Tok, unsigned Offset,
                                 unsigned ReplaceChars,
                                 StringRef PreviousPostfix,
                                 StringRef CurrentPrefix, bool InPPDirective,
-                                unsigned Newlines, unsigned IndentLevel,
-                                int Spaces);
+                                unsigned Newlines, int Spaces);
 
   /// \brief Returns all the \c Replacements created during formatting.
   const tooling::Replacements &generateReplacements();
 
   /// \brief Represents a change before a token, a break inside a token,
   /// or the layout of an unchanged token (or whitespace within).
   struct Change {
     /// \brief Functor to sort changes in original source order.
     class IsBeforeInFile {
     public:
       IsBeforeInFile(const SourceManager &SourceMgr) : SourceMgr(SourceMgr) {}
       bool operator()(const Change &C1, const Change &C2) const;
 
     private:
       const SourceManager &SourceMgr;
     };
 
-    Change() {}
-
     /// \brief Creates a \c Change.
     ///
     /// The generated \c Change will replace the characters at
     /// \p OriginalWhitespaceRange with a concatenation of
     /// \p PreviousLinePostfix, \p NewlinesBefore line breaks, \p Spaces spaces
     /// and \p CurrentLinePrefix.
     ///
     /// \p StartOfTokenColumn and \p InPPDirective will be used to lay out
     /// trailing comments and escaped newlines.
-    Change(bool CreateReplacement, SourceRange OriginalWhitespaceRange,
-           unsigned IndentLevel, int Spaces, unsigned StartOfTokenColumn,
-           unsigned NewlinesBefore, StringRef PreviousLinePostfix,
-           StringRef CurrentLinePrefix, tok::TokenKind Kind,
-           bool ContinuesPPDirective, bool IsStartOfDeclName,
-           bool IsInsideToken);
+    Change(const FormatToken &Tok, bool CreateReplacement,
+           SourceRange OriginalWhitespaceRange, int Spaces,
+           unsigned StartOfTokenColumn, unsigned NewlinesBefore,
+           StringRef PreviousLinePostfix, StringRef CurrentLinePrefix,
+           bool ContinuesPPDirective, bool IsInsideToken);
+
+    // The kind of the token whose whitespace this change replaces, or in which
+    // this change inserts whitespace.
+    // FIXME: Currently this is not set correctly for breaks inside comments, as
+    // the \c BreakableToken is still doing its own alignment.
+    const FormatToken *Tok;
 
     bool CreateReplacement;
     // Changes might be in the middle of a token, so we cannot just keep the
     // FormatToken around to query its information.
     SourceRange OriginalWhitespaceRange;
     unsigned StartOfTokenColumn;
     unsigned NewlinesBefore;
     std::string PreviousLinePostfix;
     std::string CurrentLinePrefix;
-    // The kind of the token whose whitespace this change replaces, or in which
-    // this change inserts whitespace.
-    // FIXME: Currently this is not set correctly for breaks inside comments, as
-    // the \c BreakableToken is still doing its own alignment.
-    tok::TokenKind Kind;
     bool ContinuesPPDirective;
-    bool IsStartOfDeclName;
-
-    // The number of nested blocks the token is in. This is used to add tabs
-    // only for the indentation, and not for alignment, when
-    // UseTab = US_ForIndentation.
-    unsigned IndentLevel;
 
     // The number of spaces in front of the token or broken part of the token.
     // This will be adapted when aligning tokens.
     // Can be negative to retain information about the initial relative offset
     // of the lines in a block comment. This is used when aligning trailing
     // comments. Uncompensated negative offset is truncated to 0.
     int Spaces;
 

lib/Format/WhitespaceManager.cpp

 
 bool WhitespaceManager::Change::IsBeforeInFile::
 operator()(const Change &C1, const Change &C2) const {
   return SourceMgr.isBeforeInTranslationUnit(
       C1.OriginalWhitespaceRange.getBegin(),
       C2.OriginalWhitespaceRange.getBegin());
 }
 
-WhitespaceManager::Change::Change(
-    bool CreateReplacement, SourceRange OriginalWhitespaceRange,
-    unsigned IndentLevel, int Spaces, unsigned StartOfTokenColumn,
-    unsigned NewlinesBefore, StringRef PreviousLinePostfix,
-    StringRef CurrentLinePrefix, tok::TokenKind Kind, bool ContinuesPPDirective,
-    bool IsStartOfDeclName, bool IsInsideToken)
-    : CreateReplacement(CreateReplacement),
+WhitespaceManager::Change::Change(const FormatToken &Tok,
+                                  bool CreateReplacement,
+                                  SourceRange OriginalWhitespaceRange,
+                                  int Spaces, unsigned StartOfTokenColumn,
+                                  unsigned NewlinesBefore,
+                                  StringRef PreviousLinePostfix,
+                                  StringRef CurrentLinePrefix,
+                                  bool ContinuesPPDirective, bool IsInsideToken)
+    : Tok(&Tok), CreateReplacement(CreateReplacement),
       OriginalWhitespaceRange(OriginalWhitespaceRange),
       StartOfTokenColumn(StartOfTokenColumn), NewlinesBefore(NewlinesBefore),
       PreviousLinePostfix(PreviousLinePostfix),
-      CurrentLinePrefix(CurrentLinePrefix), Kind(Kind),
-      ContinuesPPDirective(ContinuesPPDirective),
-      IsStartOfDeclName(IsStartOfDeclName), IndentLevel(IndentLevel),
-      Spaces(Spaces), IsInsideToken(IsInsideToken), IsTrailingComment(false),
-      TokenLength(0), PreviousEndOfTokenColumn(0), EscapedNewlineColumn(0),
+      CurrentLinePrefix(CurrentLinePrefix),
+      ContinuesPPDirective(ContinuesPPDirective), Spaces(Spaces),
+      IsInsideToken(IsInsideToken), IsTrailingComment(false), TokenLength(0),
+      PreviousEndOfTokenColumn(0), EscapedNewlineColumn(0),
       StartOfBlockComment(nullptr), IndentationOffset(0) {}
 
 void WhitespaceManager::replaceWhitespace(FormatToken &Tok, unsigned Newlines,
-                                          unsigned IndentLevel, unsigned Spaces,
+                                          unsigned Spaces,
                                           unsigned StartOfTokenColumn,
                                           bool InPPDirective) {
   if (Tok.Finalized)
     return;
   Tok.Decision = (Newlines > 0) ? FD_Break : FD_Continue;
-  Changes.push_back(
-      Change(/*CreateReplacement=*/true, Tok.WhitespaceRange, IndentLevel,
-             Spaces, StartOfTokenColumn, Newlines, "", "", Tok.Tok.getKind(),
-             InPPDirective && !Tok.IsFirst,
-             Tok.is(TT_StartOfName) || Tok.is(TT_FunctionDeclarationName),
+  Changes.push_back(Change(Tok, /*CreateReplacement=*/true,
+                           Tok.WhitespaceRange, Spaces, StartOfTokenColumn,
+                           Newlines, "", "", InPPDirective && !Tok.IsFirst,
                            /*IsInsideToken=*/false));
 }
 
 void WhitespaceManager::addUntouchableToken(const FormatToken &Tok,
                                             bool InPPDirective) {
   if (Tok.Finalized)
     return;
-  Changes.push_back(Change(
-      /*CreateReplacement=*/false, Tok.WhitespaceRange, /*IndentLevel=*/0,
-      /*Spaces=*/0, Tok.OriginalColumn, Tok.NewlinesBefore, "", "",
-      Tok.Tok.getKind(), InPPDirective && !Tok.IsFirst,
-      Tok.is(TT_StartOfName) || Tok.is(TT_FunctionDeclarationName),
+  Changes.push_back(Change(Tok, /*CreateReplacement=*/false,
+                           Tok.WhitespaceRange, /*Spaces=*/0,
+                           Tok.OriginalColumn, Tok.NewlinesBefore, "", "",
+                           InPPDirective && !Tok.IsFirst,
                            /*IsInsideToken=*/false));
 }
 
 void WhitespaceManager::replaceWhitespaceInToken(
     const FormatToken &Tok, unsigned Offset, unsigned ReplaceChars,
     StringRef PreviousPostfix, StringRef CurrentPrefix, bool InPPDirective,
-    unsigned Newlines, unsigned IndentLevel, int Spaces) {
+    unsigned Newlines, int Spaces) {
   if (Tok.Finalized)
     return;
   SourceLocation Start = Tok.getStartOfNonWhitespace().getLocWithOffset(Offset);
-  Changes.push_back(Change(
-      true, SourceRange(Start, Start.getLocWithOffset(ReplaceChars)),
-      IndentLevel, Spaces, std::max(0, Spaces), Newlines, PreviousPostfix,
-      CurrentPrefix, Tok.is(TT_LineComment) ? tok::comment : tok::unknown,
-      InPPDirective && !Tok.IsFirst,
-      Tok.is(TT_StartOfName) || Tok.is(TT_FunctionDeclarationName),
-      /*IsInsideToken=*/Newlines == 0));
+  Changes.push_back(
+      Change(Tok, /*CreateReplacement=*/true,
+             SourceRange(Start, Start.getLocWithOffset(ReplaceChars)), Spaces,
+             std::max(0, Spaces), Newlines, PreviousPostfix, CurrentPrefix,
+             InPPDirective && !Tok.IsFirst, /*IsInsideToken=*/true));
 }
 
 const tooling::Replacements &WhitespaceManager::generateReplacements() {
   if (Changes.empty())
     return Replaces;
 
   std::sort(Changes.begin(), Changes.end(), Change::IsBeforeInFile(SourceMgr));
   calculateLineBreakInformation();
if (Changes[i - 1].IsInsideToken)
           Changes[i - 1].TokenLength + Changes[i - 1].Spaces;
     else
       LastOutsideTokenChange = &Changes[i - 1];
 
     Changes[i].PreviousEndOfTokenColumn =
         Changes[i - 1].StartOfTokenColumn + Changes[i - 1].TokenLength;
 
     Changes[i - 1].IsTrailingComment =
-        (Changes[i].NewlinesBefore > 0 || Changes[i].Kind == tok::eof ||
-         (Changes[i].IsInsideToken && Changes[i].Kind == tok::comment)) &&
-        Changes[i - 1].Kind == tok::comment &&
+        (Changes[i].NewlinesBefore > 0 || Changes[i].Tok->is(tok::eof) ||
+         (Changes[i].IsInsideToken && Changes[i].Tok->is(tok::comment))) &&
+        Changes[i - 1].Tok->is(tok::comment) &&
         // FIXME: This is a dirty hack. The problem is that
         // BreakableLineCommentSection does comment reflow changes and here is
         // the aligning of trailing comments. Consider the case where we reflow
         // the second line up in this example:
         // 
         // // line 1
         // // line 2
         // 
Changes[i - 1].IsTrailingComment =
         // For a proper solution we need a mechanism to say to WhitespaceManager
         // that a particular change breaks the current sequence of trailing
         // comments.
         OriginalWhitespaceStart != PreviousOriginalWhitespaceEnd;
   }
   // FIXME: The last token is currently not always an eof token; in those
   // cases, setting TokenLength of the last token to 0 is wrong.
   Changes.back().TokenLength = 0;
-  Changes.back().IsTrailingComment = Changes.back().Kind == tok::comment;
+  Changes.back().IsTrailingComment = Changes.back().Tok->is(tok::comment);
 
   const WhitespaceManager::Change *LastBlockComment = nullptr;
   for (auto &Change : Changes) {
     // Reset the IsTrailingComment flag for changes inside of trailing comments
     // so they don't get realigned later.
     if (Change.IsInsideToken)
       Change.IsTrailingComment = false;
     Change.StartOfBlockComment = nullptr;
     Change.IndentationOffset = 0;
-    if (Change.Kind == tok::comment) {
+    if (Change.Tok->is(tok::comment)) {
+      if (Change.Tok->is(TT_LineComment) || !Change.IsInsideToken)
         LastBlockComment = &Change;
-    } else if (Change.Kind == tok::unknown) {
+      else {
         if ((Change.StartOfBlockComment = LastBlockComment))
           Change.IndentationOffset =
               Change.StartOfTokenColumn -
               Change.StartOfBlockComment->StartOfTokenColumn;
+      }
     } else {
       LastBlockComment = nullptr;
     }
   }
 }
 
 // Align a single sequence of tokens, see AlignTokens below.
 template <typename F>
if (Changes[i].NewlinesBefore != 0) {
       // If there is a blank line, or if the last line didn't contain any
       // matching token, the sequence ends here.
       if (Changes[i].NewlinesBefore > 1 || !FoundMatchOnLine)
         AlignCurrentSequence();
 
       FoundMatchOnLine = false;
     }
 
-    if (Changes[i].Kind == tok::comma) {
+    if (Changes[i].Tok->is(tok::comma)) {
       ++CommasBeforeMatch;
-    } else if (Changes[i].Kind == tok::r_brace ||
-               Changes[i].Kind == tok::r_paren ||
-               Changes[i].Kind == tok::r_square) {
+    } else if (Changes[i].Tok->isOneOf(tok::r_brace, tok::r_paren,
+                                       tok::r_square)) {
       --NestingLevel;
-    } else if (Changes[i].Kind == tok::l_brace ||
-               Changes[i].Kind == tok::l_paren ||
-               Changes[i].Kind == tok::l_square) {
+    } else if (Changes[i].Tok->isOneOf(tok::l_brace, tok::l_paren,
+                                       tok::l_square)) {
       // We want sequences to skip over child scopes if possible, but not the
       // other way around.
       NestingLevelOfLastMatch = std::min(NestingLevelOfLastMatch, NestingLevel);
       ++NestingLevel;
     }
 
     if (!Matches(Changes[i]))
       continue;
AlignTokens(Style,
                 // Do not align on equal signs that are first on a line.
                 if (C.NewlinesBefore > 0)
                   return false;
 
                 // Do not align on equal signs that are last on a line.
                 if (&C != &Changes.back() && (&C + 1)->NewlinesBefore > 0)
                   return false;
 
-                return C.Kind == tok::equal;
+                return C.Tok->is(tok::equal);
               },
               Changes);
 }
 
 void WhitespaceManager::alignConsecutiveDeclarations() {
   if (!Style.AlignConsecutiveDeclarations)
     return;
 
   // FIXME: Currently we don't handle properly the PointerAlignment: Right
   // The * and & are not aligned and are left dangling. Something has to be done
   // about it, but it raises the question of alignment of code like:
   //   const char* const* v1;
   //   float const* v2;
   //   SomeVeryLongType const& v3;
 
-  AlignTokens(Style, [](Change const &C) { return C.IsStartOfDeclName; },
+  AlignTokens(Style,
+              [](Change const &C) {
+                return C.Tok->isOneOf(TT_StartOfName,
+                                      TT_FunctionDeclarationName);
+              },
               Changes);
 }
 
 void WhitespaceManager::alignTrailingComments() {
   unsigned MinColumn = 0;
   unsigned MaxColumn = UINT_MAX;
   unsigned StartOfSequence = 0;
   bool BreakBeforeNext = false;
for (unsigned i = 0, e = Changes.size(); i != e; ++i) {
     if (!Changes[i].CreateReplacement)
       ChangeMaxColumn = ChangeMinColumn;
 
     if (i + 1 != e && Changes[i + 1].ContinuesPPDirective)
       ChangeMaxColumn -= 2;
     // If this comment follows an } in column 0, it probably documents the
     // closing of a namespace and we don't want to align it.
     bool FollowsRBraceInColumn0 = i > 0 && Changes[i].NewlinesBefore == 0 &&
-                                  Changes[i - 1].Kind == tok::r_brace &&
+                                  Changes[i - 1].Tok->is(tok::r_brace) &&
                                   Changes[i - 1].StartOfTokenColumn == 0;
     bool WasAlignedWithStartOfNextLine = false;
     if (Changes[i].NewlinesBefore == 1) { // A comment on its own line.
       unsigned CommentColumn = SourceMgr.getSpellingColumnNumber(
           Changes[i].OriginalWhitespaceRange.getEnd());
       for (unsigned j = i + 1; j != e; ++j) {
-        if (Changes[j].Kind == tok::comment ||
-            Changes[j].Kind == tok::unknown)
-          // Skip over comments and unknown tokens. "unknown tokens are used for
-          // the continuation of multiline comments.
+        if (Changes[j].Tok->is(tok::comment))
           continue;
 
         unsigned NextColumn = SourceMgr.getSpellingColumnNumber(
             Changes[j].OriginalWhitespaceRange.getEnd());
         // The start of the next token was previously aligned with the
         // start of this comment.
         WasAlignedWithStartOfNextLine =
             CommentColumn == NextColumn ||
for (unsigned i = 0, e = Changes.size(); i != e; ++i) {
     }
     if (C.CreateReplacement) {
       std::string ReplacementText = C.PreviousLinePostfix;
       if (C.ContinuesPPDirective)
         appendNewlineText(ReplacementText, C.NewlinesBefore,
                           C.PreviousEndOfTokenColumn, C.EscapedNewlineColumn);
       else
         appendNewlineText(ReplacementText, C.NewlinesBefore);
-      appendIndentText(ReplacementText, C.IndentLevel, std::max(0, C.Spaces),
+      appendIndentText(ReplacementText, C.Tok->IndentLevel,
+                       std::max(0, C.Spaces),
                        C.StartOfTokenColumn - std::max(0, C.Spaces));
       ReplacementText.append(C.CurrentLinePrefix);
       storeReplacement(C.OriginalWhitespaceRange, ReplacementText);
     }
   }
 }
 
 void WhitespaceManager::storeReplacement(SourceRange Range,