clang-format: Add reflow capability to line comment formatting

[faisalv]

Summary

Based on an email discussion with Manuel Klimek - (and thoughts from Daniel Jasper and Alexander Kornienko), I have hacked up a quickish patch to allow reflow of line comments (works for me reasonably well in Visual Studio) for clang-format.

The idea is to not just insert a break into a line comment if it exceeds the comment limit, but to consider its subsequent line-comments and reflow and reformat the block of line-commenst. That way you can add a word anywhere in your block of line-comments, clang-format them, and feel *reasonably*;) confident you are within your column limit without abrupt breaks.

The heuristic I have currently used to decide to merge a block of comments is as follows:
If in editor mode (which I kludge-ily/brittley detect - is there a cleaner way to determine this and exactly what text is selected?):

• for all line comments that are not separated by an additional new line, or an empty comment, or more than 3 forward slashes in sequence, they are joined and reflowed.

If in non-editor mode (which the patch actually has currently disabled - see where I have mistakenly commented out: //clang::format::isInEditorMode();), once we encounter a comment that is on a line that exceeds the column limit, only then do we start joining and then keep joining comment lines until we have enough lines to reflow the comment, and then stop (including following the rules for non-joining in editor mode).

I currently use the comment prefix of the line that is being broken (so just because the first line-comment in the block of comments is with three slashes, does not entail all of them being formatted with three, even if they were written with two) - not sure if that was the most desirable thing to do. Eitherway is ok with me.

I do not know how to run the unittests, since I've mostly worked on clang, and have not had to use the 'unittest' testing framework - so I could really use some help writing tests - Not only to show me how to write and *run* the unittests (or point me to the docs) on a windows environment but also to actually write some tests for this.

My visually confirmed tests are pasted here:
Before: http://pastebin.com/cmGYqBcc
After: http://pastebin.com/ZiRKjMB5

Any constructive feedback and or guidance will be greatly appreciated.

Thanks!

[alexfh]

Thanks for working on this!

Before I do a thorough review, I'd like you to address a few high-level issues and remove commented code (it shouldn't make it to the repository anyway, and it doesn't help understanding the intentions, FIXME comments work better).

It would be more convenient for reviewers if you generated diff using "diff -u10000" to include all context. Alternatively, you could use the Phabricator's arcanist command-line tool to send your patch for review. Both are fine, choose whatever works better for you.

Second, I would ask you to add your tests to unittest/Format/FormatTest.cpp and run the whole test suite. Unfortunately, I have no idea on how to run clang unittests in Windows. In Linux depending on the way the project is built, running all Clang tests can be done in one of the following ways:

• in CMake+make builds - "make check-clang"
• in configure+make builds it should also be "make check-clang"
• in CMake+ninja builds it can be done using "ninja check-all", iirc.

I'm not sure if the steps described in http://llvm.org/docs/GettingStartedVS.html are enough to run unit tests or not. The best way to know is to write a failing test and to try. If you don't find a way to run the unit tests on Windows, you can always install a copy of Ubuntu on a virtual machine and use it for testing.

Could you also test your patch on some real code? LLVM/Clang sources can be a good test case, just reduce the column limit (to 60 or 30, for example) to have the line breaking/merging code work. This way you can find many test cases you didn't think about.

For example, we need to avoid joining lines with code and various ascii-art, even when the column limit is exceeded and they end up being broken. I'd go with even more conservative approach: detect boundaries of what looks like paragraphs of text, and join lines only within those boundaries. I'm not sure if this fits into your current approach, and this is why I suggested to parse consecutive block comments as a single BreakableToken. This way one could also implement reflowing of block comments in a similar manner.

See a few more random comments inline.

lib/Format/ContinuationIndenter.cpp

947

Any reason to do use lambda here?

1066

We don't use braces around single-line "if" bodies.

1111

We don't use braces around single-line "if" bodies.

1121

Please remove the empty line.

lib/Format/Format.cpp

586

This name doesn't say what the function really does. You probably meant "character" not "token". Token is a lexical entity, e.g. the whole comment. And the implementation is suboptimal. If you want to check that the comment has a form of any number of slashes followed by any number of spaces, then you could write:

Text.rtrim().ltrim("/").empty();

instead.

What are the cases you want this function to detect?

1178

Any reason to do use lambda here?

tools/clang-format/ClangFormat.cpp

307

I'd rather add a separate style option controlling how/when we want to join comments (say, ReflowComments=Never|LongLines|Always). This way what "editor mode" means could be configured externally and explicitly.

330

Please remove trailing empty lines here and in other places.

[klimek]

This was open for a long time - is this still ongoing?

[faisalv]

sorry - been all sorts of busy with my job - might not be able to get
to this for a few months - if some one wishes to move the work forward
(or re-engineer from scratch) meanwhile -
please do so! =)

Faisal Vali

[klimek]

Ok, just removing myself as reviewer than, so it doesn't clog my dashboard...

lib/Format/BreakableToken.h

   /// after formatting.
   BreakableLineComment(const FormatToken &Token, unsigned IndentLevel,
                        unsigned StartColumn, bool InPPDirective,
-                       encoding::Encoding Encoding, const FormatStyle &Style);
+                       encoding::Encoding Encoding, const FormatStyle &Style,
+                       unsigned PreviousLineCommentOverflow);
 
   Split getSplit(unsigned LineIndex, unsigned TailOffset,
                  unsigned ColumnLimit) const override;

                          WhitespaceManager &Whitespaces) override;
   void replaceWhitespaceBefore(unsigned LineIndex,
                                WhitespaceManager &Whitespaces) override;
-
+  unsigned getLineLengthAfterSplit(unsigned LineIndex, unsigned TailOffset,
+                                   StringRef::size_type Length) const override;
+  void resetPreviousLineCommentOverflow() { PreviousLineCommentOverflow = 0; }
 private:
   // The prefix without an additional space if one was added.
   StringRef OriginalPrefix;
+  // When reflowing line comments we need to use the overflow from the 
+  // previous line comment.
+  unsigned PreviousLineCommentOverflow;
 };
 
 class BreakableBlockComment : public BreakableToken {

lib/Format/BreakableToken.cpp

 
 BreakableLineComment::BreakableLineComment(
     const FormatToken &Token, unsigned IndentLevel, unsigned StartColumn,
-    bool InPPDirective, encoding::Encoding Encoding, const FormatStyle &Style)
+    bool InPPDirective, encoding::Encoding Encoding, const FormatStyle &Style,
+    unsigned PreviousLineCommentOverflow)
     : BreakableSingleLineToken(Token, IndentLevel, StartColumn,
                                getLineCommentIndentPrefix(Token.TokenText), "",
-                               InPPDirective, Encoding, Style) {
+                               InPPDirective, Encoding, Style),
+      PreviousLineCommentOverflow(PreviousLineCommentOverflow) {
   OriginalPrefix = Prefix;
   if (Token.TokenText.size() > Prefix.size() &&
       isAlphanumeric(Token.TokenText[Prefix.size()])) {

   }
 }
 
+unsigned BreakableLineComment::getLineLengthAfterSplit(
+    unsigned LineIndex, unsigned Offset, StringRef::size_type Length) const {
+  if (PreviousLineCommentOverflow) {
+    // If we have a previous line comment overflow - add one char for separating 
+    // whitespace. Since we are merging the current line comment with the
+    // previous (and the previous tokens already include the prefix and postfix
+    // size, we can ignore those here).
+    return PreviousLineCommentOverflow + 1 +
+           encoding::columnWidthWithTabs(Line.substr(Offset, Length),
+                                         /*StartColumn + Prefix.size()*/ 0,
+                                         Style.TabWidth, Encoding);
+  } else
+    return BreakableSingleLineToken::getLineLengthAfterSplit(LineIndex, Offset, Length);
+}
+
+
 BreakableToken::Split
 BreakableLineComment::getSplit(unsigned LineIndex, unsigned TailOffset,
                                unsigned ColumnLimit) const {
-  return getCommentSplit(Line.substr(TailOffset), StartColumn + Prefix.size(),
-                         ColumnLimit, Style.TabWidth, Encoding);
+  if (PreviousLineCommentOverflow) {
+    assert(TailOffset == 0 && "When dealing with previous overflow, we should "
+                              "not be starting at an offset into the current "
+                              "text!");
+    if (ColumnLimit == PreviousLineCommentOverflow)
+      return BreakableToken::Split(0, 0);
+
+    auto split = getCommentSplit(Line, 0/*StartColumn + Prefix.size()*/,
+                           ColumnLimit - PreviousLineCommentOverflow,
+                           Style.TabWidth, Encoding);
+
+    // If we are exceeding the limit, start on a new line.
+    if (((split.first == StringRef::npos ? Line.size() : split.first) +
+         PreviousLineCommentOverflow) >= ColumnLimit)
+      return BreakableToken::Split(0, 0);
+    return split;
+  } else
+    return getCommentSplit(Line.substr(TailOffset), StartColumn + Prefix.size(),
+                           ColumnLimit, Style.TabWidth, Encoding);
 }
 
 void BreakableLineComment::insertBreak(unsigned LineIndex, unsigned TailOffset,

       Tok, OriginalPrefix.size() + TailOffset + Split.first, Split.second,
       Postfix, Prefix, InPPDirective, /*Newlines=*/1, IndentLevel, StartColumn);
 }
-
 void BreakableLineComment::replaceWhitespace(unsigned LineIndex,
                                              unsigned TailOffset, Split Split,
                                              WhitespaceManager &Whitespaces) {

 void
 BreakableLineComment::replaceWhitespaceBefore(unsigned LineIndex,
                                               WhitespaceManager &Whitespaces) {
-  if (OriginalPrefix != Prefix) {
-    Whitespaces.replaceWhitespaceInToken(Tok, OriginalPrefix.size(), 0, "", "",
-                                         /*InPPDirective=*/false,
-                                         /*Newlines=*/0, /*IndentLevel=*/0,
-                                         /*Spaces=*/1);
+  if (PreviousLineCommentOverflow) {
+    // Note: The difference between OriginalPrefix and Prefix is no more than a
+    // space.
+    Whitespaces.replaceBeginningWhitespaceForReflowingLineComment(
+        Tok, /*Spaces*/ OriginalPrefix != Prefix ? 1 : 0, StartColumn,
+        OriginalPrefix == Prefix ? OriginalPrefix.size() - 1
+                                 : OriginalPrefix.size());
+  } else {
+    if (OriginalPrefix != Prefix) {
+      Whitespaces.replaceWhitespaceInToken(Tok, OriginalPrefix.size(), 0, "",
+                                           "",
+                                           /*InPPDirective=*/false,
+                                           /*Newlines=*/0, /*IndentLevel=*/0,
+                                           /*Spaces=*/1);
+    }
   }
 }
 

 
 } // namespace format
 } // namespace clang
+
+

lib/Format/ContinuationIndenter.h

   /// \brief The number of used columns in the current line.
   unsigned Column;
 
+  /// \brief The start column to be used by all line-comments that are
+  /// undergoing reflow.
+  unsigned StartColumnOfReflowingLineComment;
   /// \brief The token that needs to be next formatted.
   FormatToken *NextToken;
 

lib/Format/ContinuationIndenter.cpp

                                                  unsigned ExtraSpaces) {
   FormatToken &Current = *State.NextToken;
   const FormatToken &Previous = *State.NextToken->Previous;
+  const bool IsReflowingLineComment =
+      Current.Type == TT_LineComment && Previous.Type == TT_LineComment;
   if (Current.is(tok::equal) &&
       (State.Line->First->is(tok::kw_for) || Current.NestingLevel == 0) &&
       State.Stack.back().VariablePos == 0) {

     if (Previous.PartOfMultiVariableDeclStmt)
       State.Stack.back().LastSpace = State.Stack.back().VariablePos;
   }
-
-  unsigned Spaces = Current.SpacesRequiredBefore + ExtraSpaces;
-
-  if (!DryRun)
+  // FIXME: Is this really what we want?
+  unsigned Spaces = IsReflowingLineComment
+                        ? ExtraSpaces
+                        : Current.SpacesRequiredBefore + ExtraSpaces;
+  assert((!IsReflowingLineComment || Spaces == 0) &&
+         "There should be no extra spaces added to the previous column when "
+         "reflowing comments!");
+  // No need to replace white space when reflowing a line-comment, since that is 
+  // deferred to when the merging is actually being performed and the decision
+  // to flow it on to the next line or keep it on the current is being made.
+  if (!DryRun && !IsReflowingLineComment) {
     Whitespaces.replaceWhitespace(Current, /*Newlines=*/0, /*IndentLevel=*/0,
                                   Spaces, State.Column + Spaces);
-
+  }
   if (Current.Type == TT_SelectorName &&
       !State.Stack.back().ObjCSelectorNameFound) {
     if (Current.LongestObjCSelectorName == 0)

 unsigned ContinuationIndenter::breakProtrudingToken(const FormatToken &Current,
                                                     LineState &State,
                                                     bool DryRun) {
+   const bool IsReflowingLineComments = [&] {
+    FormatToken *Previous = Current.Previous;
+    return Previous && Previous->Type == TT_LineComment &&
+            Current.Type == TT_LineComment;
+  }();
   // Don't break multi-line tokens other than block comments. Instead, just
   // update the state.
   if (Current.Type != TT_BlockComment && Current.IsMultiline)

 
   std::unique_ptr<BreakableToken> Token;
   unsigned StartColumn = State.Column - Current.ColumnWidth;
+  assert(!IsReflowingLineComments || StartColumn <= Style.ColumnLimit);
+  // Should we need it, cache the start column of the next reflow comment.
+  if (!IsReflowingLineComments && Current.Type == TT_LineComment)
+    State.StartColumnOfReflowingLineComment = StartColumn;
   unsigned ColumnLimit = getColumnLimit(State);
 
   if (Current.isStringLiteral()) {

               Current.Previous->Type != TT_ImplicitStringLiteral)) {
     if (CommentPragmasRegex.match(Current.TokenText.substr(2)))
       return 0;
-    Token.reset(new BreakableLineComment(Current, State.Line->Level,
-                                         StartColumn, /*InPPDirective=*/false,
-                                         Encoding, Style));
+    Token.reset(new BreakableLineComment(
+        Current, State.Line->Level,
+        IsReflowingLineComments ? State.StartColumnOfReflowingLineComment
+                                : StartColumn,
+        /*InPPDirective=*/false, Encoding, Style,
+        IsReflowingLineComments ? StartColumn : 0));
     // We don't insert backslashes when breaking line comments.
     ColumnLimit = Style.ColumnLimit;
   } else {

   }
   if (Current.UnbreakableTailLength >= ColumnLimit)
     return 0;
-
-  unsigned RemainingSpace = ColumnLimit - Current.UnbreakableTailLength;
+  // FIXME: Remaining Space might not be calculated correctly for reflowing line 
+  // comments.
+  const unsigned RemainingSpace = ColumnLimit - Current.UnbreakableTailLength;
   bool BreakInserted = false;
   unsigned Penalty = 0;
   unsigned RemainingTokenColumns = 0;
+  BreakableLineComment *CurLineCommentBeingReflown =
+        IsReflowingLineComments
+            ? static_cast<BreakableLineComment *>(Token.get())
+            : nullptr;
+    
   for (unsigned LineIndex = 0, EndIndex = Token->getLineCount();
        LineIndex != EndIndex; ++LineIndex) {
-    if (!DryRun)
+    if (IsReflowingLineComments) {
+      BreakableToken::Split Split = Token->getSplit(LineIndex, 0, ColumnLimit);
+      // Since we can not place any portion of the current line-comment's tokens
+      // into the previous comment's line, ignore the previous comment overflow
+      // and place the current comment on its own line, aligned with the
+      // previous coment.
+      if (Split.first == 0) {
+        CurLineCommentBeingReflown->resetPreviousLineCommentOverflow();
+        if (!DryRun) {
+          // FIXME: Get rid of this ugly const_cast hack!
+          Whitespaces.replaceWhitespace(
+            const_cast<FormatToken &>(Current), /*NewLines=*/1,
+            State.Stack.back().IndentLevel,
+            State.StartColumnOfReflowingLineComment,
+            State.StartColumnOfReflowingLineComment, State.Line->InPPDirective);
+        }
+      }
+    }
+    if (!DryRun) {
       Token->replaceWhitespaceBefore(LineIndex, Whitespaces);
+    }
+    // The Offset from the beginning of the current token's line that has been
+    // processed.
     unsigned TailOffset = 0;
     RemainingTokenColumns =
         Token->getLineLengthAfterSplit(LineIndex, TailOffset, StringRef::npos);

         break;
       }
       assert(Split.first != 0);
+      // Determine the columns used on the currently "inserted" line.
+      const unsigned ColumnsUsed =
+          Token->getLineLengthAfterSplit(LineIndex, TailOffset, Split.first);
+      // Overflow has to be less than column limit - so ok to reset - since it
+      // must have all been used in calculating the current split, so should 
+      // not be considered for remaining tokens.
+      if (CurLineCommentBeingReflown)
+        CurLineCommentBeingReflown->resetPreviousLineCommentOverflow();
       unsigned NewRemainingTokenColumns = Token->getLineLengthAfterSplit(
           LineIndex, TailOffset + Split.first + Split.second, StringRef::npos);
 

         break;
 
       assert(NewRemainingTokenColumns < RemainingTokenColumns);
-      if (!DryRun)
+      if (!DryRun) {
         Token->insertBreak(LineIndex, TailOffset, Split, Whitespaces);
+      }
       Penalty += Current.SplitPenalty;
-      unsigned ColumnsUsed =
-          Token->getLineLengthAfterSplit(LineIndex, TailOffset, Split.first);
       if (ColumnsUsed > ColumnLimit) {
         Penalty += Style.PenaltyExcessCharacter * (ColumnsUsed - ColumnLimit);
       }
       TailOffset += Split.first + Split.second;
       RemainingTokenColumns = NewRemainingTokenColumns;
       BreakInserted = true;
+      
     }
   }
 

 
     Penalty += Current.isStringLiteral() ? Style.PenaltyBreakString
                                          : Style.PenaltyBreakComment;
                                              
     State.Stack.back().LastSpace = StartColumn;
   }
   return Penalty;

lib/Format/Format.cpp

 namespace clang {
 namespace format {
 
+bool isInEditorMode();
+
 const std::error_category &getParseCategory() {
   static ParseErrorCategory C;
   return C;

   ContinuationIndenter *Indenter;
 };
 
+
+static unsigned getColumnWidthOfLine(const AnnotatedLine *Line) {
+  FormatToken *Last = Line->Last;
+  return Last->OriginalColumn + Last->ColumnWidth + 1;
+}
+
 class LineJoiner {
 public:
   LineJoiner(const FormatStyle &Style) : Style(Style) {}

   tryFitMultipleLinesInOne(unsigned Indent,
                            SmallVectorImpl<AnnotatedLine *>::const_iterator I,
                            SmallVectorImpl<AnnotatedLine *>::const_iterator E) {
-    // We can never merge stuff if there are trailing line comments.
+    
     const AnnotatedLine *TheLine = *I;
-    if (TheLine->Last->Type == TT_LineComment)
-      return 0;
+    // Determine whether we need to merge line comments for re-flow.
+    if (TheLine->Last->Type == TT_LineComment) {
+      auto HasOnlyWhitespaceTokens = [](StringRef Text) {
+        StringRef RTrimmedCommentText = Text.rtrim();
+        for (char c : RTrimmedCommentText)
+          if (c != '/')
+            return false;
+        return true;
+      };
+      FormatToken *CurCommentBeingMerged = TheLine->Last;
+      if (HasOnlyWhitespaceTokens(CurCommentBeingMerged->TokenText))
+        return 0;
+      const bool IsInEditorMode = true; //clang::format::isInEditorMode();
+      unsigned NumOfSequentialLineCommentsToJoin = 0;
+     
+      unsigned CurTotalWidthOfChars = 0;
+      if (!IsInEditorMode) {   
+        // If we are not in Editor mode, then only reflow if the current
+        // line-comment exceeds its column length.
+        if ((CurTotalWidthOfChars = getColumnWidthOfLine(TheLine)) <=
+            Style.ColumnLimit + 1)
+          return 0;
+      }
+      
+      for (auto *Next = I + 1; Next != E; ++Next) {
+        const AnnotatedLine *const NextLine = *Next;
+        const FormatToken *const FirstTok = NextLine->First;
+        const FormatToken *const LastTok = NextLine->Last;
+        // Only consider reflowing/merging those comment-containing-lines that 
+        // contain only a line-comment.
+        if (FirstTok->Type != TT_LineComment ||
+            LastTok->Type != TT_LineComment || FirstTok != LastTok)
+          break;
+        // Only consider those comments sequenced directly on the next line.
+        if (FirstTok->NewlinesBefore > 1)
+          break;
+        // Do not merge a comment that contains more than 3 forward slashes in
+        // sequence
+        if (FirstTok->TokenText.startswith("////"))
+          break;
+        // Do not merge a comment that has no non-whitespace tokens
+        if (HasOnlyWhitespaceTokens(FirstTok->TokenText))
+          break;
+          
+        if (IsInEditorMode)
+          ++NumOfSequentialLineCommentsToJoin;
+        else {
+          ++NumOfSequentialLineCommentsToJoin;
+          // FIXME: add one for space?
+          CurTotalWidthOfChars += (FirstTok->OriginalColumn + FirstTok->ColumnWidth + 1);
+          const unsigned NumDivisions = CurTotalWidthOfChars/Style.ColumnLimit; 
+          
+          // Do not keep joining if when we add to the subsequent line, we have enough
+          // columns to spare.
+          // FIXME: This might need some tweaking to be more accurate.
+          if (NumDivisions == NumOfSequentialLineCommentsToJoin)
+            break;
+          // FIXME: Or if we are just one above and our remainder is 0.
+          // if (NumDivisions == NumOfSequentialLineCommentsToJoin + 1 &&
+          //    !(CurTotalWidthOfChars % Style.ColumnLimit))
+          //  break;
+        }
+      }
+      return NumOfSequentialLineCommentsToJoin;
+    }
 
     if (Style.ColumnLimit > 0 && Indent > Style.ColumnLimit)
       return 0;

 
       // Merge multiple lines if possible.
       unsigned MergedLines = Joiner.tryFitMultipleLinesInOne(Indent, I, E);
+
       if (MergedLines > 0 && Style.ColumnLimit == 0) {
         // Disallow line merging if there is a break at the start of one of the
         // input lines.

       if (!Seen.insert(&Node->State).second)
         // State already examined with lower penalty.
         continue;
-
+      const bool IsReflowingLineComments = [&] {
+        const FormatToken *CurrentTok = Node->State.NextToken;
+        const FormatToken *PrevTok = CurrentTok->Previous;
+        return PrevTok && CurrentTok && PrevTok->Type == TT_LineComment &&
+               CurrentTok->Type == TT_LineComment;
+      }();
       FormatDecision LastFormat = Node->State.NextToken->Decision;
       if (LastFormat == FD_Unformatted || LastFormat == FD_Continue)
         addNextStateToQueue(Penalty, Node, /*NewLine=*/false, &Count, &Queue);
-      if (LastFormat == FD_Unformatted || LastFormat == FD_Break)
+      // Don't attempt to analyze a state for adding new lines when reflowing a
+      // C++ comment.
+      // FIXME: We can probably make this work if necessary - but currently
+      // it allows for joining lines it shouldn't.
+      if ((LastFormat == FD_Unformatted || LastFormat == FD_Break) &&
+          !IsReflowingLineComments)
         addNextStateToQueue(Penalty, Node, /*NewLine=*/true, &Count, &Queue);
     }
 

lib/Format/WhitespaceManager.h

                          unsigned StartOfTokenColumn,
                          bool InPPDirective = false);
 
+  /// \brief Replaces the whitespace in front of \p Tok while reflowing the
+  /// comment - erases the comment prefix./ Only call once for each 
+  /// \c AnnotatedToken.
+  void replaceBeginningWhitespaceForReflowingLineComment(
+      const FormatToken &Tok, unsigned Spaces, unsigned StartOfTokenColumn,
+      unsigned NumOfForwardSlashesInCommentMarker, bool InPPDirective = false);
+
   /// \brief Adds information about an unchangeable token's whitespace.
   ///
   /// Needs to be called for every token for which \c replaceWhitespace

lib/Format/WhitespaceManager.cpp

   Replaces.clear();
 }
 
+void WhitespaceManager::replaceBeginningWhitespaceForReflowingLineComment(
+    const FormatToken &Tok, unsigned Spaces, unsigned StartOfTokenColumn,
+    unsigned NumOfForwardSlashesInCommentMarker,
+    bool InPPDirective /*= false*/) {
+  if (Tok.Finalized)
+    return;
+  // Tok.Decision = FD_Continue;
+
+  SourceRange CommentBeginRange =
+      SourceRange(Tok.WhitespaceRange.getBegin(),
+                  Tok.WhitespaceRange.getEnd().getLocWithOffset(
+                      NumOfForwardSlashesInCommentMarker));
+  Changes.push_back(Change(true, CommentBeginRange, /*IndentLevel*/ 0, Spaces,
+                           StartOfTokenColumn, /*Newlines*/ 0, "", "",
+                           Tok.Tok.getKind(), InPPDirective && !Tok.IsFirst));
+}
 void WhitespaceManager::replaceWhitespace(FormatToken &Tok, unsigned Newlines,
                                           unsigned IndentLevel, unsigned Spaces,
                                           unsigned StartOfTokenColumn,

     if (i + 1 != End)
       Changes[i + 1].PreviousEndOfTokenColumn += Shift;
     Changes[i].StartOfTokenColumn += Shift;
+
   }
 }
 

tools/clang-format/ClangFormat.cpp

   }
   return false;
 }
+// FIXME: This is a terrible hack - there should be a better way to identify and
+// broadcast that we are in editor mode.
+static bool IsInEditorMode;
+bool isInEditorMode() { return IsInEditorMode; }
 
 }  // namespace format
 }  // namespace clang

   OS << clang::getClangToolFullVersion("clang-format") << '\n';
 }
 
+
 int main(int argc, const char **argv) {
   llvm::sys::PrintStackTraceOnErrorSignal();
 

     llvm::outs() << Config << "\n";
     return 0;
   }
+  // FIXME: Yuk! This is an unreliable hack to detect if we are in editor mode -
+  // there must be a way to have all editors identify this?
+  clang::format::IsInEditorMode =
+      Offsets.getNumOccurrences() && Lengths.getNumOccurrences();
 
   bool Error = false;
   switch (FileNames.size()) {

   }
   return Error ? 1 : 0;
 }
+