[clang-format] Break long string literals in C#, etc.

[sstwcw]

Summary

Now strings that are too long for one line in C#, Java, JavaScript, and
Verilog get broken into several lines. C# and JavaScript interpolated
strings are not broken.

A new subclass BreakableStringLiteralUsingOperators is used to handle
the logic for adding plus signs and commas. The updateAfterBroken
method was added because now parentheses or braces may be required after
the parentheses or commas are added. In order to decide whether the
added plus sign should be unindented in the BreakableToken object, the
logic for it is taken out into a separate function
shouldUnindentNextOperator.

The logic for finding the continuation indentation when the option
AlignAfterOpenBracket is set to DontAlign is not implemented yet. So in
that case the new line may have the wrong indentation, and the parts may
have the wrong length if the string needs to be broken more than once
because finding where to break the string depends on where the string
starts.

The preambles for the C# and Java unit tests are changed to the newer
style in order to allow the 3-argument verifyFormat macro. Some cases
are changed from verifyFormat to verifyImcompleteFormat because those
use incomplete code and the new verifyFormat function checks that the
code is complete.

[HazardyKnusperkeks]

I'd really prefer you put what you need to modify mutable instead of removing the const from everything else. But that's just my opinion.

clang/include/clang/Format/Format.h

2012

It definitely works in C++, and I'd suspect C# and Java too.

clang/lib/Format/BreakableToken.h

313

[sstwcw]

• Address comments

[sstwcw]

In D154093#4466246, @HazardyKnusperkeks wrote:

I'd really prefer you put what you need to modify mutable instead of removing the const from everything else. But that's just my opinion.

Why should the insertBreak method be const?

[sstwcw]

clang/include/clang/Format/Format.h

2021–2033

It looks like they made a mistake in this example. The second line gets indented by a continuation indentation both before and after this patch.

[HazardyKnusperkeks]

In D154093#4466597, @sstwcw wrote:

In D154093#4466246, @HazardyKnusperkeks wrote:

I'd really prefer you put what you need to modify mutable instead of removing the const from everything else. But that's just my opinion.

Why should the insertBreak method be const?

Because it shouldn't and up until now doesn't modify the BreakableToken the modifying changes happen in the WhitespaceManager.

clang/include/clang/Format/Format.h

2048

I'd drop that, or replace it with a nicer sentence which only speaks of "all other languages", because right now you are missing at least protobuf and json.

[sstwcw]

If it is expected that the method does not modify the field, and the expectation is not fulfilled, I prefer to make it clear rather than to pretend that the method works as expected. Can you help me come up with an alternative? Braces need to be added around the string literal when it gets broken into parts. Only one pair of braces need to be added no matter how many times the string gets broken. My solution was to use a mutable field to mark whether braces have already been added. Do you have another way?

[sstwcw]

• Drop the line in the doc

[sstwcw]

• Add back the const qualifiers for methods

[HazardyKnusperkeks]

clang/lib/Format/ContinuationIndenter.cpp

2294

This const can be readded, or not?

clang/lib/Format/WhitespaceManager.cpp

1522

Can you elaborate where the 2 replacements come from, and why we can't make sure there is no empty range?

[sstwcw]

• Add back the const qualifier

[sstwcw]

clang/lib/Format/ContinuationIndenter.cpp

2294

I forgot to add it.

[sstwcw]

Are there any more problems?

[HazardyKnusperkeks]

Apart from that I'm okay.

clang/lib/Format/WhitespaceManager.cpp

1522

This is open (at least for me).

[sstwcw]

• Add comment explaining the replacements

[HazardyKnusperkeks]

For the replacement part, I refer to @owenpan and/or @MyDeveloperDay hoping they have more insight.
The rest look good to me.

[sstwcw]

@owenpan What do you think about this revision especially the replacement part?

[owenpan]

clang/lib/Format/ContinuationIndenter.cpp

2192–2193

Can we support using + first?

[owenpan]

In D154093#4542339, @sstwcw wrote:

@owenpan What do you think about this revision especially the replacement part?

See D154093#4544495. Then we can extend it by using , instead of + for Verilog (plus inserting a pair of braces).

[sstwcw]

Add support for breaking strings with + in C# and others

[sstwcw]

In D154093#4544496, @owenpan wrote:

In D154093#4542339, @sstwcw wrote:

@owenpan What do you think about this revision especially the replacement part?

See D154093#4544495. Then we can extend it by using , instead of + for Verilog (plus inserting a pair of braces).

I added support for that. See the updated description. It turned out
that because inserting plus signs may involve inserting parentheses, all
the replacement stuff are still needed. So I didn't split the Verilog
stuff into a separate patch.

[sstwcw]

@MyDeveloperDay Can you take a look at the last test in FormatTestCSharp.Attributes? I tried to enable breaking string literals by default. The long string got broken.

[MyDeveloperDay]

clang/unittests/Format/FormatTestCSharp.cpp

221

not sure what changing here and why?

clang/unittests/Format/FormatTestJS.cpp

1575

out of interest what happens with <backtick>xxxxxx ${variable} xxxxxx<backtick> type strings? can we add a test to show we won't break in the middle of a ${vas} even though the column limit might be reached?

[sstwcw]

• Add tests for code in JavaScript template strings

clang/unittests/Format/FormatTestCSharp.cpp

221

The brace is not paired. The old verifyFormat function didn't care about it. In FormatTestBase.h there is the function verifyFormat which requires the code to be complete and verifyIncompleteFormat which doesn't.

clang/unittests/Format/FormatTestJS.cpp

1575

There can be a break inside the braces. I added a test.

[sstwcw]

• Add tests for code in JavaScript template strings

[MyDeveloperDay]

I'm ok with this if @owenpan is

[owenpan]

@sstwcw this patch caused a crash in FormatTestJS.StringLiteralConcatenation:

Assertion failed: ((!RequiresNullTerminator || BufEnd[0] == 0) && "Buffer is not null terminated!"), function init, file MemoryBuffer.cpp, line 54.

Can you revert it if you can reproduce the crash?

[DavidSpickett]

This was causing failures on a large number of Linaro's bots. Judging by how widespread it was, reproducing could be as simple as adding -DLLVM_ENABLE_ASSERTIONS=True to your CMake config.

[sstwcw]

In the JavaScript tests that I added, it was wrong to use SmallString. Would you prefer me changing it to string or expanding the 6 test cases so we don't need a variable for the string?

Shortly after I committed this patch, the server builds caught the problem in the JavaScript tests. When the program reads a file, a null byte is added at the end. The program relies on this property because it means it doesn't have to do a bound check on the source string, including in some of the code I wrote though this time it was another piece of code that caught the problem. Also the null byte is not copied when a SmallString is initialized from a const char*. A null byte will be added if the c_str method is used at least once. But it was not the case here. So my test ended up passing a string which does not have a null byte at the end. I didn't catch the problem when I ran the tests myself because ironically I had the address sanitizer enabled. The address sanitizer didn't catch the problem because the byte past the end of the string in the SmallString is the struct padding which is still inside the object as far as the address sanitizer is concerned. However the address sanitizer likes to add padding between variables. So now the uninitialized part of the struct padding is the padding null bytes added by the address sanitizer instead of the variable that used to be there. So the assertion that the byte is null now holds.

[owenpan]

In D154093#4615948, @sstwcw wrote:

In the JavaScript tests that I added, it was wrong to use SmallString. Would you prefer me changing it to string or expanding the 6 test cases so we don't need a variable for the string?

You mean std::string? That would be fine by me.

[sstwcw]

Stop using SmallString in the tests.

[eaeltsin]

This introduces an invalid TS transformation, from

type x = 'ab';

to

type x = ('a'+'b');

Which doesn't compile - https://www.typescriptlang.org/play?#code/C4TwDgpgBAHlC8UDkBDARkg3AKG6SUICUAFKklANTIYCUOQA

Should we revert?

[alexfh]

This patch makes clang-format produce invalid JS/TS code

In D154093#4644512, @eaeltsin wrote:

This introduces an invalid TS transformation, from

type x = 'ab';

to

type x = ('a'+'b');

Which doesn't compile - https://www.typescriptlang.org/play?#code/C4TwDgpgBAHlC8UDkBDARkg3AKG6SUICUAFKklANTIYCUOQA

Should we revert?

Similarly, strings in dictionary literals can't be broken: var w = {'a' + 'b': 0}; (https://gcc.godbolt.org/z/6renEv5sa). Please fix or revert.

[sstwcw]

In D154093#4644727, @alexfh wrote:

This patch makes clang-format produce invalid JS/TS code

In D154093#4644512, @eaeltsin wrote:

This introduces an invalid TS transformation, from

type x = 'ab';

to

type x = ('a'+'b');

Which doesn't compile - https://www.typescriptlang.org/play?#code/C4TwDgpgBAHlC8UDkBDARkg3AKG6SUICUAFKklANTIYCUOQA

Should we revert?

Similarly, strings in dictionary literals can't be broken: var w = {'a' + 'b': 0}; (https://gcc.godbolt.org/z/6renEv5sa). Please fix or revert.

Here is the fix. https://github.com/llvm/llvm-project/pull/66168

[eaeltsin]

This still misses contexts when string literal is required, for example https://github.com/search?q=repo%3Agoogle%2Fclosure-compiler%20%22%20must%20be%20a%20string%20literal%22&type=code

I wonder, if splitting the literal with + is a good option at all.

[eaeltsin]

This might have another issue with Verilog -

<                 import "AAA-BBB" foo bar baz
---
>                 import {"AAA-",
>                         "BBB"} foo bar baz

I wonder if Verilog allows breaking strings in import?

[aeubanks]

In D154093#4648931, @eaeltsin wrote:

This might have another issue with Verilog -

<                 import "AAA-BBB" foo bar baz
---
>                 import {"AAA-",
>                         "BBB"} foo bar baz

I wonder if Verilog allows breaking strings in import?

From the spec:
dpi_spec_string ::= "DPI-C" | "DPI"
Seems like it can't.

[aeubanks]

In D154093#4648986, @aeubanks wrote:

In D154093#4648931, @eaeltsin wrote:

This might have another issue with Verilog -

<                 import "AAA-BBB" foo bar baz
---
>                 import {"AAA-",
>                         "BBB"} foo bar baz

I wonder if Verilog allows breaking strings in import?

From the spec:
dpi_spec_string ::= "DPI-C" | "DPI"
Seems like it can't.

Sent out https://github.com/llvm/llvm-project/pull/66951

clang/docs/ClangFormatStyleOptions.rst

 
 
 
 .. _BreakStringLiterals:
 
 **BreakStringLiterals** (``Boolean``) :versionbadge:`clang-format 3.9` :ref:`¶ <BreakStringLiterals>`
   Allow breaking string literals when formatting.
 
+  In C, C++, and Objective-C:
+
   .. code-block:: c++
 
      true:
      const char* x = "veryVeryVeryVeryVeryVe"
                      "ryVeryVeryVeryVeryVery"
                      "VeryLongString";
 
      false:
      const char* x =
          "veryVeryVeryVeryVeryVeryVeryVeryVeryVeryVeryVeryLongString";
 
+  In C#, Java, and JavaScript:
+
+  .. code-block:: c++
+
+     true:
+     var x = "veryVeryVeryVeryVeryVe" +
+             "ryVeryVeryVeryVeryVery" +
+             "VeryLongString";
+
+     false:
+     var x =
+         "veryVeryVeryVeryVeryVeryVeryVeryVeryVeryVeryVeryLongString";
+  C# and JavaScript interpolated strings are not broken.
+
+  In Verilog:
+
+  .. code-block:: c++
+
+     true:
+     string x = {"veryVeryVeryVeryVeryVe",
+                 "ryVeryVeryVeryVeryVery",
+                 "VeryLongString"};
+
+     false:
+     string x =
+         "veryVeryVeryVeryVeryVeryVeryVeryVeryVeryVeryVeryLongString";
+
 .. _ColumnLimit:
 
 **ColumnLimit** (``Unsigned``) :versionbadge:`clang-format 3.7` :ref:`¶ <ColumnLimit>`
   The column limit.
 
   A column limit of ``0`` means that there is no column limit. In this case,
   clang-format will respect the input's line breaking decisions within
   statements unless they contradict other rules.

clang/include/clang/Format/Format.h

   ///    @Partial                       vs.     @Partial @Mock DataLoad loader;
   ///    @Mock
   ///    DataLoad loader;
   /// \endcode
   /// \version 3.8
   bool BreakAfterJavaFieldAnnotations;
 
   /// Allow breaking string literals when formatting.
+  ///
+  /// In C, C++, and Objective-C:
   /// \code
   ///    true:
   ///    const char* x = "veryVeryVeryVeryVeryVe"
   ///                    "ryVeryVeryVeryVeryVery"
   ///                    "VeryLongString";
   ///
   ///    false:
   ///    const char* x =
-  ///      "veryVeryVeryVeryVeryVeryVeryVeryVeryVeryVeryVeryLongString";
+  ///        "veryVeryVeryVeryVeryVeryVeryVeryVeryVeryVeryVeryLongString";
   /// \endcode
+  ///
+  /// In C#, Java, and JavaScript:
+  /// \code
+  ///    true:
+  ///    var x = "veryVeryVeryVeryVeryVe" +
+  ///            "ryVeryVeryVeryVeryVery" +
+  ///            "VeryLongString";
+  ///
+  ///    false:
+  ///    var x =
+  ///        "veryVeryVeryVeryVeryVeryVeryVeryVeryVeryVeryVeryLongString";
+  /// \endcode
+  /// C# and JavaScript interpolated strings are not broken.
+  ///
+  /// In Verilog:
+  /// \code
+  ///    true:
+  ///    string x = {"veryVeryVeryVeryVeryVe",
+  ///                "ryVeryVeryVeryVeryVery",
+  ///                "VeryLongString"};
+  ///
+  ///    false:
+  ///    string x =
+  ///        "veryVeryVeryVeryVeryVeryVeryVeryVeryVeryVeryVeryLongString";
+  /// \endcode
+  ///
   /// \version 3.9
   bool BreakStringLiterals;
 
   /// The column limit.
   ///
   /// A column limit of ``0`` means that there is no column limit. In this case,
   /// clang-format will respect the input's line breaking decisions within
   /// statements unless they contradict other rules.

clang/lib/Format/BreakableToken.h

insertBreak(getLineCount() - 1, TailOffset, SplitAfterLastLine,
                 /*ContentIndent=*/0, Whitespaces);
   }
 
   /// Updates the next token of \p State to the next token after this
   /// one. This can be used when this token manages a set of underlying tokens
   /// as a unit and is responsible for the formatting of the them.
   virtual void updateNextToken(LineState &State) const {}
 
+  /// Adds replacements that are needed when the token is broken. Such as
+  /// wrapping a JavaScript string in parentheses after it gets broken with plus
+  /// signs.
+  virtual void updateAfterBroken(WhitespaceManager &Whitespaces) const {}
+
 protected:
   BreakableToken(const FormatToken &Tok, bool InPPDirective,
                  encoding::Encoding Encoding, const FormatStyle &Style)
       : Tok(Tok), InPPDirective(InPPDirective), Encoding(Encoding),
         Style(Style) {}
 
   const FormatToken &Tok;
   const bool InPPDirective;
protected:
   StringRef Postfix;
   // The token text excluding the prefix and postfix.
   StringRef Line;
   // Length of the sequence of tokens after this string literal that cannot
   // contain line breaks.
   unsigned UnbreakableTailLength;
 };
 
+class BreakableStringLiteralUsingOperators : public BreakableStringLiteral {
+public:
+  enum QuoteStyleType {
+    DoubleQuotes,   // The string is quoted with double quotes.
+    SingleQuotes,   // The JavaScript string is quoted with single quotes.
+    AtDoubleQuotes, // The C# verbatim string is quoted with the at sign and
+                    // double quotes.
+  };
+  /// Creates a breakable token for a single line string literal for C#, Java,
+  /// JavaScript, or Verilog.
+  ///
+  /// \p StartColumn specifies the column in which the token will start
+  /// after formatting.
+  BreakableStringLiteralUsingOperators(
+      const FormatToken &Tok, QuoteStyleType QuoteStyle, bool UnindentPlus,
+      unsigned StartColumn, unsigned UnbreakableTailLength, bool InPPDirective,
+      encoding::Encoding Encoding, const FormatStyle &Style);
+  unsigned getRemainingLength(unsigned LineIndex, unsigned Offset,
+                              unsigned StartColumn) const override;
+  unsigned getContentStartColumn(unsigned LineIndex, bool Break) const override;
+  void insertBreak(unsigned LineIndex, unsigned TailOffset, Split Split,
+                   unsigned ContentIndent,
+                   WhitespaceManager &Whitespaces) const override;
+  void updateAfterBroken(WhitespaceManager &Whitespaces) const override;
+
+protected:
+  // Whether braces or parentheses should be inserted around the string to form
+  // a concatenation.
+  bool BracesNeeded;
+  QuoteStyleType QuoteStyle;
+  // The braces or parentheses along with the first character which they
+  // replace, either a quote or at sign.
+  StringRef LeftBraceQuote;
+  StringRef RightBraceQuote;
+  // Width added to the left due to the added. Does not apply to the first line.
+  int ContinuationIndent;
+};
+
 class BreakableComment : public BreakableToken {
 protected:
   /// Creates a breakable token for a comment.
   ///
   /// \p StartColumn specifies the column in which the comment will start after
   /// formatting.
   BreakableComment(const FormatToken &Token, unsigned StartColumn,
                    bool InPPDirective, encoding::Encoding Encoding,

clang/lib/Format/BreakableToken.cpp

void BreakableStringLiteral::insertBreak(unsigned LineIndex,
                                          unsigned TailOffset, Split Split,
                                          unsigned ContentIndent,
                                          WhitespaceManager &Whitespaces) const {
   Whitespaces.replaceWhitespaceInToken(
       Tok, Prefix.size() + TailOffset + Split.first, Split.second, Postfix,
       Prefix, InPPDirective, 1, StartColumn);
 }
 
+BreakableStringLiteralUsingOperators::BreakableStringLiteralUsingOperators(
+    const FormatToken &Tok, QuoteStyleType QuoteStyle, bool UnindentPlus,
+    unsigned StartColumn, unsigned UnbreakableTailLength, bool InPPDirective,
+    encoding::Encoding Encoding, const FormatStyle &Style)
+    : BreakableStringLiteral(
+          Tok, StartColumn, /*Prefix=*/QuoteStyle == SingleQuotes ? "'"
+                            : QuoteStyle == AtDoubleQuotes        ? "@\""
+                                                                  : "\"",
+          /*Postfix=*/QuoteStyle == SingleQuotes ? "'" : "\"",
+          UnbreakableTailLength, InPPDirective, Encoding, Style),
+      BracesNeeded(Tok.isNot(TT_StringInConcatenation)),
+      QuoteStyle(QuoteStyle) {
+  // Find the replacement text for inserting braces and quotes and line breaks.
+  // We don't create an allocated string concatenated from parts here because it
+  // has to outlive the BreakableStringliteral object.  The brace replacements
+  // include a quote so that WhitespaceManager can tell it apart from whitespace
+  // replacements between the string and surrounding tokens.
+
+  // The option is not implemented in JavaScript.
+  bool SignOnNewLine =
+      !Style.isJavaScript() &&
+      Style.BreakBeforeBinaryOperators != FormatStyle::BOS_None;
+
+  if (Style.isVerilog()) {
+    // In Verilog, all strings are quoted by double quotes, joined by commas,
+    // and wrapped in braces.  The comma is always before the newline.
+    assert(QuoteStyle == DoubleQuotes);
+    LeftBraceQuote = Style.Cpp11BracedListStyle ? "{\"" : "{ \"";
+    RightBraceQuote = Style.Cpp11BracedListStyle ? "\"}" : "\" }";
+    Postfix = "\",";
+    Prefix = "\"";
+  } else {
+    // The plus sign may be on either line.  And also C# and JavaScript have
+    // different quoting styles.
+    if (QuoteStyle == SingleQuotes) {
+      LeftBraceQuote = Style.SpacesInParensOptions.Other ? "( '" : "('";
+      RightBraceQuote = Style.SpacesInParensOptions.Other ? "' )" : "')";
+      Postfix = SignOnNewLine ? "'" : "' +";
+      Prefix = SignOnNewLine ? "+ '" : "'";
+    } else {
+      if (QuoteStyle == AtDoubleQuotes) {
+        LeftBraceQuote = Style.SpacesInParensOptions.Other ? "( @" : "(@";
+        Prefix = SignOnNewLine ? "+ @\"" : "@\"";
+      } else {
+        LeftBraceQuote = Style.SpacesInParensOptions.Other ? "( \"" : "(\"";
+        Prefix = SignOnNewLine ? "+ \"" : "\"";
+      }
+      RightBraceQuote = Style.SpacesInParensOptions.Other ? "\" )" : "\")";
+      Postfix = SignOnNewLine ? "\"" : "\" +";
+    }
+  }
+
+  // Following lines are indented by the width of the brace and space if any.
+  ContinuationIndent = BracesNeeded ? LeftBraceQuote.size() - 1 : 0;
+  // The plus sign may need to be unindented depending on the style.
+  // FIXME: Add support for DontAlign.
+  if (!Style.isVerilog() && SignOnNewLine && !BracesNeeded && UnindentPlus &&
+      Style.AlignOperands == FormatStyle::OAS_AlignAfterOperator) {
+    ContinuationIndent -= 2;
+  }
+}
+
+unsigned BreakableStringLiteralUsingOperators::getRemainingLength(
+    unsigned LineIndex, unsigned Offset, unsigned StartColumn) const {
+  return UnbreakableTailLength + (BracesNeeded ? RightBraceQuote.size() : 1) +
+         encoding::columnWidthWithTabs(Line.substr(Offset), StartColumn,
+                                       Style.TabWidth, Encoding);
+}
+
+unsigned
+BreakableStringLiteralUsingOperators::getContentStartColumn(unsigned LineIndex,
+                                                            bool Break) const {
+  return std::max(
+      0,
+      static_cast<int>(StartColumn) +
+          (Break ? ContinuationIndent + static_cast<int>(Prefix.size())
+                 : (BracesNeeded ? static_cast<int>(LeftBraceQuote.size()) - 1
+                                 : 0) +
+                       (QuoteStyle == AtDoubleQuotes ? 2 : 1)));
+}
+
+void BreakableStringLiteralUsingOperators::insertBreak(
+    unsigned LineIndex, unsigned TailOffset, Split Split,
+    unsigned ContentIndent, WhitespaceManager &Whitespaces) const {
+  Whitespaces.replaceWhitespaceInToken(
+      Tok, /*Offset=*/(QuoteStyle == AtDoubleQuotes ? 2 : 1) + TailOffset +
+               Split.first,
+      /*ReplaceChars=*/Split.second, /*PreviousPostfix=*/Postfix,
+      /*CurrentPrefix=*/Prefix, InPPDirective, /*NewLines=*/1,
+      /*Spaces=*/
+      std::max(0, static_cast<int>(StartColumn) + ContinuationIndent));
+}
+
+void BreakableStringLiteralUsingOperators::updateAfterBroken(
+    WhitespaceManager &Whitespaces) const {
+  // Add the braces required for breaking the token if they are needed.
+  if (!BracesNeeded)
+    return;
+
+  // To add a brace or parenthesis, we replace the quote (or the at sign) with a
+  // brace and another quote.  This is because the rest of the program requires
+  // one replacement for each source range.  If we replace the empty strings
+  // around the string, it may conflict with whitespace replacements between the
+  // string and adjacent tokens.
+  Whitespaces.replaceWhitespaceInToken(
+      Tok, /*Offset=*/0, /*ReplaceChars=*/1, /*PreviousPostfix=*/"",
+      /*CurrentPrefix=*/LeftBraceQuote, InPPDirective, /*NewLines=*/0,
+      /*Spaces=*/0);
+  Whitespaces.replaceWhitespaceInToken(
+      Tok, /*Offset=*/Tok.TokenText.size() - 1, /*ReplaceChars=*/1,
+      /*PreviousPostfix=*/RightBraceQuote,
+      /*CurrentPrefix=*/"", InPPDirective, /*NewLines=*/0, /*Spaces=*/0);
+}
+
 BreakableComment::BreakableComment(const FormatToken &Token,
                                    unsigned StartColumn, bool InPPDirective,
                                    encoding::Encoding Encoding,
                                    const FormatStyle &Style)
     : BreakableToken(Token, InPPDirective, Encoding, Style),
       StartColumn(StartColumn) {}
 
 unsigned BreakableComment::getLineCount() const { return Lines.size(); }

clang/lib/Format/ContinuationIndenter.cpp

 
 // Returns true if a TT_SelectorName should be indented when wrapped,
 // false otherwise.
 static bool shouldIndentWrappedSelectorName(const FormatStyle &Style,
                                             LineType LineType) {
   return Style.IndentWrappedFunctionNames || LineType == LT_ObjCMethodDecl;
 }
 
+// Returns true if a binary operator following \p Tok should be unindented when
+// the style permits it.
+static bool shouldUnindentNextOperator(const FormatToken &Tok) {
+  const FormatToken *Previous = Tok.getPreviousNonComment();
+  return Previous && (Previous->getPrecedence() == prec::Assignment ||
+                      Previous->isOneOf(tok::kw_return, TT_RequiresClause));
+}
+
 // Returns the length of everything up to the first possible line break after
 // the ), ], } or > matching \c Tok.
 static unsigned getLengthToMatchingParen(const FormatToken &Tok,
                                          ArrayRef<ParenState> Stack) {
   // Normally whether or not a break before T is possible is calculated and
   // stored in T.CanBreakBefore. Braces, array initializers and text proto
   // messages like `key: < ... >` are an exception: a break is possible
   // before a closing brace R if a break was inserted after the corresponding
for (const auto &PrecedenceLevel : llvm::reverse(Current.FakeLParens)) {
     }
 
     // Special case for generic selection expressions, its comma-separated
     // expressions are not aligned to the opening paren like regular calls, but
     // rather continuation-indented relative to the _Generic keyword.
     if (Previous && Previous->endsSequence(tok::l_paren, tok::kw__Generic))
       NewParenState.Indent = CurrentState.LastSpace;
 
-    if (Previous &&
-        (Previous->getPrecedence() == prec::Assignment ||
-         Previous->isOneOf(tok::kw_return, TT_RequiresClause) ||
-         (PrecedenceLevel == prec::Conditional && Previous->is(tok::question) &&
-          Previous->is(TT_ConditionalExpr))) &&
+    if ((shouldUnindentNextOperator(Current) ||
+         (Previous &&
+          (PrecedenceLevel == prec::Conditional &&
+           Previous->is(tok::question) && Previous->is(TT_ConditionalExpr)))) &&
         !Newline) {
       // If BreakBeforeBinaryOperators is set, un-indent a bit to account for
       // the operator and keep the operands aligned.
       if (Style.AlignOperands == FormatStyle::OAS_AlignAfterOperator)
         NewParenState.UnindentOperator = true;
       // Mark indentation as alignment if the expression is aligned.
       if (Style.AlignOperands != FormatStyle::OAS_DontAlign)
         NewParenState.IsAligned = true;
ContinuationIndenter::getRawStringStyle(const FormatToken &Current,
   RawStringStyle->ColumnLimit = getColumnLimit(State);
   return RawStringStyle;
 }
 
 std::unique_ptr<BreakableToken>
 ContinuationIndenter::createBreakableToken(const FormatToken &Current,
                                            LineState &State, bool AllowBreak) {
   unsigned StartColumn = State.Column - Current.ColumnWidth;
   if (Current.isStringLiteral()) {
-    // FIXME: String literal breaking is currently disabled for C#, Java, Json
-    // and JavaScript, as it requires strings to be merged using "+" which we
-    // don't support.
-    if (Style.Language == FormatStyle::LK_Java || Style.isJavaScript() ||
-        Style.isCSharp() || Style.isJson() || !Style.BreakStringLiterals ||
-        !AllowBreak) {
+    // Strings in JSON can not be broken.
+    if (Style.isJson() || !Style.BreakStringLiterals || !AllowBreak)
       return nullptr;
-    }
 
     // Don't break string literals inside preprocessor directives (except for
     // #define directives, as their contents are stored in separate lines and
     // are not affected by this check).
     // This way we avoid breaking code with line directives and unknown
     // preprocessor directives that contain long string literals.
     if (State.Line->Type == LT_PreprocessorDirective)
       return nullptr;
     // Exempts unterminated string literals from line breaking. The user will
     // likely want to terminate the string before any line breaking is done.
     if (Current.IsUnterminatedLiteral)
       return nullptr;
     // Don't break string literals inside Objective-C array literals (doing so
     // raises the warning -Wobjc-string-concatenation).
     if (State.Stack.back().IsInsideObjCArrayLiteral)
       return nullptr;
 
     StringRef Text = Current.TokenText;
+    // We need this to address the case where there is an unbreakable tail only
+    // if certain other formatting decisions have been taken. The
+    // UnbreakableTailLength of Current is an overapproximation is that case and
+    // we need to be correct here.
+    unsigned UnbreakableTailLength = (State.NextToken && canBreak(State))
+                                         ? 0
+                                         : Current.UnbreakableTailLength;
+
+    if (Style.isVerilog() || Style.Language == FormatStyle::LK_Java ||
+        Style.isJavaScript() || Style.isCSharp()) {
+      BreakableStringLiteralUsingOperators::QuoteStyleType QuoteStyle;
+      if (Style.isJavaScript() && Text.startswith("'") && Text.endswith("'")) {
+        QuoteStyle = BreakableStringLiteralUsingOperators::SingleQuotes;
+      } else if (Style.isCSharp() && Text.startswith("@\"") &&
+                 Text.endswith("\"")) {
+        QuoteStyle = BreakableStringLiteralUsingOperators::AtDoubleQuotes;
+      } else if (Text.startswith("\"") && Text.endswith("\"")) {
+        QuoteStyle = BreakableStringLiteralUsingOperators::DoubleQuotes;
+      } else {
+        return nullptr;
+      }
+      return std::make_unique<BreakableStringLiteralUsingOperators>(
+          Current, QuoteStyle,
+          /*UnindentPlus=*/shouldUnindentNextOperator(Current), StartColumn,
+          UnbreakableTailLength, State.Line->InPPDirective, Encoding, Style);
+    }
+
     StringRef Prefix;
     StringRef Postfix;
     // FIXME: Handle whitespace between '_T', '(', '"..."', and ')'.
     // FIXME: Store Prefix and Suffix (or PrefixLength and SuffixLength to
     // reduce the overhead) for each FormatToken, which is a string, so that we
     // don't run multiple checks here on the hot path.
     if ((Text.endswith(Postfix = "\"") &&
          (Text.startswith(Prefix = "@\"") || Text.startswith(Prefix = "\"") ||
           Text.startswith(Prefix = "u\"") || Text.startswith(Prefix = "U\"") ||
           Text.startswith(Prefix = "u8\"") ||
           Text.startswith(Prefix = "L\""))) ||
         (Text.startswith(Prefix = "_T(\"") && Text.endswith(Postfix = "\")"))) {
-      // We need this to address the case where there is an unbreakable tail
-      // only if certain other formatting decisions have been taken. The
-      // UnbreakableTailLength of Current is an overapproximation is that case
-      // and we need to be correct here.
-      unsigned UnbreakableTailLength = (State.NextToken && canBreak(State))
-                                           ? 0
-                                           : Current.UnbreakableTailLength;
       return std::make_unique<BreakableStringLiteral>(
           Current, StartColumn, Prefix, Postfix, UnbreakableTailLength,
           State.Line->InPPDirective, Encoding, Style);
     }
   } else if (Current.is(TT_BlockComment)) {
     if (!Style.ReflowComments ||
         // If a comment token switches formatting, like
         // /* clang-format on */, we don't want to break it further,
ContinuationIndenter::createBreakableToken(const FormatToken &Current,
   }
   return nullptr;
 }
 
 std::pair<unsigned, bool>
 ContinuationIndenter::breakProtrudingToken(const FormatToken &Current,
                                            LineState &State, bool AllowBreak,
                                            bool DryRun, bool Strict) {
   std::unique_ptr<const BreakableToken> Token =
       createBreakableToken(Current, State, AllowBreak);
   if (!Token)
     return {0, false};
   assert(Token->getLineCount() > 0);
   unsigned ColumnLimit = getColumnLimit(State);
   if (Current.is(TT_LineComment)) {
     // We don't insert backslashes when breaking line comments.
     ColumnLimit = Style.ColumnLimit;
RemainingTokenColumns = Token->getRemainingLength(
         TailOffset + SplitAfterLastLine.first + SplitAfterLastLine.second,
         ContentStartColumn);
   }
 
   State.Column = ContentStartColumn + RemainingTokenColumns -
                  Current.UnbreakableTailLength;
 
   if (BreakInserted) {
+    if (!DryRun)
+      Token->updateAfterBroken(Whitespaces);
+
     // If we break the token inside a parameter list, we need to break before
     // the next parameter on all levels, so that the next parameter is clearly
     // visible. Line comments already introduce a break.
     if (Current.isNot(TT_LineComment))
       for (ParenState &Paren : State.Stack)
         Paren.BreakBeforeParameter = true;
 
     if (Current.is(TT_BlockComment))

clang/lib/Format/FormatToken.h

#define LIST_TOKEN_TYPES                                                       \
   TYPE(RequiresClauseInARequiresExpression)                                    \
   TYPE(RequiresExpression)                                                     \
   TYPE(RequiresExpressionLBrace)                                               \
   TYPE(RequiresExpressionLParen)                                               \
   TYPE(SelectorName)                                                           \
   TYPE(StartOfName)                                                            \
   TYPE(StatementAttributeLikeMacro)                                            \
   TYPE(StatementMacro)                                                         \
+  /* A string that is part of a string concatenation. For C#, JavaScript, and  \
+   * Java, it is used for marking whether a string needs parentheses around it \
+   * if it is to be split into parts joined by `+`. For Verilog, whether       \
+   * braces need to be added to split it. Not used for other languages. */     \
+  TYPE(StringInConcatenation)                                                  \
   TYPE(StructLBrace)                                                           \
   TYPE(StructuredBindingLSquare)                                               \
   TYPE(TemplateCloser)                                                         \
   TYPE(TemplateOpener)                                                         \
   TYPE(TemplateString)                                                         \
   TYPE(TrailingAnnotation)                                                     \
   TYPE(TrailingReturnArrow)                                                    \
   TYPE(TrailingUnaryOperator)                                                  \

clang/lib/Format/TokenAnnotator.cpp

bool parseBrace() {
     ScopedContextCreator ContextCreator(*this, tok::l_brace, 1);
     Contexts.back().ColonIsDictLiteral = true;
     if (OpeningBrace.is(BK_BracedInit))
       Contexts.back().IsExpression = true;
     if (Style.isJavaScript() && OpeningBrace.Previous &&
         OpeningBrace.Previous->is(TT_JsTypeColon)) {
       Contexts.back().IsExpression = false;
     }
+    if (Style.isVerilog() &&
+        (!OpeningBrace.getPreviousNonComment() ||
+         OpeningBrace.getPreviousNonComment()->isNot(Keywords.kw_apostrophe))) {
+      Contexts.back().VerilogMayBeConcatenation = true;
+    }
 
     unsigned CommaCount = 0;
     while (CurrentToken) {
       if (CurrentToken->is(tok::r_brace)) {
         assert(!Scopes.empty());
         assert(Scopes.back() == getScopeType(OpeningBrace));
         Scopes.pop_back();
         assert(OpeningBrace.Optional == CurrentToken->Optional);
struct Context {
     bool ColonIsObjCMethodExpr = false;
     FormatToken *FirstObjCSelectorName = nullptr;
     FormatToken *FirstStartOfName = nullptr;
     bool CanBeExpression = true;
     bool CaretFound = false;
     bool InCpp11AttributeSpecifier = false;
     bool InCSharpAttributeSpecifier = false;
     bool VerilogAssignmentFound = false;
+    // Whether the braces may mean concatenation instead of structure or array
+    // literal.
+    bool VerilogMayBeConcatenation = false;
     enum {
       Unknown,
       // Like the part after `:` in a constructor.
       //   Context(...) : IsExpression(IsExpression)
       CtorInitializer,
       // Like in the parentheses in a foreach.
       ForEachMacro,
       // Like the inheritance list in a class declaration.
if (Current.is(Keywords.kw_instanceof)) {
         } else {
           // The lexer has for some reason determined a comment here. But we
           // cannot really handle it, if it isn't properly terminated.
           Current.Tok.setKind(tok::unknown);
         }
       } else {
         Current.setType(TT_LineComment);
       }
+    } else if (Current.is(tok::string_literal)) {
+      if (Style.isVerilog() && Contexts.back().VerilogMayBeConcatenation &&
+          Current.getPreviousNonComment() &&
+          Current.getPreviousNonComment()->isOneOf(tok::comma, tok::l_brace) &&
+          Current.getNextNonComment() &&
+          Current.getNextNonComment()->isOneOf(tok::comma, tok::r_brace)) {
+        Current.setType(TT_StringInConcatenation);
+      }
     } else if (Current.is(tok::l_paren)) {
       if (lParenStartsCppCast(Current))
         Current.setType(TT_CppCastLParen);
     } else if (Current.is(tok::r_paren)) {
       if (rParenEndsCast(Current))
         Current.setType(TT_CastRParen);
       if (Current.MatchingParen && Current.Next &&
           !Current.Next->isBinaryOperator() &&
while (Current) {
 
       if (Precedence == CurrentPrecedence && Current &&
           Current->is(TT_SelectorName)) {
         if (LatestOperator)
           addFakeParenthesis(Start, prec::Level(Precedence));
         Start = Current;
       }
 
+      if ((Style.isCSharp() || Style.isJavaScript() ||
+           Style.Language == FormatStyle::LK_Java) &&
+          Precedence == prec::Additive && Current) {
+        // A string can be broken without parentheses around it when it is
+        // already in a sequence of strings joined by `+` signs.
+        FormatToken *Prev = Current->getPreviousNonComment();
+        if (Prev && Prev->is(tok::string_literal) &&
+            (Prev == Start || Prev->endsSequence(tok::string_literal, tok::plus,
+                                                 TT_StringInConcatenation))) {
+          Prev->setType(TT_StringInConcatenation);
+        }
+      }
+
       // At the end of the line or when an operator with lower precedence is
       // found, insert fake parenthesis and return.
       if (!Current ||
           (Current->closesScope() &&
            (Current->MatchingParen || Current->is(TT_TemplateString))) ||
           (CurrentPrecedence != -1 && CurrentPrecedence < Precedence) ||
           (CurrentPrecedence == prec::Conditional &&
            Precedence == prec::Assignment && Current->is(tok::colon))) {

clang/lib/Format/WhitespaceManager.cpp

 #include <algorithm>
 
 namespace clang {
 namespace format {
 
 bool WhitespaceManager::Change::IsBeforeInFile::operator()(
     const Change &C1, const Change &C2) const {
   return SourceMgr.isBeforeInTranslationUnit(
              C1.OriginalWhitespaceRange.getBegin(),
-      C2.OriginalWhitespaceRange.getBegin());
+             C2.OriginalWhitespaceRange.getBegin()) ||
+         (C1.OriginalWhitespaceRange.getBegin() ==
+              C2.OriginalWhitespaceRange.getBegin() &&
+          SourceMgr.isBeforeInTranslationUnit(
+              C1.OriginalWhitespaceRange.getEnd(),
+              C2.OriginalWhitespaceRange.getEnd()));
 }
 
 WhitespaceManager::Change::Change(const FormatToken &Tok,
                                   bool CreateReplacement,
                                   SourceRange OriginalWhitespaceRange,
                                   int Spaces, unsigned StartOfTokenColumn,
                                   unsigned NewlinesBefore,
                                   StringRef PreviousLinePostfix,
for (auto *CellIter = Cells.begin(); CellIter != Cells.end(); ++CellIter) {
     }
   }
   return std::move(CellDesc);
 }
 
 void WhitespaceManager::generateChanges() {
   for (unsigned i = 0, e = Changes.size(); i != e; ++i) {
     const Change &C = Changes[i];
-    if (i > 0 && Changes[i - 1].OriginalWhitespaceRange.getBegin() ==
-                     C.OriginalWhitespaceRange.getBegin()) {
-      // Do not generate two replacements for the same location.
+    if (i > 0) {
+      auto Last = Changes[i - 1].OriginalWhitespaceRange;
+      auto New = Changes[i].OriginalWhitespaceRange;
+      // Do not generate two replacements for the same location.  As a special
+      // case, it is allowed if there is a replacement for the empty range
+      // between 2 tokens and another non-empty range at the start of the second
+      // token.  We didn't implement logic to combine replacements for 2
+      // consecutive source ranges into a single replacement, because the
+      // program works fine without it.
+      //
+      // We can't eliminate empty original whitespace ranges.  They appear when
+      // 2 tokens have no whitespace in between in the input.  It does not
+      // matter whether whitespace is to be added.  If no whitespace is to be
+      // added, the replacement will be empty, and it gets eliminated after this
+      // step in storeReplacement.  For example, if the input is `foo();`,
+      // there will be a replacement for the range between every consecutive
+      // pair of tokens.
+      //
+      // A replacement at the start of a token can be added by
+      // BreakableStringLiteralUsingOperators::insertBreak when it adds braces
+      // around the string literal.  Say Verilog code is being formatted and the
+      // first line is to become the next 2 lines.
+      //     x("long string");
+      //     x({"long ",
+      //        "string"});
+      // There will be a replacement for the empty range between the parenthesis
+      // and the string and another replacement for the quote character.  The
+      // replacement for the empty range between the parenthesis and the quote
+      // comes from ContinuationIndenter::addTokenOnCurrentLine when it changes
+      // the original empty range between the parenthesis and the string to
+      // another empty one.  The replacement for the quote character comes from
+      // BreakableStringLiteralUsingOperators::insertBreak when it adds the
+      // brace.  In the example, the replacement for the empty range is the same
+      // as the original text.  However, eliminating replacements that are same
+      // as the original does not help in general.  For example, a newline can
+      // be inserted, causing the first line to become the next 3 lines.
+      //     xxxxxxxxxxx("long string");
+      //     xxxxxxxxxxx(
+      //         {"long ",
+      //          "string"});
+      // In that case, the empty range between the parenthesis and the string
+      // will be replaced by a newline and 4 spaces.  So we will still have to
+      // deal with a replacement for an empty source range followed by a
+      // replacement for a non-empty source range.
+      if (Last.getBegin() == New.getBegin() &&
+          (Last.getEnd() != Last.getBegin() ||
+           New.getEnd() == New.getBegin())) {
         continue;
       }
+    }
     if (C.CreateReplacement) {
       std::string ReplacementText = C.PreviousLinePostfix;
       if (C.ContinuesPPDirective) {
         appendEscapedNewlineText(ReplacementText, C.NewlinesBefore,
                                  C.PreviousEndOfTokenColumn,
                                  C.EscapedNewlineColumn);
       } else {
         appendNewlineText(ReplacementText, C.NewlinesBefore);

clang/unittests/Format/FormatTestCSharp.cpp

 //===- unittest/Format/FormatTestCSharp.cpp - Formatting tests for CSharp -===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
-#include "FormatTestUtils.h"
-#include "clang/Format/Format.h"
-#include "llvm/Support/Debug.h"
-#include "gtest/gtest.h"
+#include "FormatTestBase.h"
 
 #define DEBUG_TYPE "format-test"
 
 namespace clang {
 namespace format {
+namespace test {
+namespace {
 
-class FormatTestCSharp : public ::testing::Test {
+class FormatTestCSharp : public test::FormatTestBase {
 protected:
+  FormatStyle getDefaultStyle() const override {
+    return getMicrosoftStyle(FormatStyle::LK_CSharp);
+  }
+
   static std::string format(llvm::StringRef Code, unsigned Offset,
                             unsigned Length, const FormatStyle &Style) {
     LLVM_DEBUG(llvm::errs() << "---\n");
     LLVM_DEBUG(llvm::errs() << Code << "\n\n");
     std::vector<tooling::Range> Ranges(1, tooling::Range(Offset, Length));
     tooling::Replacements Replaces = reformat(Style, Code, Ranges);
     auto Result = applyAllReplacements(Code, Replaces);
     EXPECT_TRUE(static_cast<bool>(Result));
     LLVM_DEBUG(llvm::errs() << "\n" << *Result << "\n\n");
     return *Result;
   }
 
   static std::string
   format(llvm::StringRef Code,
          const FormatStyle &Style = getMicrosoftStyle(FormatStyle::LK_CSharp)) {
     return format(Code, 0, Code.size(), Style);
   }
 
   static FormatStyle getStyleWithColumns(unsigned ColumnLimit) {
     FormatStyle Style = getMicrosoftStyle(FormatStyle::LK_CSharp);
     Style.ColumnLimit = ColumnLimit;
     return Style;
   }
-
-  static void verifyFormat(
-      llvm::StringRef Code,
-      const FormatStyle &Style = getMicrosoftStyle(FormatStyle::LK_CSharp)) {
-    EXPECT_EQ(Code.str(), format(Code, Style)) << "Expected code is not stable";
-    EXPECT_EQ(Code.str(), format(test::messUp(Code), Style));
-  }
 };
 
 TEST_F(FormatTestCSharp, CSharpClass) {
   verifyFormat("public class SomeClass\n"
                "{\n"
                "    void f()\n"
                "    {\n"
                "    }\n"
verifyFormat("internal override String toString()\n"
                "}");
 
   verifyFormat("internal static String toString()\n"
                "{\n"
                "}");
 }
 
 TEST_F(FormatTestCSharp, NoStringLiteralBreaks) {
+  // Breaking of interpolated strings is not implemented.
+  auto Style = getDefaultStyle();
+  Style.ColumnLimit = 40;
+  Style.BreakStringLiterals = true;
+  verifyFormat("foo("
+               "$\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+               "aaaaaaa\");",
+               Style);
+}
+
+TEST_F(FormatTestCSharp, StringLiteralBreaks) {
+  // The line is 75 characters long.  The default limit for the Microsoft style
+  // is 120.
+  auto Style = getDefaultStyle();
+  Style.BreakStringLiterals = true;
   verifyFormat("foo("
                "\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
-               "aaaaaa\");");
+               "aaaaaa\");",
+               Style);
+  // When the column limit is smaller, the string should get broken.
+  Style.ColumnLimit = 40;
+  verifyFormat(R"(foo("aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" +
+    "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" +
+    "aaa");)",
+               "foo("
+               "\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+               "aaaaaa\");",
+               Style);
+  // The new quotes should be the same as the original.
+  verifyFormat(R"(foo(@"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" +
+    @"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" +
+    @"aaaaa");)",
+               "foo("
+               "@\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+               "aaaaaaa\");",
+               Style);
+  // The operators can be on either line.
+  Style.BreakBeforeBinaryOperators = FormatStyle::BOS_NonAssignment;
+  verifyFormat(R"(foo("aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+    + "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+    + "a");)",
+               "foo("
+               "\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+               "aaaaaa\");",
+               Style);
+  Style.AlignOperands = FormatStyle::OAS_AlignAfterOperator;
+  verifyFormat(R"(foo("aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+    + "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+    + "a");)",
+               "foo("
+               "\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+               "aaaaaa\");",
+               Style);
+  verifyFormat(R"(x = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+  + "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";)",
+               "x = "
+               "\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+               "aaaaaa\";",
+               Style);
 }
 
 TEST_F(FormatTestCSharp, CSharpVerbatiumStringLiterals) {
   verifyFormat("foo(@\"aaaaaaaa\\abc\\aaaa\");");
   // @"ABC\" + ToString("B") - handle embedded \ in literal string at
   // the end
   //
   /*
TEST_F(FormatTestCSharp, CSharpInterpolatedStringLiterals) {
   // the end
   verifyFormat("string s = $\"A{abc}BC\" + ToString(\"B\");");
   verifyFormat("$\"{domain}\\\\{user}\"");
   verifyFormat(
       "var verbatimInterpolated = $@\"C:\\Users\\{userName}\\Documents\\\";");
 }
 
 TEST_F(FormatTestCSharp, CSharpFatArrows) {
-  verifyFormat("Task serverTask = Task.Run(async() => {");
+  verifyIncompleteFormat("Task serverTask = Task.Run(async() => {");
   verifyFormat("public override string ToString() => \"{Name}\\{Age}\";");
 }
 
 TEST_F(FormatTestCSharp, CSharpConditionalExpressions) {
   FormatStyle Style = getGoogleStyle(FormatStyle::LK_CSharp);
   // conditional expression is not seen as a NullConditional.
   verifyFormat("var y = A < B ? -1 : 1;", Style);
 }
verifyFormat("[XmlIgnore]\n"
                "[ScriptIgnore]\n"
                "[JsonProperty(\"bar\")]\n"
                "public string Bar { set; get; }");
 
   verifyFormat("[TestMethod(\"start\", HelpText = \"Starts the server "
                "listening on provided host\")]\n"
                "public string Host { set; get; }");
 
-  verifyFormat(
+  verifyIncompleteFormat(
       "[DllImport(\"Hello\", EntryPoint = \"hello_world\")]\n"
       "// The const char* returned by hello_world must not be deleted.\n"
       "private static extern IntPtr HelloFromCpp();)");
 
   // Class attributes go on their own line and do not affect layout of
   // interfaces. Line wrapping decisions previously caused each interface to be
   // on its own line.
   verifyFormat("[SomeAttribute]\n"
TEST_F(FormatTestCSharp, CSharpSpaces) {
   verifyFormat(R"(new string[] { "A" })", Style);
   verifyFormat(R"(new string[] {})", Style);
   verifyFormat(R"(new Car { someVariableName })", Style);
   verifyFormat(R"(new Car { someVariableName })", Style);
   verifyFormat(R"(new Dictionary<string, string> { ["Key"] = "Value" };)",
                Style);
   verifyFormat(R"(Apply(x => x.Name, x => () => x.ID);)", Style);
   verifyFormat(R"(bool[] xs = { true, true };)", Style);
-  verifyFormat(R"(taskContext.Factory.Run(async () => doThing(args);)", Style);
+  verifyIncompleteFormat(
+      R"(taskContext.Factory.Run(async () => doThing(args);)", Style);
   verifyFormat(R"(catch (TestException) when (innerFinallyExecuted))", Style);
   verifyFormat(R"(private float[,] Values;)", Style);
   verifyFormat(R"(Result this[Index x] => Foo(x);)", Style);
 
   verifyFormat(R"(char[,,] rawCharArray = MakeCharacterGrid();)", Style);
   verifyFormat(R"(var (key, value))", Style);
 
   // `&&` is not seen as a reference.
verifyFormat("namespace union {\n"
                "};",
                Style);
 }
 
 TEST_F(FormatTestCSharp, BrokenBrackets) {
   EXPECT_NE("", format("int where b <")); // reduced from crasher
 }
 
+} // namespace
+} // namespace test
 } // namespace format
-} // end namespace clang
+} // namespace clang

clang/unittests/Format/FormatTestJS.cpp

TEST_F(FormatTestJS, TryCatch) {
   // But, of course, "catch" is a perfectly fine function name in JavaScript.
   verifyFormat("someObject.catch();");
   verifyFormat("someObject.new();");
 }
 
 TEST_F(FormatTestJS, StringLiteralConcatenation) {
   verifyFormat("var literal = 'hello ' +\n"
                "    'world';");
+
+  // Long strings should be broken.
+  verifyFormat("var literal =\n"
+               "    'xxxxxxxx ' +\n"
+               "    'xxxxxxxx';",
+               "var literal = 'xxxxxxxx xxxxxxxx';",
+               getGoogleJSStyleWithColumns(17));
+  verifyFormat("var literal =\n"
+               "    'xxxxxxxx ' +\n"
+               "    'xxxxxxxx';",
+               "var literal = 'xxxxxxxx xxxxxxxx';",
+               getGoogleJSStyleWithColumns(18));
+  verifyFormat("var literal =\n"
+               "    'xxxxxxxx' +\n"
+               "    ' xxxxxxxx';",
+               "var literal = 'xxxxxxxx xxxxxxxx';",
+               getGoogleJSStyleWithColumns(16));
+  // The quotes should be correct.
+  for (char OriginalQuote : {'\'', '"'}) {
+    auto VerifyQuotes = [=](FormatStyle::JavaScriptQuoteStyle StyleQuote,
+                            char TargetQuote) {
+      auto Style = getGoogleJSStyleWithColumns(17);
+      Style.JavaScriptQuotes = StyleQuote;
+      SmallString<48> Target{"var literal =\n"
+                             "    \"xxxxxxxx \" +\n"
+                             "    \"xxxxxxxx\";"};
+      SmallString<34> Original{"var literal = \"xxxxxxxx xxxxxxxx\";"};
+      std::replace(Target.begin(), Target.end(), '"', TargetQuote);
+      std::replace(Original.begin(), Original.end(), '"', OriginalQuote);
+      verifyFormat(Target, Original, Style);
+    };
+    VerifyQuotes(FormatStyle::JSQS_Leave, OriginalQuote);
+    VerifyQuotes(FormatStyle::JSQS_Single, '\'');
+    VerifyQuotes(FormatStyle::JSQS_Double, '"');
+  }
+  // Parentheses should be added when necessary.
+  verifyFormat("var literal =\n"
+               "    ('xxxxxxxx ' +\n"
+               "     'xxxxxx')[0];",
+               "var literal = 'xxxxxxxx xxxxxx'[0];",
+               getGoogleJSStyleWithColumns(18));
+  auto Style = getGoogleJSStyleWithColumns(20);
+  Style.SpacesInParens = FormatStyle::SIPO_Custom;
+  Style.SpacesInParensOptions.Other = true;
+  verifyFormat("var literal =\n"
+               "    ( 'xxxxxxxx ' +\n"
+               "      'xxxxxx' )[0];",
+               "var literal = 'xxxxxxxx xxxxxx'[0];", Style);
+  // FIXME: When the part before the string literal is shorter than the
+  // continuation indentation, and the option AlignAfterOpenBracket is set to
+  // AlwaysBreak which is the default for the Google style, the unbroken string
+  // does not get to a new line while the broken string does due to the added
+  // parentheses. The formatter does not do it in one pass.
+  EXPECT_EQ(
+      "x = ('xxxxxxxx ' +\n"
+      "     'xxxxxx')[0];",
+      format("x = 'xxxxxxxx xxxxxx'[0];", getGoogleJSStyleWithColumns(18)));
+  verifyFormat("x =\n"
+               "    ('xxxxxxxx ' +\n"
+               "     'xxxxxx')[0];",
+               getGoogleJSStyleWithColumns(18));
+  // Breaking of template strings ans regular expressions is not implemented.
+  verifyFormat("var literal =\n"
+               "    `xxxxxxxx xxxxxxxx`;",
+               getGoogleJSStyleWithColumns(18));
+  verifyFormat("var literal =\n"
+               "    /xxxxxxxx xxxxxxxx/;",
+               getGoogleJSStyleWithColumns(18));
+  // There can be breaks in the code inside a template string.
+  verifyFormat("var literal = `xxxxxx ${\n"
+               "    xxxxxxxxxx} xxxxxx`;",
+               "var literal = `xxxxxx ${xxxxxxxxxx} xxxxxx`;",
+               getGoogleJSStyleWithColumns(14));
+  verifyFormat("var literal = `xxxxxx ${\n"
+               "    xxxxxxxxxx} xxxxxx`;",
+               "var literal = `xxxxxx ${xxxxxxxxxx} xxxxxx`;",
+               getGoogleJSStyleWithColumns(15));
+  // Identifiers inside the code inside a template string should not be broken
+  // even if the column limit is exceeded.  This following behavior is not
+  // optimal.  The part after the closing brace which exceeds the column limit
+  // can be put on a new line.  Change this test when it is implemented.
+  verifyFormat("var literal = `xxxxxx ${\n"
+               "    xxxxxxxxxxxxxxxxxxxxxx} xxxxxx`;",
+               "var literal = `xxxxxx ${xxxxxxxxxxxxxxxxxxxxxx} xxxxxx`;",
+               getGoogleJSStyleWithColumns(14));
+  verifyFormat("var literal = `xxxxxx ${\n"
+               "    xxxxxxxxxxxxxxxxxxxxxx +\n"
+               "    xxxxxxxxxxxxxxxxxxxxxx} xxxxxx`;",
+               "var literal = `xxxxxx ${xxxxxxxxxxxxxxxxxxxxxx + "
+               "xxxxxxxxxxxxxxxxxxxxxx} xxxxxx`;",
+               getGoogleJSStyleWithColumns(14));
 }
 
 TEST_F(FormatTestJS, RegexLiteralClassification) {
   // Regex literals.
   verifyFormat("var regex = /abc/;");
   verifyFormat("f(/abc/);");
   verifyFormat("f(abc, /abc/);");
   verifyFormat("some_map[/abc/];");

clang/unittests/Format/FormatTestJava.cpp

 //===- unittest/Format/FormatTestJava.cpp - Formatting tests for Java -----===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
-#include "FormatTestUtils.h"
-#include "clang/Format/Format.h"
-#include "llvm/Support/Debug.h"
-#include "gtest/gtest.h"
+#include "FormatTestBase.h"
 
 #define DEBUG_TYPE "format-test"
 
 namespace clang {
 namespace format {
+namespace test {
+namespace {
 
-class FormatTestJava : public ::testing::Test {
+class FormatTestJava : public test::FormatTestBase {
 protected:
-  static std::string format(llvm::StringRef Code, unsigned Offset,
-                            unsigned Length, const FormatStyle &Style) {
-    LLVM_DEBUG(llvm::errs() << "---\n");
-    LLVM_DEBUG(llvm::errs() << Code << "\n\n");
-    std::vector<tooling::Range> Ranges(1, tooling::Range(Offset, Length));
-    tooling::Replacements Replaces = reformat(Style, Code, Ranges);
-    auto Result = applyAllReplacements(Code, Replaces);
-    EXPECT_TRUE(static_cast<bool>(Result));
-    LLVM_DEBUG(llvm::errs() << "\n" << *Result << "\n\n");
-    return *Result;
-  }
-
-  static std::string
-  format(llvm::StringRef Code,
-         const FormatStyle &Style = getGoogleStyle(FormatStyle::LK_Java)) {
-    return format(Code, 0, Code.size(), Style);
+  FormatStyle getDefaultStyle() const override {
+    return getGoogleStyle(FormatStyle::LK_Java);
   }
 
   static FormatStyle getStyleWithColumns(unsigned ColumnLimit) {
     FormatStyle Style = getGoogleStyle(FormatStyle::LK_Java);
     Style.ColumnLimit = ColumnLimit;
     return Style;
   }
-
-  static void verifyFormat(
-      llvm::StringRef Code,
-      const FormatStyle &Style = getGoogleStyle(FormatStyle::LK_Java)) {
-    EXPECT_EQ(Code.str(), format(Code, Style)) << "Expected code is not stable";
-    EXPECT_EQ(Code.str(), format(test::messUp(Code), Style));
-  }
 };
 
 TEST_F(FormatTestJava, NoAlternativeOperatorNames) {
   verifyFormat("someObject.and();");
 }
 
 TEST_F(FormatTestJava, UnderstandsCasts) {
   verifyFormat("a[b >> 1] = (byte) (c() << 4);");
verifyFormat("Runnable someLambda = () -> {\n"
                "}");
 
   verifyFormat("Runnable someLambda =\n"
                "    (int aaaaa) -> DoSomething(aaaaa);",
                getStyleWithColumns(40));
 }
 
 TEST_F(FormatTestJava, BreaksStringLiterals) {
-  // FIXME: String literal breaking is currently disabled for Java and JS, as it
-  // requires strings to be merged using "+" which we don't support.
-  verifyFormat("\"some text other\";", getStyleWithColumns(14));
+  verifyFormat("x = \"some text \"\n"
+               "    + \"other\";",
+               "x = \"some text other\";", getStyleWithColumns(18));
 }
 
 TEST_F(FormatTestJava, AlignsBlockComments) {
   EXPECT_EQ("/*\n"
             " * Really multi-line\n"
             " * comment.\n"
             " */\n"
             "void f() {}",
TEST_F(FormatTestJava, ShortFunctions) {
   verifyFormat("enum Enum {\n"
                "  E1,\n"
                "  E2;\n"
                "  void f() { return; }\n"
                "}",
                Style);
 }
 
+} // namespace
+} // namespace test
 } // namespace format
-} // end namespace clang
+} // namespace clang

clang/unittests/Format/FormatTestVerilog.cpp

TEST_F(FormatTestVerilog, Streaming) {
   verifyFormat("{>>{a, b, c}} = 23'b1;");
   verifyFormat("{>>{a, b, c}} = x;");
   verifyFormat("{>>{j}} = x;");
   verifyFormat("{>>byte{j}} = x;");
   verifyFormat("{<<{j}} = x;");
   verifyFormat("{<<byte{j}} = x;");
 }
 
+TEST_F(FormatTestVerilog, StringLiteral) {
+  // Long strings should be broken.
+  verifyFormat(R"(x({"xxxxxxxxxxxxxxxx ",
+   "xxxx"});)",
+               R"(x({"xxxxxxxxxxxxxxxx xxxx"});)",
+               getStyleWithColumns(getDefaultStyle(), 23));
+  verifyFormat(R"(x({"xxxxxxxxxxxxxxxx",
+   " xxxx"});)",
+               R"(x({"xxxxxxxxxxxxxxxx xxxx"});)",
+               getStyleWithColumns(getDefaultStyle(), 22));
+  // Braces should be added when they don't already exist.
+  verifyFormat(R"(x({"xxxxxxxxxxxxxxxx ",
+   "xxxx"});)",
+               R"(x("xxxxxxxxxxxxxxxx xxxx");)",
+               getStyleWithColumns(getDefaultStyle(), 23));
+  verifyFormat(R"(x({"xxxxxxxxxxxxxxxx",
+   " xxxx"});)",
+               R"(x("xxxxxxxxxxxxxxxx xxxx");)",
+               getStyleWithColumns(getDefaultStyle(), 22));
+  verifyFormat(R"(x({{"xxxxxxxxxxxxxxxx ",
+    "xxxx"} == x});)",
+               R"(x({"xxxxxxxxxxxxxxxx xxxx" == x});)",
+               getStyleWithColumns(getDefaultStyle(), 24));
+  verifyFormat(R"(string x = {"xxxxxxxxxxxxxxxx ",
+            "xxxxxxxx"};)",
+               R"(string x = "xxxxxxxxxxxxxxxx xxxxxxxx";)",
+               getStyleWithColumns(getDefaultStyle(), 32));
+  // Space around braces should be correct.
+  auto Style = getStyleWithColumns(getDefaultStyle(), 24);
+  Style.Cpp11BracedListStyle = false;
+  verifyFormat(R"(x({ "xxxxxxxxxxxxxxxx ",
+    "xxxx" });)",
+               R"(x("xxxxxxxxxxxxxxxx xxxx");)", Style);
+  // Braces should not be added twice.
+  verifyFormat(R"(x({"xxxxxxxx",
+   "xxxxxxxx",
+   "xxxxxx"});)",
+               R"(x("xxxxxxxxxxxxxxxxxxxxxx");)",
+               getStyleWithColumns(getDefaultStyle(), 14));
+  verifyFormat(R"(x({"xxxxxxxxxxxxxxxx ",
+   "xxxxxxxxxxxxxxxx ",
+   "xxxx"});)",
+               R"(x("xxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxx xxxx");)",
+               getStyleWithColumns(getDefaultStyle(), 23));
+  verifyFormat(R"(x({"xxxxxxxxxxxxxxxx ",
+   "xxxxxxxxxxxxxxxx ",
+   "xxxx"});)",
+               R"(x({"xxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxx ", "xxxx"});)",
+               getStyleWithColumns(getDefaultStyle(), 23));
+  // These kinds of strings don't exist in Verilog.
+  verifyNoCrash(R"(x(@"xxxxxxxxxxxxxxxx xxxx");)",
+                getStyleWithColumns(getDefaultStyle(), 23));
+  verifyNoCrash(R"(x(u"xxxxxxxxxxxxxxxx xxxx");)",
+                getStyleWithColumns(getDefaultStyle(), 23));
+  verifyNoCrash(R"(x(u8"xxxxxxxxxxxxxxxx xxxx");)",
+                getStyleWithColumns(getDefaultStyle(), 23));
+  verifyNoCrash(R"(x(_T("xxxxxxxxxxxxxxxx xxxx"));)",
+                getStyleWithColumns(getDefaultStyle(), 23));
+}
+
 TEST_F(FormatTestVerilog, StructLiteral) {
   verifyFormat("c = '{0, 0.0};");
   verifyFormat("c = '{'{1, 1.0}, '{2, 2.0}};");
   verifyFormat("c = '{a: 0, b: 0.0};");
   verifyFormat("c = '{a: 0, b: 0.0, default: 0};");
   verifyFormat("c = ab'{a: 0, b: 0.0};");
   verifyFormat("c = ab'{cd: cd'{1, 1.0}, ef: ef'{2, 2.0}};");
   verifyFormat("c = ab'{cd'{1, 1.0}, ef'{2, 2.0}};");

clang/unittests/Format/TokenAnnotatorTest.cpp

TEST_F(TokenAnnotatorTest, UnderstandsVerilogOperators) {
   // Sensitivity list. The TT_Unknown type is clearly not binding for the
   // future, please adapt if those tokens get annotated.  This test is only here
   // to prevent the comma from being annotated as TT_VerilogInstancePortComma.
   Tokens = Annotate("always @(posedge x, posedge y);");
   ASSERT_EQ(Tokens.size(), 11u) << Tokens;
   EXPECT_TOKEN(Tokens[2], tok::l_paren, TT_Unknown);
   EXPECT_TOKEN(Tokens[5], tok::comma, TT_Unknown);
   EXPECT_TOKEN(Tokens[8], tok::r_paren, TT_Unknown);
+
+  // String literals in concatenation.
+  Tokens = Annotate("x = {\"\"};");
+  ASSERT_EQ(Tokens.size(), 7u) << Tokens;
+  EXPECT_TOKEN(Tokens[3], tok::string_literal, TT_StringInConcatenation);
+  Tokens = Annotate("x = {\"\", \"\"};");
+  ASSERT_EQ(Tokens.size(), 9u) << Tokens;
+  EXPECT_TOKEN(Tokens[3], tok::string_literal, TT_StringInConcatenation);
+  EXPECT_TOKEN(Tokens[5], tok::string_literal, TT_StringInConcatenation);
+  Tokens = Annotate("x = '{{\"\"}};");
+  ASSERT_EQ(Tokens.size(), 10u) << Tokens;
+  EXPECT_TOKEN(Tokens[5], tok::string_literal, TT_StringInConcatenation);
+  // Cases where the string should not be annotated that type.  Fix the
+  // `TT_Unknown` if needed in the future.
+  Tokens = Annotate("x = {\"\" == \"\"};");
+  ASSERT_EQ(Tokens.size(), 9u) << Tokens;
+  EXPECT_TOKEN(Tokens[3], tok::string_literal, TT_Unknown);
+  EXPECT_TOKEN(Tokens[5], tok::string_literal, TT_Unknown);
+  Tokens = Annotate("x = {(\"\")};");
+  ASSERT_EQ(Tokens.size(), 9u) << Tokens;
+  EXPECT_TOKEN(Tokens[4], tok::string_literal, TT_Unknown);
+  Tokens = Annotate("x = '{\"\"};");
+  ASSERT_EQ(Tokens.size(), 8u) << Tokens;
+  EXPECT_TOKEN(Tokens[4], tok::string_literal, TT_Unknown);
 }
 
 TEST_F(TokenAnnotatorTest, UnderstandConstructors) {
   auto Tokens = annotate("Class::Class() : BaseClass(), Member() {}");
 
   // The TT_Unknown is clearly not binding for the future, please adapt if those
   // tokens get annotated.
   ASSERT_EQ(Tokens.size(), 16u) << Tokens;