[clang-format] Formatter does not handle c++11 string literal prefix with stringize #

[MyDeveloperDay]

Summary

https://github.com/llvm/llvm-project/issues/27740

Ensure

#define _u(str) u#str
#define _u(str) u8#str
#define _u(str) U#str

behave the same as

#define _u(str) L#str

when formatted, ensure clang-format follows the conventions for L u U u8

https://docs.microsoft.com/en-us/cpp/cpp/string-and-character-literals-cpp?redirectedfrom=MSDN&view=msvc-170

Fixes #27740

[curdeius]

When at it, should we also take care of LR"(string)", R, uR, u8R and UR? Cf. https://en.cppreference.com/w/cpp/language/string_literal
From MS doc:

// Raw string literals containing unescaped \ and "
auto R0 =   R"("Hello \ world")"; // const char*
auto R1 = u8R"("Hello \ world")"; // const char* before C++20, encoded as UTF-8,
                                  // const char8_t* in C++20
auto R2 =  LR"("Hello \ world")"; // const wchar_t*
auto R3 =  uR"("Hello \ world")"; // const char16_t*, encoded as UTF-16
auto R4 =  UR"("Hello \ world")"; // const char32_t*, encoded as UTF-32

[MyDeveloperDay]

In D115938#3199832, @curdeius wrote:

When at it, should we also take care of LR"(string)", R, uR, u8R and UR? Cf. https://en.cppreference.com/w/cpp/language/string_literal
From MS doc:

// Raw string literals containing unescaped \ and "
auto R0 =   R"("Hello \ world")"; // const char*
auto R1 = u8R"("Hello \ world")"; // const char* before C++20, encoded as UTF-8,
                                  // const char8_t* in C++20
auto R2 =  LR"("Hello \ world")"; // const wchar_t*
auto R3 =  uR"("Hello \ world")"; // const char16_t*, encoded as UTF-16
auto R4 =  UR"("Hello \ world")"; // const char32_t*, encoded as UTF-32

I did think about that but I was thinking about how would the calling side work?

#define MyRawString(str) R#str


void foo()
{
  const char *s = MyRawString("(" Hello \ world ")")
}

I think trying to pass the raw string into the macro confuses it no? this was why I left them out for now. Unless someone can give me an example of how it might work.

Microsoft (R) C/C++ Optimizing Compiler Version 19.29.30137 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

test12.cpp
test12.cpp(5): error C3513: '\': unsupported raw string literal delimiter character
test12.cpp(7): error C3516: unexpected end-of-file found while processing the raw string literal; delimiter sequence '")' was not matched
test12.cpp(5): note: start of raw string literal
test12.cpp(7): fatal error C1903: unable to recover from previous error(s); stopping compilation

[curdeius]

You shouldn't have added the outer quotes as #str adds them.
This works:

// clang-format off
#define MyRawString(str) R#str

void foo()
{
    const auto * s1 = MyRawString((" Hello \ world "));
    const auto * s2 = MyRawString(abc(" Hello \ world ")abc);
}

[MyDeveloperDay]

Add the raw string literal cases

[curdeius]

LGTM.

clang/lib/Format/TokenAnnotator.cpp

bool TokenAnnotator::spaceRequiredBetween(const AnnotatedLine &Line,
   if ((Left.isOneOf(tok::identifier, tok::greater, tok::r_square,
                     tok::r_paren) ||
        Left.isSimpleTypeSpecifier()) &&
       Right.is(tok::l_brace) && Right.getNextNonComment() &&
       Right.isNot(BK_Block))
     return false;
   if (Left.is(tok::period) || Right.is(tok::period))
     return false;
-  if (Right.is(tok::hash) && Left.is(tok::identifier) && Left.TokenText == "L")
+  // u#str, U#str, L#str, u8#str
+  // uR#str, UR#str, LR#str, u8R#str
+  if (Right.is(tok::hash) && Left.is(tok::identifier) &&
+      (Left.TokenText == "L" || Left.TokenText == "u" ||
+       Left.TokenText == "U" || Left.TokenText == "u8" ||
+       Left.TokenText == "LR" || Left.TokenText == "uR" ||
+       Left.TokenText == "UR" || Left.TokenText == "u8R"))
     return false;
   if (Left.is(TT_TemplateCloser) && Left.MatchingParen &&
       Left.MatchingParen->Previous &&
       (Left.MatchingParen->Previous->is(tok::period) ||
        Left.MatchingParen->Previous->is(tok::coloncolon)))
     // Java call to generic function with explicit type:
     // A.<B<C<...>>>DoSomething();
     // A::<B<C<...>>>DoSomething();  // With a Java 8 method reference.

clang/unittests/Format/FormatTest.cpp

TEST_F(FormatTest, MacroDefinitionInsideStatement) {
   EXPECT_EQ("int x,\n"
             "#define A\n"
             "    y;",
             format("int x,\n#define A\ny;"));
 }
 
 TEST_F(FormatTest, HashInMacroDefinition) {
   EXPECT_EQ("#define A(c) L#c", format("#define A(c) L#c", getLLVMStyle()));
+  EXPECT_EQ("#define A(c) u#c", format("#define A(c) u#c", getLLVMStyle()));
+  EXPECT_EQ("#define A(c) U#c", format("#define A(c) U#c", getLLVMStyle()));
+  EXPECT_EQ("#define A(c) u8#c", format("#define A(c) u8#c", getLLVMStyle()));
+  EXPECT_EQ("#define A(c) LR#c", format("#define A(c) LR#c", getLLVMStyle()));
+  EXPECT_EQ("#define A(c) uR#c", format("#define A(c) uR#c", getLLVMStyle()));
+  EXPECT_EQ("#define A(c) UR#c", format("#define A(c) UR#c", getLLVMStyle()));
+  EXPECT_EQ("#define A(c) u8R#c", format("#define A(c) u8R#c", getLLVMStyle()));
   verifyFormat("#define A \\\n  b #c;", getLLVMStyleWithColumns(11));
   verifyFormat("#define A  \\\n"
                "  {        \\\n"
                "    f(#c); \\\n"
                "  }",
                getLLVMStyleWithColumns(11));
 
   verifyFormat("#define A(X)         \\\n"