[Clang] Add support for Unicode identifiers (UAX31) in C2x mode.

[cor3ntin]

Summary

This implements
N2836 Identifier Syntax using Unicode Standard Annex 31.

The feature was already implemented for C++,
and the semantics are the same.

Unlike C++ there was, afaict, no decision to
backport the feature in older languages mode,
so C17 and earlier are not modified and the
code point tables for these language modes are conserved.

[cor3ntin]

Update a comment

[aaron.ballman]

LGTM aside from a typo fix in a comment. One thing worth thinking about is whether it would be worth it to have a diagnostic for valid C code in older modes that will change behavior in C2x due to this feature, but that can be done as follow-up work.

clang/lib/Lex/Lexer.cpp

1469

[cor3ntin]

Fix typo

[tahonermann]

Apologies for the much delayed review (I know this already landed).

I added a comment, but I don't know if there is actually an issue to be fixed. This looks good to me otherwise.

clang/lib/Lex/Lexer.cpp

1490–1492

Perhaps this should permit $ when LangOpts.DollarIdents is true. However, my attempts to produce a test case that illustrates such a need failed. I don't know why; I guess isAllowedInitiallyIDChar() isn't called for all identifiers? Or there is special handling of LangOpts.DollarIdents somewhere? I wonder if that might imply an issue elsewhere; such as a missing call to isAllowedInitiallyIDChar() somewhere?

I did verify that '$' (U+003F) is not in XID_Start: https://util.unicode.org/UnicodeJsps/list-unicodeset.jsp?a=[:XID_Start=Yes:]

[cor3ntin]

clang/lib/Lex/Lexer.cpp

1490–1492

Thanks for the review. I think checking for underscore here is actually the issue, it's most likely never true.
isAllowedInitiallyIDChar is never called on ascii codepoints as the fast path uses isAsciiIdentifierStart - which is where $ is checked for.

(LexIdentifierContinue handles ascii characters too and correctly deal with $ )

clang/docs/ReleaseNotes.rst

 - Implemented the ``*_WIDTH`` macros to complete support for
   `WG14 N2412 Two's complement sign representation for C2x <https://www9.open-std.org/jtc1/sc22/wg14/www/docs/n2412.pdf>`_.
 - Implemented `WG14 N2418 Adding the u8 character prefix <http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2418.pdf>`_.
 - Removed support for implicit function declarations. This was a C89 feature
   that was removed in C99, but cannot be supported in C2x because it requires
   support for functions without prototypes, which no longer exist in C2x.
 - Implemented `WG14 N2841 No function declarators without prototypes <https://www9.open-std.org/jtc1/sc22/wg14/www/docs/n2841.htm>`_
   and `WG14 N2432 Remove support for function definitions with identifier lists <https://www9.open-std.org/jtc1/sc22/wg14/www/docs/n2432.pdf>`_.
+- Implemented `WG14 N2836 Identifier Syntax using Unicode Standard Annex 31 <https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2836.pdf>`_.
 
 C++ Language Changes in Clang
 -----------------------------
 
 - Improved ``-O0`` code generation for calls to ``std::move``, ``std::forward``,
   ``std::move_if_noexcept``, ``std::addressof``, and ``std::as_const``. These
   are now treated as compiler builtins and implemented directly, rather than
   instantiating the definition from the standard library.

clang/lib/Lex/Lexer.cpp

static bool isUnicodeWhitespace(uint32_t Codepoint) {
   return UnicodeWhitespaceChars.contains(Codepoint);
 }
 
 static bool isAllowedIDChar(uint32_t C, const LangOptions &LangOpts) {
   if (LangOpts.AsmPreprocessor) {
     return false;
   } else if (LangOpts.DollarIdents && '$' == C) {
     return true;
-  } else if (LangOpts.CPlusPlus) {
+  } else if (LangOpts.CPlusPlus || LangOpts.C2x) {
     // A non-leading codepoint must have the XID_Continue property.
     // XIDContinueRanges doesn't contains characters also in XIDStartRanges,
     // so we need to check both tables.
     // '_' doesn't have the XID_Continue property but is allowed in C++.
     static const llvm::sys::UnicodeCharSet XIDStartChars(XIDStartRanges);
     static const llvm::sys::UnicodeCharSet XIDContinueChars(XIDContinueRanges);
     return C == '_' || XIDStartChars.contains(C) ||
            XIDContinueChars.contains(C);
   } else if (LangOpts.C11) {
     static const llvm::sys::UnicodeCharSet C11AllowedIDChars(
         C11AllowedIDCharRanges);
     return C11AllowedIDChars.contains(C);
   } else {
     static const llvm::sys::UnicodeCharSet C99AllowedIDChars(
         C99AllowedIDCharRanges);
     return C99AllowedIDChars.contains(C);
   }
 }
 
 static bool isAllowedInitiallyIDChar(uint32_t C, const LangOptions &LangOpts) {
   if (LangOpts.AsmPreprocessor) {
     return false;
   }
-  if (LangOpts.CPlusPlus) {
+  if (LangOpts.CPlusPlus || LangOpts.C2x) {
     static const llvm::sys::UnicodeCharSet XIDStartChars(XIDStartRanges);
     // '_' doesn't have the XID_Start property but is allowed in C++.
     return C == '_' || XIDStartChars.contains(C);
   }
   if (!isAllowedIDChar(C, LangOpts))
     return false;
   if (LangOpts.C11) {
     static const llvm::sys::UnicodeCharSet C11DisallowedInitialIDChars(
         C11DisallowedInitialIDCharRanges);
     return !C11DisallowedInitialIDChars.contains(C);
   }

clang/test/Lexer/unicode.c

 // RUN: %clang_cc1 -fsyntax-only -verify -x c -std=c11 %s
-// RUN: %clang_cc1 -fsyntax-only -verify -x c++ -std=c++11 %s
+// RUN: %clang_cc1 -fsyntax-only -verify=expected,c2x -x c -std=c2x %s
+// RUN: %clang_cc1 -fsyntax-only -verify=expected,cxx -x c++ -std=c++11 %s
 // RUN: %clang_cc1 -std=c99 -E -DPP_ONLY=1 %s | FileCheck %s --strict-whitespace
 // RUN: %clang_cc1 -E -DPP_ONLY=1 %s | FileCheck %s --strict-whitespace
 
 // This file contains Unicode characters; please do not "fix" them!
 
 extern int!x; // expected-warning {{treating Unicode character as whitespace}}
 extern int　x; // expected-warning {{treating Unicode character as whitespace}}
 

 CHECK : The preprocessor should not complain about Unicode characters like ©.
 #endif
 
         int _;
 
 extern int X\UAAAAAAAA; // expected-error {{not allowed in an identifier}}
 int Y = '\UAAAAAAAA'; // expected-error {{invalid universal character}}
 
-#ifdef __cplusplus
+#if defined(__cplusplus) || (defined(__STDC_VERSION__) && __STDC_VERSION__ >= 202000L)
 
 extern int ༀ;
 extern int 𑩐;
 extern int 𐠈;
 extern int ꙮ;
 extern int  \u1B4C;     // BALINESE LETTER ARCHAIC JNYA - Added in Unicode 14
 extern int  \U00016AA2; // TANGSA LETTER GA - Added in Unicode 14
 extern int _\N{TANGSA LETTER GA};
 extern int _\N{TANGSALETTERGA}; // expected-error {{'TANGSALETTERGA' is not a valid Unicode character name}} \
                                 // expected-note {{characters names in Unicode escape sequences are sensitive to case and whitespace}}
 
 
 
 // This character doesn't have the XID_Start property
-extern int  \U00016AC0; // TANGSA DIGIT ZERO  // expected-error {{expected unqualified-id}}
+extern int  \U00016AC0; // TANGSA DIGIT ZERO  // cxx-error {{expected unqualified-id}} \
+                                              // c2x-error {{expected identifier or '('}}
 
 extern int 🌹; // expected-error {{unexpected character <U+1F339>}} \
                   expected-warning {{declaration does not declare anything}}
 
 extern int 👷; // expected-error {{unexpected character <U+1F477>}} \
                   expected-warning {{declaration does not declare anything}}
 
 extern int 👷‍♀; // expected-warning {{declaration does not declare anything}} \

clang/www/c_status.html

</tr>
     <tr>
       <td>Make assert() macro user friendly for C and C++ v2</td>
       <td><a href="https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2829.htm">N2829</a></td>
       <td class="none" align="center">No</td>
     </tr>
     <tr>
       <td>Identifier Syntax using Unicode Standard Annex 31</td>
       <td><a href="https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2836.pdf">N2836</a></td>
-      <td class="none" align="center">No</td>
+      <td class="unreleased" align="center">Clang 15</td>
     </tr>
     <tr>
       <td>No function declarators without prototypes</td>
       <td><a href="https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2841.htm">N2841</a></td>
       <td class="unreleased" align="center">Clang 15</td>
     </tr>
     <tr>
       <td>Remove default argument promotions for _FloatN types</td>