[Clang] Add tests and mark as implemented WG14 N2728 (char16_t & char32_t string literals shall be UTF-16 & UTF-32)

[tahonermann]

Summary

This change expands testing of UTF-8, UTF-16, and UTF-32 character and string literals as validation that WG14 N2728 (char16_t & char32_t string literals shall be UTF-16 & UTF-32) has been implemented.

WG14 N2728: https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2728.htm

[tahonermann]

clang/test/Lexer/char-literal.cpp

2–5

C++17 and C2x are added so that UTF-8 character literals are exercised. C++20 is added to exercise the change in type of UTF-8 literals due to char8_t.

48–50

C apparently prefers that programmers use actual control characters rather than naming them via UCNs, even in character and string literals. I know not why, but that is what N3096 6.4.3 (Universal character names) says.

73–99

The unsigned char casts are to work around conversion issues with (signed) char and the change of type to char8_t in C++20 vs C++17.

clang/www/c_status.html

932

As far as I can tell, no changes are needed for Clang to implement N2728; UTF-16 and UTF-32 have been used for char16_t and char32_t literals since their introduction in C11 and C++11, so there is no specific Clang version to mark as a conformance point.

[cor3ntin]

LGTM (modulo nitpicking comment)

clang/test/Lexer/char-literal.cpp

49

I think these tests would be clearer with a different verify tag rather than an ifdef, but it's kinda preexisting so feel free to ignore.

[aaron.ballman]

LGTM!

clang/test/Lexer/char-literal.cpp

49

That'd be a nice cleanup for post-commit though.

[tahonermann]

Thank you @cor3ntin, that was an excellent suggestion; this is much more readable now! I updated the new code to use custom verify tags (I left the existing code alone).

clang/test/Lexer/char-literal.cpp

 // RUN: %clang_cc1 -triple x86_64-apple-darwin -std=c++11 -Wfour-char-constants -fsyntax-only -verify %s
+// RUN: %clang_cc1 -triple x86_64-apple-darwin -std=c++17 -Wfour-char-constants -fsyntax-only -verify %s
+// RUN: %clang_cc1 -triple x86_64-apple-darwin -std=c++20 -Wfour-char-constants -fsyntax-only -verify %s
 // RUN: %clang_cc1 -triple x86_64-apple-darwin -std=c11 -x c -Wfour-char-constants -fsyntax-only -verify %s
+// RUN: %clang_cc1 -triple x86_64-apple-darwin -std=c2x -x c -Wfour-char-constants -fsyntax-only -verify %s
 
 #ifndef __cplusplus
 typedef __WCHAR_TYPE__ wchar_t;
 typedef __CHAR16_TYPE__ char16_t;
 typedef __CHAR32_TYPE__ char32_t;
 #endif
 
 int a = 'ab'; // expected-warning {{multi-character character constant}}

 char32_t n = U'ab'; // expected-error {{Unicode character literals may not contain multiple characters}}
 char16_t o = '👽'; // expected-error {{character too large for enclosing character literal type}}
 
 char16_t p[2] = u"\U0000FFFF";
 char16_t q[2] = u"\U00010000";
 #ifdef __cplusplus
 // expected-error@-2 {{too long}}
 #endif
+
+// UTF-8 character literal code point ranges.
+#if __cplusplus >= 201703L || __STDC_VERSION__ >= 201710L
+_Static_assert(u8'\U00000000' == 0x00, "");
+#ifndef __cplusplus
+// expected-error@-2 {{universal character name refers to a control character}}
+#endif
+_Static_assert(u8'\U0000007F' == 0x7F, "");
+#ifndef __cplusplus
+// expected-error@-2 {{universal character name refers to a control character}}
+#endif
+_Static_assert(u8'\U00000080', "");
+#ifndef __cplusplus
+// expected-error@-2 {{universal character name refers to a control character}}
+#else
+// expected-error@-4 {{character too large for enclosing character literal type}}
+#endif
+_Static_assert((unsigned char)u8'\xFF' == (unsigned char)0xFF, "");
+#endif
+
+// UTF-8 string literal code point ranges.
+_Static_assert(u8"\U00000000"[0] == 0x00, "");
+#ifndef __cplusplus
+// expected-error@-2 {{universal character name refers to a control character}}
+#endif
+_Static_assert(u8"\U0000007F"[0] == 0x7F, "");
+#ifndef __cplusplus
+// expected-error@-2 {{universal character name refers to a control character}}
+#endif
+_Static_assert((unsigned char)u8"\U00000080"[0] == (unsigned char)0xC2, "");
+#ifndef __cplusplus
+// expected-error@-2 {{universal character name refers to a control character}}
+#endif
+_Static_assert((unsigned char)u8"\U00000080"[1] == (unsigned char)0x80, "");
+#ifndef __cplusplus
+// expected-error@-2 {{universal character name refers to a control character}}
+#endif
+_Static_assert((unsigned char)u8"\U000007FF"[0] == (unsigned char)0xDF, "");
+_Static_assert((unsigned char)u8"\U000007FF"[1] == (unsigned char)0xBF, "");
+_Static_assert((unsigned char)u8"\U00000800"[0] == (unsigned char)0xE0, "");
+_Static_assert((unsigned char)u8"\U00000800"[1] == (unsigned char)0xA0, "");
+_Static_assert((unsigned char)u8"\U00000800"[2] == (unsigned char)0x80, "");
+_Static_assert(u8"\U0000D800"[0], ""); // expected-error {{invalid universal character}}
+_Static_assert(u8"\U0000DFFF"[0], ""); // expected-error {{invalid universal character}}
+_Static_assert((unsigned char)u8"\U0000FFFF"[0] == (unsigned char)0xEF, "");
+_Static_assert((unsigned char)u8"\U0000FFFF"[1] == (unsigned char)0xBF, "");
+_Static_assert((unsigned char)u8"\U0000FFFF"[2] == (unsigned char)0xBF, "");
+_Static_assert((unsigned char)u8"\U00010000"[0] == (unsigned char)0xF0, "");
+_Static_assert((unsigned char)u8"\U00010000"[1] == (unsigned char)0x90, "");
+_Static_assert((unsigned char)u8"\U00010000"[2] == (unsigned char)0x80, "");
+_Static_assert((unsigned char)u8"\U00010000"[3] == (unsigned char)0x80, "");
+_Static_assert((unsigned char)u8"\U0010FFFF"[0] == (unsigned char)0xF4, "");
+_Static_assert((unsigned char)u8"\U0010FFFF"[1] == (unsigned char)0x8F, "");
+_Static_assert((unsigned char)u8"\U0010FFFF"[2] == (unsigned char)0xBF, "");
+_Static_assert((unsigned char)u8"\U0010FFFF"[3] == (unsigned char)0xBF, "");
+_Static_assert(u8"\U00110000"[0], ""); // expected-error {{invalid universal character}}
+
+#if !defined(__STDC_UTF_16__)
+#error __STDC_UTF_16__ is not defined.
+#endif
+#if __STDC_UTF_16__ != 1
+#error __STDC_UTF_16__ has the wrong value.
+#endif
+
+// UTF-16 character literal code point ranges.
+_Static_assert(u'\U00000000' == 0x0000, "");
+#ifndef __cplusplus
+// expected-error@-2 {{universal character name refers to a control character}}
+#endif
+_Static_assert(u'\U0000D800', ""); // expected-error {{invalid universal character}}
+_Static_assert(u'\U0000DFFF', ""); // expected-error {{invalid universal character}}
+_Static_assert(u'\U0000FFFF' == 0xFFFF, "");
+_Static_assert(u'\U00010000', ""); // expected-error {{character too large for enclosing character literal type}}
+
+// UTF-16 string literal code point ranges.
+_Static_assert(u"\U00000000"[0] == 0x0000, "");
+#ifndef __cplusplus
+// expected-error@-2 {{universal character name refers to a control character}}
+#endif
+_Static_assert(u"\U0000D800"[0], ""); // expected-error {{invalid universal character}}
+_Static_assert(u"\U0000DFFF"[0], ""); // expected-error {{invalid universal character}}
+_Static_assert(u"\U0000FFFF"[0] == 0xFFFF, "");
+_Static_assert(u"\U00010000"[0] == 0xD800, "");
+_Static_assert(u"\U00010000"[1] == 0xDC00, "");
+_Static_assert(u"\U0010FFFF"[0] == 0xDBFF, "");
+_Static_assert(u"\U0010FFFF"[1] == 0xDFFF, "");
+_Static_assert(u"\U00110000"[0], ""); // expected-error {{invalid universal character}}
+
+#if !defined(__STDC_UTF_32__)
+#error __STDC_UTF_32__ is not defined.
+#endif
+#if __STDC_UTF_32__ != 1
+#error __STDC_UTF_32__ has the wrong value.
+#endif
+
+// UTF-32 character literal code point ranges.
+_Static_assert(U'\U00000000' == 0x00000000, "");
+#ifndef __cplusplus
+// expected-error@-2 {{universal character name refers to a control character}}
+#endif
+_Static_assert(U'\U0010FFFF' == 0x0010FFFF, "");
+_Static_assert(U'\U00110000', ""); // expected-error {{invalid universal character}}
+
+// UTF-32 string literal code point ranges.
+_Static_assert(U"\U00000000"[0] == 0x00000000, "");
+#ifndef __cplusplus
+// expected-error@-2 {{universal character name refers to a control character}}
+#endif
+_Static_assert(U"\U0000D800"[0], ""); // expected-error {{invalid universal character}}
+_Static_assert(U"\U0000DFFF"[0], ""); // expected-error {{invalid universal character}}
+_Static_assert(U"\U0010FFFF"[0] == 0x0010FFFF, "");
+_Static_assert(U"\U00110000"[0], ""); // expected-error {{invalid universal character}}

clang/www/c_status.html

</tr>
     <tr>
       <td>Numerically equal</td>
       <td><a href="https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2716.htm">N2716</a></td>
       <td class="full" align="center">Yes</td>
     </tr>
     <tr>
       <td>char16_t & char32_t string literals shall be UTF-16 & UTF-32</td>
       <td><a href="https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2728.htm">N2728</a></td>
-      <td class="unknown" align="center">Unknown</td>
+      <td class="full" align="center">Yes</td>
     </tr>
     <tr>
       <td>IEC 60559 binding</td>
       <td><a href="https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2749.pdf">N2749</a></td>
       <td class="unknown" align="center">Unknown</td>
     </tr>
     <tr>
       <td>__has_include for C</td>