[Lexer] Allow UCN for dollar symbol '\u0024' in identifiers when using -fdollars-in-identifiers flag.

[s.egerton]

Summary

Previously, the -fdollars-in-identifiers flag allows the '$' symbol to be used
in an identifier but the universal character name equivalent '\u0024' is not
allowed.
This patch changes this, so that \u0024 is valid in identifiers.

[rsmith]

This seems reasonable to me, and matches GCC 10's behavior; $ is not in the basic source character set, so encoding it with a UCN should be permitted.

clang/lib/Lex/Lexer.cpp

void Lexer::SetByteOffset(unsigned Offset, bool StartOfLine) {
   // unexpanded line, or the first token on the expanded line.
   IsAtStartOfLine = StartOfLine;
   IsAtPhysicalStartOfLine = StartOfLine;
 }
 
 static bool isAllowedIDChar(uint32_t C, const LangOptions &LangOpts) {
   if (LangOpts.AsmPreprocessor) {
     return false;
+  } else if (LangOpts.DollarIdents && '$' == C) {
+    return true;
   } else if (LangOpts.CPlusPlus11 || LangOpts.C11) {
     static const llvm::sys::UnicodeCharSet C11AllowedIDChars(
         C11AllowedIDCharRanges);
     return C11AllowedIDChars.contains(C);
   } else if (LangOpts.CPlusPlus) {
     static const llvm::sys::UnicodeCharSet CXX03AllowedIDChars(
         CXX03AllowedIDCharRanges);
     return CXX03AllowedIDChars.contains(C);

clang/test/Preprocessor/ucn-pp-identifier.c

 #if \U0001000  // expected-warning{{incomplete universal character name; treating as '\' followed by identifier}} expected-error {{invalid token at start of a preprocessor expression}}
 #endif
 
 // Make sure we reject disallowed UCNs
 #define \ufffe // expected-error {{macro name must be an identifier}}
 #define \U10000000  // expected-error {{macro name must be an identifier}}
 #define \u0061  // expected-error {{character 'a' cannot be specified by a universal character name}} expected-error {{macro name must be an identifier}}
 
-// FIXME: Not clear what our behavior should be here; \u0024 is "$".
-#define a\u0024  // expected-warning {{whitespace}}
+#define a\u0024
 
 #if \u0110 // expected-warning {{is not defined, evaluates to 0}}
 #endif
 
 
 #define \u0110 1 / 0
 #if \u0110 // expected-error {{division by zero in preprocessor expression}}
 #endif