Cleanup identifier parsing.

[cor3ntin]

Summary

• Rename methods to clearly signal when they only deal with ASCII
• Simplify the parsing of identifier
• use start/continue instead of head/body for consistency with

Unicode terminology

[cor3ntin]

Spell ASCII in upper case

[cor3ntin]

Looks better in lower case after all

[cor3ntin]

Remove file committed accidentally

[cor3ntin]

@aaron.ballman Let me know what you think.
The PR does not contain new behavior, only renames and refactor the function lexing identifiers. I ran the build a few times and did not measure performance differences on my system. The code should behave exactly the same except with one loop instead of 3.
I also moved the 2 identifier lexing functions near one another to make it easier to understand.

This makes it apparent that some places in tools, maybe header names or module parsing too only check for ASCII identifiers when they may want to check for Unicode, This is not addressed here.

[aaron.ballman]

In general, I'm in favor of these changes. They help identify (pun *totally* intended) where we're improperly expecting ASCII identifiers in places, which can hopefully be addressed in follow-up work. @rsmith, do you have any concerns with this direction?

Can you remove the [WIP] from the title so it's clear that this is no longer in progress? Also, I'd recommend slapping an NFC in the title somewhere to make it clear there's no functional changes intended.

clang/include/clang/Lex/Lexer.h

702–706

Should this be LexUnicodeIdentifierContinue()? If so, perhaps it can also be moved up to line 578 so it's near the "start" function?

Or does this function handle both Unicode and ASCII identifiers? If so, the comments could probably be updated.

clang/lib/Lex/Lexer.cpp

1758

Is the comment here still accurate? Might be worth rewriting in prose rather than regex?

1761

1766

1784–1789

1789

[cor3ntin]

clang/include/clang/Lex/Lexer.h

702–706

This handles all identifiers - after the first codepoint has been parsed - Which comment are you referring to?

clang/lib/Lex/Lexer.cpp

1758

I don't think the comment was accurate before, I'll find somehing better!

[aaron.ballman]

clang/include/clang/Lex/Lexer.h

701

Something like this then?

[cor3ntin]

Fix comments following Aaron's feedback, remove
braces deemed unecessary by the guidelines

[cor3ntin]

clang/include/clang/Lex/Lexer.h

702–706

I kept the comment as is - because it applies to all function underneath, but added a comment in the definition in the cpp

[aaron.ballman]

The changes LGTM, modulo a commenting request. Let's wait a bit before landing in case @rsmith has concerns.

clang/include/clang/Lex/Lexer.h

702–706

It'd be a bit better to add the comments here; the header file is where users often first find some functionality, so that's where the comments explaining the high-level functionality of how to call the function are most useful. Also, IDEs often pick up comments "attached to" declarations and display them automatically for users which is a nice benefit.

[cor3ntin]

Move a comment to the header file per Aaron request

[aaron.ballman]

Thank you for the cleanup! I've landed in 601102d282d5e9a1429fea52ee17303aec8a7c10.

clang-tools-extra/clang-include-fixer/IncludeFixer.cpp

auto ExtendNestedNameSpecifier = [this](CharSourceRange Range) {
     //      unknown, last callback
     //                  ^~~~~~~~~~~
     //                  no callback
     //
     // With the extension we get the full nested name specifier including
     // parent_path.
     // FIXME: Don't rely on source text.
     const char *End = Source.end();
-    while (isIdentifierBody(*End) || *End == ':')
+    while (isAsciiIdentifierContinue(*End) || *End == ':')
       ++End;
 
     return std::string(Source.begin(), End);
   };
 
   /// If we have a scope specification, use that to get more precise results.
   std::string QueryString;
   tooling::Range SymbolRange;

clang-tools-extra/clang-tidy/google/IntegerTypesCheck.cpp

default:
     return;
   }
 
   // We allow "unsigned short port" as that's reasonably common and required by
   // the sockets API.
   const StringRef Port = "unsigned short port";
   const char *Data = Result.SourceManager->getCharacterData(Loc);
   if (!std::strncmp(Data, Port.data(), Port.size()) &&
-      !isIdentifierBody(Data[Port.size()]))
+      !isAsciiIdentifierContinue(Data[Port.size()]))
     return;
 
   std::string Replacement =
       ((IsSigned ? SignedTypePrefix : UnsignedTypePrefix) + Twine(Width) +
        TypeSuffix)
           .str();
 
   // We don't add a fix-it as changing the type can easily break code,

clang-tools-extra/clang-tidy/utils/RenamerClangTidyCheck.cpp

if (const auto *Decl = Result.Nodes.getNodeAs<NamedDecl>("decl")) {
     const IdentifierTable &Idents = Decl->getASTContext().Idents;
     auto CheckNewIdentifier = Idents.find(Info.Fixup);
     if (CheckNewIdentifier != Idents.end()) {
       const IdentifierInfo *Ident = CheckNewIdentifier->second;
       if (Ident->isKeyword(getLangOpts()))
         Failure.FixStatus = ShouldFixStatus::ConflictsWithKeyword;
       else if (Ident->hasMacroDefinition())
         Failure.FixStatus = ShouldFixStatus::ConflictsWithMacroDefinition;
-    } else if (!isValidIdentifier(Info.Fixup)) {
+    } else if (!isValidAsciiIdentifier(Info.Fixup)) {
       Failure.FixStatus = ShouldFixStatus::FixInvalidIdentifier;
     }
 
     Failure.Info = std::move(Info);
     addUsage(Decl, Range);
   }
 }
 

clang-tools-extra/clangd/CodeComplete.cpp

 
 CompletionPrefix guessCompletionPrefix(llvm::StringRef Content,
                                        unsigned Offset) {
   assert(Offset <= Content.size());
   StringRef Rest = Content.take_front(Offset);
   CompletionPrefix Result;
 
   // Consume the unqualified name. We only handle ASCII characters.
-  // isIdentifierBody will let us match "0invalid", but we don't mind.
-  while (!Rest.empty() && isIdentifierBody(Rest.back()))
+  // isAsciiIdentifierContinue will let us match "0invalid", but we don't mind.
+  while (!Rest.empty() && isAsciiIdentifierContinue(Rest.back()))
     Rest = Rest.drop_back();
   Result.Name = Content.slice(Rest.size(), Offset);
 
   // Consume qualifiers.
   while (Rest.consume_back("::") && !Rest.endswith(":")) // reject ::::
-    while (!Rest.empty() && isIdentifierBody(Rest.back()))
+    while (!Rest.empty() && isAsciiIdentifierContinue(Rest.back()))
       Rest = Rest.drop_back();
   Result.Qualifier =
       Content.slice(Rest.size(), Result.Name.begin() - Content.begin());
 
   return Result;
 }
 
 CodeCompleteResult codeComplete(PathRef FileName, Position Pos,
if (Content.endswith(".") || Content.endswith("->") || Content.endswith("::"))
     return true;
   // Complete after `#include <` and #include `<foo/`.
   if ((Content.endswith("<") || Content.endswith("\"") ||
        Content.endswith("/")) &&
       isIncludeFile(Content))
     return true;
 
   // Complete words. Give non-ascii characters the benefit of the doubt.
-  return !Content.empty() &&
-         (isIdentifierBody(Content.back()) || !llvm::isASCII(Content.back()));
+  return !Content.empty() && (isAsciiIdentifierContinue(Content.back()) ||
+                              !llvm::isASCII(Content.back()));
 }
 
 } // namespace clangd
 } // namespace clang

clang-tools-extra/clangd/SourceCode.cpp

llvm::Optional<SpelledWord> SpelledWord::touching(SourceLocation SpelledLoc,
   FileID File;
   unsigned Offset;
   std::tie(File, Offset) = SM.getDecomposedLoc(SpelledLoc);
   bool Invalid = false;
   llvm::StringRef Code = SM.getBufferData(File, &Invalid);
   if (Invalid)
     return llvm::None;
   unsigned B = Offset, E = Offset;
-  while (B > 0 && isIdentifierBody(Code[B - 1]))
+  while (B > 0 && isAsciiIdentifierContinue(Code[B - 1]))
     --B;
-  while (E < Code.size() && isIdentifierBody(Code[E]))
+  while (E < Code.size() && isAsciiIdentifierContinue(Code[E]))
     ++E;
   if (B == E)
     return llvm::None;
 
   SpelledWord Result;
   Result.Location = SM.getComposedLoc(File, B);
   Result.Text = Code.slice(B, E);
   Result.LikelyIdentifier =

clang-tools-extra/clangd/refactor/Rename.cpp

 
 static bool mayBeValidIdentifier(llvm::StringRef Ident) {
   assert(llvm::json::isUTF8(Ident));
   if (Ident.empty())
     return false;
   // We don't check all the rules for non-ascii characters (most are allowed).
   bool AllowDollar = true; // lenient
   if (llvm::isASCII(Ident.front()) &&
-      !isIdentifierHead(Ident.front(), AllowDollar))
+      !isAsciiIdentifierStart(Ident.front(), AllowDollar))
     return false;
   for (char C : Ident) {
-    if (llvm::isASCII(C) && !isIdentifierBody(C, AllowDollar))
+    if (llvm::isASCII(C) && !isAsciiIdentifierContinue(C, AllowDollar))
       return false;
   }
   return true;
 }
 
 // Check if we can rename the given RenameDecl into NewName.
 // Return details if the rename would produce a conflict.
 llvm::Optional<InvalidName> checkName(const NamedDecl &RenameDecl,

clang/include/clang/Basic/CharInfo.h

 
 LLVM_READNONE inline bool isASCII(unsigned char c) { return c <= 127; }
 
 /// Returns true if this is an ASCII character.
 LLVM_READNONE inline bool isASCII(uint32_t c) { return c <= 127; }
 
 /// Returns true if this is a valid first character of a C identifier,
 /// which is [a-zA-Z_].
-LLVM_READONLY inline bool isIdentifierHead(unsigned char c,
+LLVM_READONLY inline bool isAsciiIdentifierStart(unsigned char c,
                                                  bool AllowDollar = false) {
   using namespace charinfo;
   if (InfoTable[c] & (CHAR_UPPER|CHAR_LOWER|CHAR_UNDER))
     return true;
   return AllowDollar && c == '$';
 }
 
 /// Returns true if this is a body character of a C identifier,
 /// which is [a-zA-Z0-9_].
-LLVM_READONLY inline bool isIdentifierBody(unsigned char c,
+LLVM_READONLY inline bool isAsciiIdentifierContinue(unsigned char c,
                                                     bool AllowDollar = false) {
   using namespace charinfo;
   if (InfoTable[c] & (CHAR_UPPER|CHAR_LOWER|CHAR_DIGIT|CHAR_UNDER))
     return true;
   return AllowDollar && c == '$';
 }
 
 /// Returns true if this character is horizontal ASCII whitespace:
 /// ' ', '\\t', '\\f', '\\v'.
LLVM_READONLY inline char toUppercase(char c) {
   return c;
 }
 
 
 /// Return true if this is a valid ASCII identifier.
 ///
 /// Note that this is a very simple check; it does not accept UCNs as valid
 /// identifier characters.
-LLVM_READONLY inline bool isValidIdentifier(StringRef S,
+LLVM_READONLY inline bool isValidAsciiIdentifier(StringRef S,
                                                  bool AllowDollar = false) {
-  if (S.empty() || !isIdentifierHead(S[0], AllowDollar))
+  if (S.empty() || !isAsciiIdentifierStart(S[0], AllowDollar))
     return false;
 
   for (StringRef::iterator I = S.begin(), E = S.end(); I != E; ++I)
-    if (!isIdentifierBody(*I, AllowDollar))
+    if (!isAsciiIdentifierContinue(*I, AllowDollar))
       return false;
 
   return true;
 }
 
 } // end namespace clang
 
 #endif

clang/include/clang/Lex/Lexer.h

public:
   /// invalid.
   static SourceLocation findLocationAfterToken(SourceLocation loc,
                                          tok::TokenKind TKind,
                                          const SourceManager &SM,
                                          const LangOptions &LangOpts,
                                          bool SkipTrailingWhitespaceAndNewLine);
 
   /// Returns true if the given character could appear in an identifier.
-  static bool isIdentifierBodyChar(char c, const LangOptions &LangOpts);
+  static bool isAsciiIdentifierContinueChar(char c,
+                                            const LangOptions &LangOpts);
 
   /// Checks whether new line pointed by Str is preceded by escape
   /// sequence.
   static bool isNewLineEscaped(const char *BufferStart, const char *Str);
 
   /// getCharAndSizeNoWarn - Like the getCharAndSize method, but does not ever
   /// emit a warning.
   static inline char getCharAndSizeNoWarn(const char *Ptr, unsigned &Size,
private:
 
   /// LexTokenInternal - Internal interface to lex a preprocessing token. Called
   /// by Lex.
   ///
   bool LexTokenInternal(Token &Result, bool TokAtPhysicalStartOfLine);
 
   bool CheckUnicodeWhitespace(Token &Result, uint32_t C, const char *CurPtr);
 
-  /// Given that a token begins with the Unicode character \p C, figure out
-  /// what kind of token it is and dispatch to the appropriate lexing helper
-  /// function.
-  bool LexUnicode(Token &Result, uint32_t C, const char *CurPtr);
+  bool LexUnicodeIdentifierStart(Token &Result, uint32_t C, const char *CurPtr);
 
   /// FormTokenWithChars - When we lex a token, we have identified a span
   /// starting at BufferPtr, going to TokEnd that forms the token.  This method
   /// takes that range and assigns it to the token as its location and size.  In
   /// addition, since tokens cannot overlap, this also updates BufferPtr to be
   /// TokEnd.
   void FormTokenWithChars(Token &Result, const char *TokEnd,
                           tok::TokenKind Kind) {
private:
 
   void SetByteOffset(unsigned Offset, bool StartOfLine);
 
   void PropagateLineStartLeadingSpaceInfo(Token &Result);
 
   const char *LexUDSuffix(Token &Result, const char *CurPtr,
                           bool IsStringLiteral);
 
   // Helper functions to lex the remainder of a token of the specific type.
-  bool LexIdentifier         (Token &Result, const char *CurPtr);
+
+  // This function handles both ASCII and Unicode identifiers after
+  // the first codepoint of the identifyier has been parsed.
+  bool LexIdentifierContinue(Token &Result, const char *CurPtr);
+
   bool LexNumericConstant    (Token &Result, const char *CurPtr);
   bool LexStringLiteral      (Token &Result, const char *CurPtr,
                               tok::TokenKind Kind);
   bool LexRawStringLiteral   (Token &Result, const char *CurPtr,
                               tok::TokenKind Kind);
   bool LexAngledStringLiteral(Token &Result, const char *CurPtr);
   bool LexCharConstant       (Token &Result, const char *CurPtr,
                               tok::TokenKind Kind);

clang/lib/ARCMigrate/ObjCMT.cpp

static bool AttributesMatch(const Decl *Decl1, const Decl *Decl2,
   bool match = MatchTwoAttributeLists(Attrs1, Attrs2, AvailabilityArgsMatch);
   if (match && (Attrs2.size() > Attrs1.size()))
     return MatchTwoAttributeLists(Attrs2, Attrs1, AvailabilityArgsMatch);
   return match;
 }
 
 static bool IsValidIdentifier(ASTContext &Ctx,
                               const char *Name) {
-  if (!isIdentifierHead(Name[0]))
+  if (!isAsciiIdentifierStart(Name[0]))
     return false;
   std::string NameString = Name;
   NameString[0] = toLowercase(NameString[0]);
   IdentifierInfo *II = &Ctx.Idents.get(NameString);
   return II->getTokenID() ==  tok::identifier;
 }
 
 bool ObjCMigrateASTConsumer::migrateProperty(ASTContext &Ctx,

clang/lib/ARCMigrate/TransUnbridgedCasts.cpp

if (Kind == OBC_Bridge || !Pass.CFBridgingFunctionsDefined()) {
       assert(Kind == OBC_BridgeTransfer || Kind == OBC_BridgeRetained);
       SmallString<32> BridgeCall;
 
       Expr *WrapE = E->getSubExpr();
       SourceLocation InsertLoc = WrapE->getBeginLoc();
 
       SourceManager &SM = Pass.Ctx.getSourceManager();
       char PrevChar = *SM.getCharacterData(InsertLoc.getLocWithOffset(-1));
-      if (Lexer::isIdentifierBodyChar(PrevChar, Pass.Ctx.getLangOpts()))
+      if (Lexer::isAsciiIdentifierContinueChar(PrevChar,
+                                               Pass.Ctx.getLangOpts()))
         BridgeCall += ' ';
 
       if (Kind == OBC_BridgeTransfer)
         BridgeCall += "CFBridgingRelease";
       else
         BridgeCall += "CFBridgingRetain";
 
       if (isa<ParenExpr>(WrapE)) {

clang/lib/AST/MicrosoftMangle.cpp

void MicrosoftMangleContextImpl::mangleStringLiteral(const StringLiteral *SL,
   // the constituent bytes.
   auto MangleByte = [&Mangler](char Byte) {
     // There are five different manglings for characters:
     // - [a-zA-Z0-9_$]: A one-to-one mapping.
     // - ?[a-z]: The range from \xe1 to \xfa.
     // - ?[A-Z]: The range from \xc1 to \xda.
     // - ?[0-9]: The set of [,/\:. \n\t'-].
     // - ?$XX: A fallback which maps nibbles.
-    if (isIdentifierBody(Byte, /*AllowDollar=*/true)) {
+    if (isAsciiIdentifierContinue(Byte, /*AllowDollar=*/true)) {
       Mangler.getStream() << Byte;
     } else if (isLetter(Byte & 0x7f)) {
       Mangler.getStream() << '?' << static_cast<char>(Byte & 0x7f);
     } else {
       const char SpecialChars[] = {',', '/',  '\\', ':',  '.',
                                    ' ', '\n', '\t', '\'', '-'};
       const char *Pos = llvm::find(SpecialChars, Byte);
       if (Pos != std::end(SpecialChars)) {

clang/lib/Basic/Module.cpp

 template<typename InputIter>
 static void printModuleId(raw_ostream &OS, InputIter Begin, InputIter End,
                           bool AllowStringLiterals = true) {
   for (InputIter It = Begin; It != End; ++It) {
     if (It != Begin)
       OS << ".";
 
     StringRef Name = getModuleNameFromComponent(*It);
-    if (!AllowStringLiterals || isValidIdentifier(Name))
+    if (!AllowStringLiterals || isValidAsciiIdentifier(Name))
       OS << Name;
     else {
       OS << '"';
       OS.write_escaped(Name);
       OS << '"';
     }
   }
 }

clang/lib/Edit/EditedSource.cpp

bool EditedSource::commit(const Commit &commit) {
 
   return true;
 }
 
 // Returns true if it is ok to make the two given characters adjacent.
 static bool canBeJoined(char left, char right, const LangOptions &LangOpts) {
   // FIXME: Should use TokenConcatenation to make sure we don't allow stuff like
   // making two '<' adjacent.
-  return !(Lexer::isIdentifierBodyChar(left, LangOpts) &&
-           Lexer::isIdentifierBodyChar(right, LangOpts));
+  return !(Lexer::isAsciiIdentifierContinueChar(left, LangOpts) &&
+           Lexer::isAsciiIdentifierContinueChar(right, LangOpts));
 }
 
 /// Returns true if it is ok to eliminate the trailing whitespace between
 /// the given characters.
 static bool canRemoveWhitespace(char left, char beforeWSpace, char right,
                                 const LangOptions &LangOpts) {
   if (!canBeJoined(left, right, LangOpts))
     return false;

clang/lib/Frontend/LayoutOverrideSource.cpp

 #include "llvm/Support/raw_ostream.h"
 #include <fstream>
 #include <string>
 
 using namespace clang;
 
 /// Parse a simple identifier.
 static std::string parseName(StringRef S) {
-  if (S.empty() || !isIdentifierHead(S[0]))
+  if (S.empty() || !isAsciiIdentifierStart(S[0]))
     return "";
 
   unsigned Offset = 1;
-  while (Offset < S.size() && isIdentifierBody(S[Offset]))
+  while (Offset < S.size() && isAsciiIdentifierContinue(S[Offset]))
     ++Offset;
 
   return S.substr(0, Offset).str();
 }
 
 LayoutOverrideSource::LayoutOverrideSource(StringRef Filename) {
   std::ifstream Input(Filename.str().c_str());
   if (!Input.is_open())

clang/lib/Frontend/Rewrite/FrontendActions.cpp

void visitModuleFile(StringRef Filename,
     // Not interested in PCH / preambles.
     if (!MF->isModule())
       return;
 
     auto OS = Out.lock();
     assert(OS && "loaded module file after finishing rewrite action?");
 
     (*OS) << "#pragma clang module build ";
-    if (isValidIdentifier(MF->ModuleName))
+    if (isValidAsciiIdentifier(MF->ModuleName))
       (*OS) << MF->ModuleName;
     else {
       (*OS) << '"';
       OS->write_escaped(MF->ModuleName);
       (*OS) << '"';
     }
     (*OS) << '\n';
 

clang/lib/Lex/DependencyDirectivesSourceMinimizer.cpp

LLVM_NODISCARD static bool isRawStringLiteral(const char *First,
   // Check if we can even back up.
   if (*Current != '"' || First == Current)
     return false;
 
   // Check for an "R".
   --Current;
   if (*Current != 'R')
     return false;
-  if (First == Current || !isIdentifierBody(*--Current))
+  if (First == Current || !isAsciiIdentifierContinue(*--Current))
     return true;
 
   // Check for a prefix of "u", "U", or "L".
   if (*Current == 'u' || *Current == 'U' || *Current == 'L')
-    return First == Current || !isIdentifierBody(*--Current);
+    return First == Current || !isAsciiIdentifierContinue(*--Current);
 
   // Check for a prefix of "u8".
   if (*Current != '8' || First == Current || *Current-- != 'u')
     return false;
-  return First == Current || !isIdentifierBody(*--Current);
+  return First == Current || !isAsciiIdentifierContinue(*--Current);
 }
 
 static void skipRawString(const char *&First, const char *const End) {
   assert(First[0] == '"');
   assert(First[-1] == 'R');
 
   const char *Last = ++First;
   while (Last != End && *Last != '(')
static bool isQuoteCppDigitSeparator(const char *const Start,
   char Prev = *(Cur - 1);
   if (Prev == 'L' || Prev == 'U' || Prev == 'u')
     return false;
   if (Prev == '8' && (Cur - 1 != Start) && *(Cur - 2) == 'u')
     return false;
   if (!isPreprocessingNumberBody(Prev))
     return false;
   // The next character should be a valid identifier body character.
-  return (Cur + 1) < End && isIdentifierBody(*(Cur + 1));
+  return (Cur + 1) < End && isAsciiIdentifierContinue(*(Cur + 1));
 }
 
 static void skipLine(const char *&First, const char *const End) {
   for (;;) {
     assert(First <= End);
     if (First == End)
       return;
 

 }
 
 void Minimizer::printAdjacentModuleNameParts(const char *&First,
                                              const char *const End) {
   // Skip over parts of the body.
   const char *Last = First;
   do
     ++Last;
-  while (Last != End && (isIdentifierBody(*Last) || *Last == '.'));
+  while (Last != End && (isAsciiIdentifierContinue(*Last) || *Last == '.'));
   append(First, Last);
   First = Last;
 }
 
 bool Minimizer::printAtImportBody(const char *&First, const char *const End) {
   for (;;) {
     skipWhitespace(First, End);
     if (First == End)
       return true;
 
     if (isVerticalWhitespace(*First)) {
       skipNewline(First, End);
       continue;
     }
 
     // Found a semicolon.
     if (*First == ';') {
       put(*First++).put('\n');
       return false;
     }
 
     // Don't handle macro expansions inside @import for now.
-    if (!isIdentifierBody(*First) && *First != '.')
+    if (!isAsciiIdentifierContinue(*First) && *First != '.')
       return true;
 
     printAdjacentModuleNameParts(First, End);
   }
 }
 
 void Minimizer::printDirectiveBody(const char *&First, const char *const End) {
   skipWhitespace(First, End); // Skip initial whitespace.
   printToNewline(First, End);
   while (Out.back() == ' ')
     Out.pop_back();
   put('\n');
 }
 
 LLVM_NODISCARD static const char *lexRawIdentifier(const char *First,
                                                    const char *const End) {
-  assert(isIdentifierBody(*First) && "invalid identifer");
+  assert(isAsciiIdentifierContinue(*First) && "invalid identifer");
   const char *Last = First + 1;
-  while (Last != End && isIdentifierBody(*Last))
+  while (Last != End && isAsciiIdentifierContinue(*Last))
     ++Last;
   return Last;
 }
 
 LLVM_NODISCARD static const char *
 getIdentifierContinuation(const char *First, const char *const End) {
   if (End - First < 3 || First[0] != '\\' || !isVerticalWhitespace(First[1]))
     return nullptr;
 
   ++First;
   skipNewline(First, End);
   if (First == End)
     return nullptr;
-  return isIdentifierBody(First[0]) ? First : nullptr;
+  return isAsciiIdentifierContinue(First[0]) ? First : nullptr;
 }
 
 Minimizer::IdInfo Minimizer::lexIdentifier(const char *First,
                                            const char *const End) {
   const char *Last = lexRawIdentifier(First, End);
   const char *Next = getIdentifierContinuation(Last, End);
   if (LLVM_LIKELY(!Next))
     return IdInfo{Last, StringRef(First, Last - First)};

 
 void Minimizer::printAdjacentMacroArgs(const char *&First,
                                        const char *const End) {
   // Skip over parts of the body.
   const char *Last = First;
   do
     ++Last;
   while (Last != End &&
-         (isIdentifierBody(*Last) || *Last == '.' || *Last == ','));
+         (isAsciiIdentifierContinue(*Last) || *Last == '.' || *Last == ','));
   append(First, Last);
   First = Last;
 }
 
 bool Minimizer::printMacroArgs(const char *&First, const char *const End) {
   assert(*First == '(');
   put(*First++);
   for (;;) {
     skipWhitespace(First, End);
     if (First == End)
       return true;
 
     if (*First == ')') {
       put(*First++);
       return false;
     }
 
     // This is intentionally fairly liberal.
-    if (!(isIdentifierBody(*First) || *First == '.' || *First == ','))
+    if (!(isAsciiIdentifierContinue(*First) || *First == '.' || *First == ','))
       return true;
 
     printAdjacentMacroArgs(First, End);
   }
 }
 
 /// Looks for an identifier starting from Last.
 ///
 /// Updates "First" to just past the next identifier, if any.  Returns true iff
 /// the identifier matches "Id".
 bool Minimizer::isNextIdentifier(StringRef Id, const char *&First,
                                  const char *const End) {
   skipWhitespace(First, End);
-  if (First == End || !isIdentifierHead(*First))
+  if (First == End || !isAsciiIdentifierStart(*First))
     return false;
 
   IdInfo FoundId = lexIdentifier(First, End);
   First = FoundId.Last;
   return FoundId.Name == Id;
 }
 
 bool Minimizer::lexAt(const char *&First, const char *const End) {

 
 bool Minimizer::lexModule(const char *&First, const char *const End) {
   IdInfo Id = lexIdentifier(First, End);
   First = Id.Last;
   bool Export = false;
   if (Id.Name == "export") {
     Export = true;
     skipWhitespace(First, End);
-    if (!isIdentifierBody(*First)) {
+    if (!isAsciiIdentifierContinue(*First)) {
       skipLine(First, End);
       return false;
     }
     Id = lexIdentifier(First, End);
     First = Id.Last;
   }
 
   if (Id.Name != "module" && Id.Name != "import") {
     skipLine(First, End);
     return false;
   }
 
   skipWhitespace(First, End);
 
   // Ignore this as a module directive if the next character can't be part of
   // an import.
 
   switch (*First) {
   case ':':
   case '<':
   case '"':
     break;
   default:
-    if (!isIdentifierBody(*First)) {
+    if (!isAsciiIdentifierContinue(*First)) {
       skipLine(First, End);
       return false;
     }
   }
 
   if (Export) {
     makeToken(cxx_export_decl);
     append("export ");
bool Minimizer::lexModule(const char *&First, const char *const End) {
   return false;
 }
 
 bool Minimizer::lexDefine(const char *&First, const char *const End) {
   makeToken(pp_define);
   append("#define ");
   skipWhitespace(First, End);
 
-  if (!isIdentifierHead(*First))
+  if (!isAsciiIdentifierStart(*First))
     return reportError(First, diag::err_pp_macro_not_identifier);
 
   IdInfo Id = lexIdentifier(First, End);
   const char *Last = Id.Last;
   append(Id.Name);
   if (Last == End)
     return false;
   if (*Last == '(') {
bool Minimizer::lexDefine(const char *&First, const char *const End) {
   printDirectiveBody(Last, End);
   First = Last;
   return false;
 }
 
 bool Minimizer::lexPragma(const char *&First, const char *const End) {
   // #pragma.
   skipWhitespace(First, End);
-  if (First == End || !isIdentifierHead(*First))
+  if (First == End || !isAsciiIdentifierStart(*First))
     return false;
 
   IdInfo FoundId = lexIdentifier(First, End);
   First = FoundId.Last;
   if (FoundId.Name == "once") {
     // #pragma once
     skipLine(First, End);
     makeToken(pp_pragma_once);
bool Minimizer::lexPPLine(const char *&First, const char *const End) {
 
   // Handle preprocessing directives.
   ++First; // Skip over '#'.
   skipWhitespace(First, End);
 
   if (First == End)
     return reportError(First, diag::err_pp_expected_eol);
 
-  if (!isIdentifierHead(*First)) {
+  if (!isAsciiIdentifierStart(*First)) {
     skipLine(First, End);
     return false;
   }
 
   // Figure out the token.
   IdInfo Id = lexIdentifier(First, End);
   First = Id.Last;
   auto Kind = llvm::StringSwitch<TokenKind>(Id.Name)

clang/lib/Lex/Lexer.cpp

StringRef Lexer::getImmediateMacroNameForDiagnostics(
   // Dig out the buffer where the macro name was spelled and the extents of the
   // name so that we can render it into the expansion note.
   std::pair<FileID, unsigned> ExpansionInfo = SM.getDecomposedLoc(Loc);
   unsigned MacroTokenLength = Lexer::MeasureTokenLength(Loc, SM, LangOpts);
   StringRef ExpansionBuffer = SM.getBufferData(ExpansionInfo.first);
   return ExpansionBuffer.substr(ExpansionInfo.second, MacroTokenLength);
 }
 
-bool Lexer::isIdentifierBodyChar(char c, const LangOptions &LangOpts) {
-  return isIdentifierBody(c, LangOpts.DollarIdents);
+bool Lexer::isAsciiIdentifierContinueChar(char c, const LangOptions &LangOpts) {
+  return isAsciiIdentifierContinue(c, LangOpts.DollarIdents);
 }
 
 bool Lexer::isNewLineEscaped(const char *BufferStart, const char *Str) {
   assert(isVerticalWhitespace(Str[0]));
   if (Str - 1 < BufferStart)
     return false;
 
   if ((Str[0] == '\n' && Str[-1] == '\r') ||
if (!isAllowedIDChar(static_cast<uint32_t>(CodePoint), LangOpts)) {
     maybeDiagnoseUTF8Homoglyph(PP->getDiagnostics(), CodePoint,
                                makeCharRange(*this, CurPtr, UnicodePtr));
   }
 
   CurPtr = UnicodePtr;
   return true;
 }
 
-bool Lexer::LexIdentifier(Token &Result, const char *CurPtr) {
-  // Match [_A-Za-z0-9]*, we have already matched [_A-Za-z$]
-  unsigned Size;
-  unsigned char C = *CurPtr++;
-  while (isIdentifierBody(C))
-    C = *CurPtr++;
+bool Lexer::LexUnicodeIdentifierStart(Token &Result, uint32_t C,
+                                      const char *CurPtr) {
+  if (isAllowedInitiallyIDChar(C, LangOpts)) {
+    if (!isLexingRawMode() && !ParsingPreprocessorDirective &&
+        !PP->isPreprocessedOutput()) {
+      maybeDiagnoseIDCharCompat(PP->getDiagnostics(), C,
+                                makeCharRange(*this, BufferPtr, CurPtr),
+                                /*IsFirst=*/true);
+      maybeDiagnoseUTF8Homoglyph(PP->getDiagnostics(), C,
+                                 makeCharRange(*this, BufferPtr, CurPtr));
+    }
 
-  --CurPtr;   // Back up over the skipped character.
+    MIOpt.ReadToken();
+    return LexIdentifierContinue(Result, CurPtr);
+  }
+
+  if (!isLexingRawMode() && !ParsingPreprocessorDirective &&
+      !PP->isPreprocessedOutput() && !isASCII(*BufferPtr) &&
+      !isAllowedInitiallyIDChar(C, LangOpts) && !isUnicodeWhitespace(C)) {
+    // Non-ASCII characters tend to creep into source code unintentionally.
+    // Instead of letting the parser complain about the unknown token,
+    // just drop the character.
+    // Note that we can /only/ do this when the non-ASCII character is actually
+    // spelled as Unicode, not written as a UCN. The standard requires that
+    // we not throw away any possible preprocessor tokens, but there's a
+    // loophole in the mapping of Unicode characters to basic character set
+    // characters that allows us to map these particular characters to, say,
+    // whitespace.
+    diagnoseInvalidUnicodeCodepointInIdentifier(
+        PP->getDiagnostics(), LangOpts, C,
+        makeCharRange(*this, BufferPtr, CurPtr), /*IsStart*/ true);
+    BufferPtr = CurPtr;
+    return false;
+  }
+
+  // Otherwise, we have an explicit UCN or a character that's unlikely to show
+  // up by accident.
+  MIOpt.ReadToken();
+  FormTokenWithChars(Result, CurPtr, tok::unknown);
+  return true;
+}
+
+bool Lexer::LexIdentifierContinue(Token &Result, const char *CurPtr) {
+  // Match [_A-Za-z0-9]*, we have already matched an identifier start.
+  while (true) {
+    unsigned char C = *CurPtr;
+    // Fast path.
+    if (isAsciiIdentifierContinue(C)) {
+      ++CurPtr;
+      continue;
+    }
+
+    unsigned Size;
+    // Slow path: handle trigraph, unicode codepoints, UCNs.
+    C = getCharAndSize(CurPtr, Size);
+    if (isAsciiIdentifierContinue(C)) {
+      CurPtr = ConsumeChar(CurPtr, Size, Result);
+      continue;
+    }
+    if (C == '$') {
+      // If we hit a $ and they are not supported in identifiers, we are done.
+      if (!LangOpts.DollarIdents)
+        break;
+      // Otherwise, emit a diagnostic and continue.
+      if (!isLexingRawMode())
+        Diag(CurPtr, diag::ext_dollar_in_identifier);
+      CurPtr = ConsumeChar(CurPtr, Size, Result);
+      continue;
+    }
+    if (C == '\\' && tryConsumeIdentifierUCN(CurPtr, Size, Result))
+      continue;
+    if (!isASCII(C) && tryConsumeIdentifierUTF8Char(CurPtr))
+      continue;
+    // Neither an expected Unicode codepoint nor a UCN.
+    break;
+  }
 
-  // Fast path, no $,\,? in identifier found.  '\' might be an escaped newline
-  // or UCN, and ? might be a trigraph for '\', an escaped newline or UCN.
-  //
-  // TODO: Could merge these checks into an InfoTable flag to make the
-  // comparison cheaper
-  if (isASCII(C) && C != '\\' && C != '?' &&
-      (C != '$' || !LangOpts.DollarIdents)) {
-FinishIdentifier:
-    const char *IdStart = BufferPtr;
-    FormTokenWithChars(Result, CurPtr, tok::raw_identifier);
-    Result.setRawIdentifierData(IdStart);
 
-    // If we are in raw mode, return this identifier raw.  There is no need to
-    // look up identifier information or attempt to macro expand it.
-    if (LexingRawMode)
-      return true;
 
-    // Fill in Result.IdentifierInfo and update the token kind,
-    // looking up the identifier in the identifier table.
-    IdentifierInfo *II = PP->LookUpIdentifierInfo(Result);
-    // Note that we have to call PP->LookUpIdentifierInfo() even for code
-    // completion, it writes IdentifierInfo into Result, and callers rely on it.
 
-    // If the completion point is at the end of an identifier, we want to treat
-    // the identifier as incomplete even if it resolves to a macro or a keyword.
-    // This allows e.g. 'class^' to complete to 'classifier'.
-    if (isCodeCompletionPoint(CurPtr)) {
-      // Return the code-completion token.
-      Result.setKind(tok::code_completion);
-      // Skip the code-completion char and all immediate identifier characters.
-      // This ensures we get consistent behavior when completing at any point in
-      // an identifier (i.e. at the start, in the middle, at the end). Note that
-      // only simple cases (i.e. [a-zA-Z0-9_]) are supported to keep the code
-      // simpler.
-      assert(*CurPtr == 0 && "Completion character must be 0");
-      ++CurPtr;
-      // Note that code completion token is not added as a separate character
-      // when the completion point is at the end of the buffer. Therefore, we need
-      // to check if the buffer has ended.
-      if (CurPtr < BufferEnd) {
-        while (isIdentifierBody(*CurPtr))
-          ++CurPtr;
-      }
-      BufferPtr = CurPtr;
-      return true;
-    }
+      while (isAsciiIdentifierContinue(*CurPtr))
 
-    // Finally, now that we know we have an identifier, pass this off to the
-    // preprocessor, which may macro expand it or something.
-    if (II->isHandleIdentifierCase())
-      return PP->HandleIdentifier(Result);
 
-    return true;
-  }
 
-  // Otherwise, $,\,? in identifier found.  Enter slower path.
-
-  C = getCharAndSize(CurPtr, Size);
-  while (true) {
-    if (C == '$') {
-      // If we hit a $ and they are not supported in identifiers, we are done.
-      if (!LangOpts.DollarIdents) goto FinishIdentifier;
-
-      // Otherwise, emit a diagnostic and continue.
-      if (!isLexingRawMode())
-        Diag(CurPtr, diag::ext_dollar_in_identifier);
-      CurPtr = ConsumeChar(CurPtr, Size, Result);
-      C = getCharAndSize(CurPtr, Size);
-      continue;
-    } else if (C == '\\' && tryConsumeIdentifierUCN(CurPtr, Size, Result)) {
-      C = getCharAndSize(CurPtr, Size);
-      continue;
-    } else if (!isASCII(C) && tryConsumeIdentifierUTF8Char(CurPtr)) {
-      C = getCharAndSize(CurPtr, Size);
-      continue;
-    } else if (!isIdentifierBody(C)) {
-      goto FinishIdentifier;
-    }
-
-    // Otherwise, this character is good, consume it.
-    CurPtr = ConsumeChar(CurPtr, Size, Result);
-
-    C = getCharAndSize(CurPtr, Size);
-    while (isIdentifierBody(C)) {
-      CurPtr = ConsumeChar(CurPtr, Size, Result);
-      C = getCharAndSize(CurPtr, Size);
-    }
-  }
-}
-
 /// isHexaLiteral - Return true if Start points to a hex constant.
 /// in microsoft mode (where this is supposed to be several different tokens).
 bool Lexer::isHexaLiteral(const char *Start, const LangOptions &LangOpts) {
   unsigned Size;
   char C1 = Lexer::getCharAndSizeNoWarn(Start, Size, LangOpts);
   if (C1 != '0')
     return false;
   char C2 = Lexer::getCharAndSizeNoWarn(Start + Size, Size, LangOpts);
if ((C == '-' || C == '+') && (PrevCh == 'P' || PrevCh == 'p')) {
     if (IsHexFloat)
       return LexNumericConstant(Result, ConsumeChar(CurPtr, Size, Result));
   }
 
   // If we have a digit separator, continue.
   if (C == '\'' && (getLangOpts().CPlusPlus14 || getLangOpts().C2x)) {
     unsigned NextSize;
     char Next = getCharAndSizeNoWarn(CurPtr + Size, NextSize, getLangOpts());
-    if (isIdentifierBody(Next)) {
+    if (isAsciiIdentifierContinue(Next)) {
       if (!isLexingRawMode())
         Diag(CurPtr, getLangOpts().CPlusPlus
                          ? diag::warn_cxx11_compat_digit_separator
                          : diag::warn_c2x_compat_digit_separator);
       CurPtr = ConsumeChar(CurPtr, Size, Result);
       CurPtr = ConsumeChar(CurPtr, NextSize, Result);
       return LexNumericConstant(Result, CurPtr);
     }
const char *Lexer::LexUDSuffix(Token &Result, const char *CurPtr,
                                bool IsStringLiteral) {
   assert(getLangOpts().CPlusPlus);
 
   // Maximally munch an identifier.
   unsigned Size;
   char C = getCharAndSize(CurPtr, Size);
   bool Consumed = false;
 
-  if (!isIdentifierHead(C)) {
+  if (!isAsciiIdentifierStart(C)) {
     if (C == '\\' && tryConsumeIdentifierUCN(CurPtr, Size, Result))
       Consumed = true;
     else if (!isASCII(C) && tryConsumeIdentifierUTF8Char(CurPtr))
       Consumed = true;
     else
       return CurPtr;
   }
 
else if (IsStringLiteral && getLangOpts().CPlusPlus14) {
       const unsigned MaxStandardSuffixLength = 3;
       char Buffer[MaxStandardSuffixLength] = { C };
       unsigned Consumed = Size;
       unsigned Chars = 1;
       while (true) {
         unsigned NextSize;
         char Next = getCharAndSizeNoWarn(CurPtr + Consumed, NextSize,
                                          getLangOpts());
-        if (!isIdentifierBody(Next)) {
+        if (!isAsciiIdentifierContinue(Next)) {
           // End of suffix. Check whether this is on the allowed list.
           const StringRef CompleteSuffix(Buffer, Chars);
           IsUDSuffix = StringLiteralParser::isValidUDSuffix(getLangOpts(),
                                                             CompleteSuffix);
           break;
         }
 
         if (Chars == MaxStandardSuffixLength)
if (!Consumed) {
     }
 
     CurPtr = ConsumeChar(CurPtr, Size, Result);
   }
 
   Result.setFlag(Token::HasUDSuffix);
   while (true) {
     C = getCharAndSize(CurPtr, Size);
-    if (isIdentifierBody(C)) { CurPtr = ConsumeChar(CurPtr, Size, Result); }
-    else if (C == '\\' && tryConsumeIdentifierUCN(CurPtr, Size, Result)) {}
-    else if (!isASCII(C) && tryConsumeIdentifierUTF8Char(CurPtr)) {}
-    else break;
+    if (isAsciiIdentifierContinue(C)) {
+      CurPtr = ConsumeChar(CurPtr, Size, Result);
+    } else if (C == '\\' && tryConsumeIdentifierUCN(CurPtr, Size, Result)) {
+    } else if (!isASCII(C) && tryConsumeIdentifierUTF8Char(CurPtr)) {
+    } else
+      break;
   }
 
   return CurPtr;
 }
 
 /// LexStringLiteral - Lex the remainder of a string literal, after having lexed
 /// either " or L" or u8" or u" or U".
 bool Lexer::LexStringLiteral(Token &Result, const char *CurPtr,
Diag(BufferPtr, diag::ext_unicode_whitespace)
       << makeCharRange(*this, BufferPtr, CurPtr);
 
     Result.setFlag(Token::LeadingSpace);
     return true;
   }
   return false;
 }
 
-bool Lexer::LexUnicode(Token &Result, uint32_t C, const char *CurPtr) {
-  if (isAllowedInitiallyIDChar(C, LangOpts)) {
-    if (!isLexingRawMode() && !ParsingPreprocessorDirective &&
-        !PP->isPreprocessedOutput()) {
-      maybeDiagnoseIDCharCompat(PP->getDiagnostics(), C,
-                                makeCharRange(*this, BufferPtr, CurPtr),
-                                /*IsFirst=*/true);
-      maybeDiagnoseUTF8Homoglyph(PP->getDiagnostics(), C,
-                                 makeCharRange(*this, BufferPtr, CurPtr));
-    }
-
-    MIOpt.ReadToken();
-    return LexIdentifier(Result, CurPtr);
-  }
-
-  if (!isLexingRawMode() && !ParsingPreprocessorDirective &&
-      !PP->isPreprocessedOutput() && !isASCII(*BufferPtr) &&
-      !isAllowedInitiallyIDChar(C, LangOpts) && !isUnicodeWhitespace(C)) {
-    // Non-ASCII characters tend to creep into source code unintentionally.
-    // Instead of letting the parser complain about the unknown token,
-    // just drop the character.
-    // Note that we can /only/ do this when the non-ASCII character is actually
-    // spelled as Unicode, not written as a UCN. The standard requires that
-    // we not throw away any possible preprocessor tokens, but there's a
-    // loophole in the mapping of Unicode characters to basic character set
-    // characters that allows us to map these particular characters to, say,
-    // whitespace.
-    diagnoseInvalidUnicodeCodepointInIdentifier(
-        PP->getDiagnostics(), LangOpts, C,
-        makeCharRange(*this, BufferPtr, CurPtr), /*IsStart*/ true);
-    BufferPtr = CurPtr;
-    return false;
-  }
-
-  // Otherwise, we have an explicit UCN or a character that's unlikely to show
-  // up by accident.
-  MIOpt.ReadToken();
-  FormTokenWithChars(Result, CurPtr, tok::unknown);
-  return true;
-}
-
 void Lexer::PropagateLineStartLeadingSpaceInfo(Token &Result) {
   IsAtStartOfLine = Result.isAtStartOfLine();
   HasLeadingSpace = Result.hasLeadingSpace();
   HasLeadingEmptyMacro = Result.hasLeadingEmptyMacro();
   // Note that this doesn't affect IsAtPhysicalStartOfLine.
 }
 
 bool Lexer::Lex(Token &Result) {
if (LangOpts.CPlusPlus11 || LangOpts.C11) {
                                SizeTmp3, Result),
                    tok::utf8_string_literal);
           }
         }
       }
     }
 
     // treat u like the start of an identifier.
-    return LexIdentifier(Result, CurPtr);
+    return LexIdentifierContinue(Result, CurPtr);
 
   case 'U':   // Identifier (Uber) or C11/C++11 UTF-32 string literal
     // Notify MIOpt that we read a non-whitespace/non-comment token.
     MIOpt.ReadToken();
 
     if (LangOpts.CPlusPlus11 || LangOpts.C11) {
       Char = getCharAndSize(CurPtr, SizeTmp);
 
if (LangOpts.CPlusPlus11 || LangOpts.C11) {
           getCharAndSize(CurPtr + SizeTmp, SizeTmp2) == '"')
         return LexRawStringLiteral(Result,
                                ConsumeChar(ConsumeChar(CurPtr, SizeTmp, Result),
                                            SizeTmp2, Result),
                                tok::utf32_string_literal);
     }
 
     // treat U like the start of an identifier.
-    return LexIdentifier(Result, CurPtr);
+    return LexIdentifierContinue(Result, CurPtr);
 
   case 'R': // Identifier or C++0x raw string literal
     // Notify MIOpt that we read a non-whitespace/non-comment token.
     MIOpt.ReadToken();
 
     if (LangOpts.CPlusPlus11) {
       Char = getCharAndSize(CurPtr, SizeTmp);
 
       if (Char == '"')
         return LexRawStringLiteral(Result,
                                    ConsumeChar(CurPtr, SizeTmp, Result),
                                    tok::string_literal);
     }
 
     // treat R like the start of an identifier.
-    return LexIdentifier(Result, CurPtr);
+    return LexIdentifierContinue(Result, CurPtr);
 
   case 'L':   // Identifier (Loony) or wide literal (L'x' or L"xyz").
     // Notify MIOpt that we read a non-whitespace/non-comment token.
     MIOpt.ReadToken();
     Char = getCharAndSize(CurPtr, SizeTmp);
 
     // Wide string literal.
     if (Char == '"')
LexNextToken:
   case 'V': case 'W': case 'X': case 'Y': case 'Z':
   case 'a': case 'b': case 'c': case 'd': case 'e': case 'f': case 'g':
   case 'h': case 'i': case 'j': case 'k': case 'l': case 'm': case 'n':
   case 'o': case 'p': case 'q': case 'r': case 's': case 't':    /*'u'*/
   case 'v': case 'w': case 'x': case 'y': case 'z':
   case '_':
     // Notify MIOpt that we read a non-whitespace/non-comment token.
     MIOpt.ReadToken();
-    return LexIdentifier(Result, CurPtr);
+    return LexIdentifierContinue(Result, CurPtr);
 
   case '$':   // $ in identifiers.
     if (LangOpts.DollarIdents) {
       if (!isLexingRawMode())
         Diag(CurPtr-1, diag::ext_dollar_in_identifier);
       // Notify MIOpt that we read a non-whitespace/non-comment token.
       MIOpt.ReadToken();
-      return LexIdentifier(Result, CurPtr);
+      return LexIdentifierContinue(Result, CurPtr);
     }
 
     Kind = tok::unknown;
     break;
 
   // C99 6.4.4: Character Constants.
   case '\'':
     // Notify MIOpt that we read a non-whitespace/non-comment token.
if (!LangOpts.AsmPreprocessor) {
           if (SkipWhitespace(Result, CurPtr, TokAtPhysicalStartOfLine))
             return true; // KeepWhitespaceMode
 
           // We only saw whitespace, so just try again with this lexer.
           // (We manually eliminate the tail call to avoid recursion.)
           goto LexNextToken;
         }
 
-        return LexUnicode(Result, CodePoint, CurPtr);
+        return LexUnicodeIdentifierStart(Result, CodePoint, CurPtr);
       }
     }
 
     Kind = tok::unknown;
     break;
 
   default: {
     if (isASCII(Char)) {
if (Status == llvm::conversionOK) {
       if (CheckUnicodeWhitespace(Result, CodePoint, CurPtr)) {
         if (SkipWhitespace(Result, CurPtr, TokAtPhysicalStartOfLine))
           return true; // KeepWhitespaceMode
 
         // We only saw whitespace, so just try again with this lexer.
         // (We manually eliminate the tail call to avoid recursion.)
         goto LexNextToken;
       }
-      return LexUnicode(Result, CodePoint, CurPtr);
+      return LexUnicodeIdentifierStart(Result, CodePoint, CurPtr);
     }
 
     if (isLexingRawMode() || ParsingPreprocessorDirective ||
         PP->isPreprocessedOutput()) {
       ++CurPtr;
       Kind = tok::unknown;
       break;
     }

clang/lib/Lex/ModuleMap.cpp

 }
 
 /// "Sanitize" a filename so that it can be used as an identifier.
 static StringRef sanitizeFilenameAsIdentifier(StringRef Name,
                                               SmallVectorImpl<char> &Buffer) {
   if (Name.empty())
     return Name;
 
-  if (!isValidIdentifier(Name)) {
+  if (!isValidAsciiIdentifier(Name)) {
     // If we don't already have something with the form of an identifier,
     // create a buffer with the sanitized name.
     Buffer.clear();
     if (isDigit(Name[0]))
       Buffer.push_back('_');
     Buffer.reserve(Buffer.size() + Name.size());
     for (unsigned I = 0, N = Name.size(); I != N; ++I) {
-      if (isIdentifierBody(Name[I]))
+      if (isAsciiIdentifierContinue(Name[I]))
         Buffer.push_back(Name[I]);
       else
         Buffer.push_back('_');
     }
 
     Name = StringRef(Buffer.data(), Buffer.size());
   }
 

clang/lib/Sema/SemaAvailability.cpp

if (SlotNames.size() != 1)
       return None;
     NumParams = 0;
   }
   // Verify all slot names are valid.
   bool AllowDollar = LangOpts.DollarIdents;
   for (StringRef S : SlotNames) {
     if (S.empty())
       continue;
-    if (!isValidIdentifier(S, AllowDollar))
+    if (!isValidAsciiIdentifier(S, AllowDollar))
       return None;
   }
   return NumParams;
 }
 
 /// Returns a source location in which it's appropriate to insert a new
 /// attribute for the given declaration \D.
 static Optional<AttributeInsertion>

clang/lib/Sema/SemaDeclAttr.cpp

validateSwiftFunctionName(Sema &S, const ParsedAttr &AL, SourceLocation Loc,
   std::tie(BaseName, Parameters) = Name.split('(');
 
   // Split at the first '.', if it exists, which separates the context name
   // from the base name.
   std::tie(ContextName, BaseName) = BaseName.split('.');
   if (BaseName.empty()) {
     BaseName = ContextName;
     ContextName = StringRef();
-  } else if (ContextName.empty() || !isValidIdentifier(ContextName)) {
+  } else if (ContextName.empty() || !isValidAsciiIdentifier(ContextName)) {
     S.Diag(Loc, diag::warn_attr_swift_name_invalid_identifier)
         << AL << /*context*/ 1;
     return false;
   } else {
     IsMember = true;
   }
 
-  if (!isValidIdentifier(BaseName) || BaseName == "_") {
+  if (!isValidAsciiIdentifier(BaseName) || BaseName == "_") {
     S.Diag(Loc, diag::warn_attr_swift_name_invalid_identifier)
         << AL << /*basename*/ 0;
     return false;
   }
 
   bool IsSubscript = BaseName == "subscript";
   // A subscript accessor must be a getter or setter.
   if (IsSubscript && !IsGetter && !IsSetter) {
validateSwiftFunctionName(Sema &S, const ParsedAttr &AL, SourceLocation Loc,
 
   StringRef CurrentParam;
   llvm::Optional<unsigned> SelfLocation;
   unsigned NewValueCount = 0;
   llvm::Optional<unsigned> NewValueLocation;
   do {
     std::tie(CurrentParam, Parameters) = Parameters.split(':');
 
-    if (!isValidIdentifier(CurrentParam)) {
+    if (!isValidAsciiIdentifier(CurrentParam)) {
       S.Diag(Loc, diag::warn_attr_swift_name_invalid_identifier)
           << AL << /*parameter*/2;
       return false;
     }
 
     if (IsMember && CurrentParam == "self") {
       // "self" indicates the "self" argument for a member.
 
if (!ParamCountValid) {
               isa<IndirectFieldDecl>(D) || isa<FieldDecl>(D)) &&
              !IsAsync) {
     StringRef ContextName, BaseName;
 
     std::tie(ContextName, BaseName) = Name.split('.');
     if (BaseName.empty()) {
       BaseName = ContextName;
       ContextName = StringRef();
-    } else if (!isValidIdentifier(ContextName)) {
+    } else if (!isValidAsciiIdentifier(ContextName)) {
       Diag(Loc, diag::warn_attr_swift_name_invalid_identifier) << AL
           << /*context*/1;
       return false;
     }
 
-    if (!isValidIdentifier(BaseName)) {
+    if (!isValidAsciiIdentifier(BaseName)) {
       Diag(Loc, diag::warn_attr_swift_name_invalid_identifier) << AL
           << /*basename*/0;
       return false;
     }
   } else {
     Diag(Loc, diag::warn_attr_swift_name_decl_kind) << AL;
     return false;
   }

clang/lib/Sema/SemaExprObjC.cpp

if (CFBridgeName) {
     if (CCK == Sema::CCK_OtherCast) {
       if (const CXXNamedCastExpr *NCE = dyn_cast<CXXNamedCastExpr>(realCast)) {
         SourceRange range(NCE->getOperatorLoc(),
                           NCE->getAngleBrackets().getEnd());
         SmallString<32> BridgeCall;
 
         SourceManager &SM = S.getSourceManager();
         char PrevChar = *SM.getCharacterData(range.getBegin().getLocWithOffset(-1));
-        if (Lexer::isIdentifierBodyChar(PrevChar, S.getLangOpts()))
+        if (Lexer::isAsciiIdentifierContinueChar(PrevChar, S.getLangOpts()))
           BridgeCall += ' ';
 
         BridgeCall += CFBridgeName;
         DiagB.AddFixItHint(FixItHint::CreateReplacement(range, BridgeCall));
       }
       return;
     }
     Expr *castedE = castExpr;
     if (CStyleCastExpr *CCE = dyn_cast<CStyleCastExpr>(castedE))
       castedE = CCE->getSubExpr();
     castedE = castedE->IgnoreImpCasts();
     SourceRange range = castedE->getSourceRange();
 
     SmallString<32> BridgeCall;
 
     SourceManager &SM = S.getSourceManager();
     char PrevChar = *SM.getCharacterData(range.getBegin().getLocWithOffset(-1));
-    if (Lexer::isIdentifierBodyChar(PrevChar, S.getLangOpts()))
+    if (Lexer::isAsciiIdentifierContinueChar(PrevChar, S.getLangOpts()))
       BridgeCall += ' ';
 
     BridgeCall += CFBridgeName;
 
     if (isa<ParenExpr>(castedE)) {
       DiagB.AddFixItHint(FixItHint::CreateInsertion(range.getBegin(),
                          BridgeCall));
     } else {

clang/lib/Sema/SemaType.cpp

static void fixItNullability(Sema &S, DiagBuilderT &Diag,
 
   if (isWhitespace(*NextChar)) {
     InsertionText = InsertionText.drop_back();
   } else if (NextChar[-1] == '[') {
     if (NextChar[0] == ']')
       InsertionText = InsertionText.drop_back().drop_front();
     else
       InsertionText = InsertionText.drop_front();
-  } else if (!isIdentifierBody(NextChar[0], /*allow dollar*/true) &&
-             !isIdentifierBody(NextChar[-1], /*allow dollar*/true)) {
+  } else if (!isAsciiIdentifierContinue(NextChar[0], /*allow dollar*/ true) &&
+             !isAsciiIdentifierContinue(NextChar[-1], /*allow dollar*/ true)) {
     InsertionText = InsertionText.drop_back().drop_front();
   }
 
   Diag << FixItHint::CreateInsertion(FixItLoc, InsertionText);
 }
 
 static void emitNullabilityConsistencyWarning(Sema &S,
                                               SimplePointerKind PointerKind,

clang/lib/Tooling/Transformer/Parsing.cpp

return makeParseError(State,
                           ("expected char not found: " + llvm::Twine(c)).str());
   return makeParseProgress(advance(State, 1), llvm::None);
 }
 
 // Parses an identitifer "token" -- handles preceding whitespace.
 static ExpectedProgress<std::string> parseId(ParseState State) {
   State.Input = consumeWhitespace(State.Input);
   auto Id = State.Input.take_while(
-      [](char c) { return isASCII(c) && isIdentifierBody(c); });
+      [](char c) { return isASCII(c) && isAsciiIdentifierContinue(c); });
   if (Id.empty())
     return makeParseError(State, "failed to parse name");
   return makeParseProgress(advance(State, Id.size()), Id.str());
 }
 
 // For consistency with the AST matcher parser and C++ code, node ids are
 // written as strings. However, we do not support escaping in the string.
 static ExpectedProgress<std::string> parseStringId(ParseState State) {

clang/unittests/Basic/CharInfoTest.cpp

TEST(CharInfoTest, isASCII) {
   EXPECT_TRUE(isASCII(' '));
   EXPECT_TRUE(isASCII('a'));
   EXPECT_TRUE(isASCII('\x7f'));
   EXPECT_FALSE(isASCII('\x80'));
   EXPECT_FALSE(isASCII('\xc2'));
   EXPECT_FALSE(isASCII('\xff'));
 }
 
-TEST(CharInfoTest, isIdentifierHead) {
-  EXPECT_TRUE(isIdentifierHead('a'));
-  EXPECT_TRUE(isIdentifierHead('A'));
-  EXPECT_TRUE(isIdentifierHead('z'));
-  EXPECT_TRUE(isIdentifierHead('Z'));
-  EXPECT_TRUE(isIdentifierHead('_'));
-
-  EXPECT_FALSE(isIdentifierHead('0'));
-  EXPECT_FALSE(isIdentifierHead('.'));
-  EXPECT_FALSE(isIdentifierHead('`'));
-  EXPECT_FALSE(isIdentifierHead('\0'));
-
-  EXPECT_FALSE(isIdentifierHead('$'));
-  EXPECT_TRUE(isIdentifierHead('$', /*AllowDollar=*/true));
-
-  EXPECT_FALSE(isIdentifierHead('\x80'));
-  EXPECT_FALSE(isIdentifierHead('\xc2'));
-  EXPECT_FALSE(isIdentifierHead('\xff'));
-}
-
-TEST(CharInfoTest, isIdentifierBody) {
-  EXPECT_TRUE(isIdentifierBody('a'));
-  EXPECT_TRUE(isIdentifierBody('A'));
-  EXPECT_TRUE(isIdentifierBody('z'));
-  EXPECT_TRUE(isIdentifierBody('Z'));
-  EXPECT_TRUE(isIdentifierBody('_'));
-
-  EXPECT_TRUE(isIdentifierBody('0'));
-  EXPECT_FALSE(isIdentifierBody('.'));
-  EXPECT_FALSE(isIdentifierBody('`'));
-  EXPECT_FALSE(isIdentifierBody('\0'));
-
-  EXPECT_FALSE(isIdentifierBody('$'));
-  EXPECT_TRUE(isIdentifierBody('$', /*AllowDollar=*/true));
-
-  EXPECT_FALSE(isIdentifierBody('\x80'));
-  EXPECT_FALSE(isIdentifierBody('\xc2'));
-  EXPECT_FALSE(isIdentifierBody('\xff'));
+TEST(CharInfoTest, isAsciiIdentifierStart) {
+  EXPECT_TRUE(isAsciiIdentifierStart('a'));
+  EXPECT_TRUE(isAsciiIdentifierStart('A'));
+  EXPECT_TRUE(isAsciiIdentifierStart('z'));
+  EXPECT_TRUE(isAsciiIdentifierStart('Z'));
+  EXPECT_TRUE(isAsciiIdentifierStart('_'));
+
+  EXPECT_FALSE(isAsciiIdentifierStart('0'));
+  EXPECT_FALSE(isAsciiIdentifierStart('.'));
+  EXPECT_FALSE(isAsciiIdentifierStart('`'));
+  EXPECT_FALSE(isAsciiIdentifierStart('\0'));
+
+  EXPECT_FALSE(isAsciiIdentifierStart('$'));
+  EXPECT_TRUE(isAsciiIdentifierStart('$', /*AllowDollar=*/true));
+
+  EXPECT_FALSE(isAsciiIdentifierStart('\x80'));
+  EXPECT_FALSE(isAsciiIdentifierStart('\xc2'));
+  EXPECT_FALSE(isAsciiIdentifierStart('\xff'));
+}
+
+TEST(CharInfoTest, isAsciiIdentifierContinue) {
+  EXPECT_TRUE(isAsciiIdentifierContinue('a'));
+  EXPECT_TRUE(isAsciiIdentifierContinue('A'));
+  EXPECT_TRUE(isAsciiIdentifierContinue('z'));
+  EXPECT_TRUE(isAsciiIdentifierContinue('Z'));
+  EXPECT_TRUE(isAsciiIdentifierContinue('_'));
+
+  EXPECT_TRUE(isAsciiIdentifierContinue('0'));
+  EXPECT_FALSE(isAsciiIdentifierContinue('.'));
+  EXPECT_FALSE(isAsciiIdentifierContinue('`'));
+  EXPECT_FALSE(isAsciiIdentifierContinue('\0'));
+
+  EXPECT_FALSE(isAsciiIdentifierContinue('$'));
+  EXPECT_TRUE(isAsciiIdentifierContinue('$', /*AllowDollar=*/true));
+
+  EXPECT_FALSE(isAsciiIdentifierContinue('\x80'));
+  EXPECT_FALSE(isAsciiIdentifierContinue('\xc2'));
+  EXPECT_FALSE(isAsciiIdentifierContinue('\xff'));
 }
 
 TEST(CharInfoTest, isHorizontalWhitespace) {
   EXPECT_FALSE(isHorizontalWhitespace('a'));
   EXPECT_FALSE(isHorizontalWhitespace('_'));
   EXPECT_FALSE(isHorizontalWhitespace('0'));
   EXPECT_FALSE(isHorizontalWhitespace('.'));
   EXPECT_FALSE(isHorizontalWhitespace('`'));
TEST(CharInfoTest, toUppercase) {
 
   EXPECT_EQ('.', toUppercase('.'));
   EXPECT_EQ('_', toUppercase('_'));
 
   EXPECT_EQ('/', toUppercase('/'));
   EXPECT_EQ('\0', toUppercase('\0'));
 }
 
-TEST(CharInfoTest, isValidIdentifier) {
-  EXPECT_FALSE(isValidIdentifier(""));
+TEST(CharInfoTest, isValidAsciiIdentifier) {
+  EXPECT_FALSE(isValidAsciiIdentifier(""));
 
   // 1 character
-  EXPECT_FALSE(isValidIdentifier("."));
-  EXPECT_FALSE(isValidIdentifier("\n"));
-  EXPECT_FALSE(isValidIdentifier(" "));
-  EXPECT_FALSE(isValidIdentifier("\x80"));
-  EXPECT_FALSE(isValidIdentifier("\xc2"));
-  EXPECT_FALSE(isValidIdentifier("\xff"));
-  EXPECT_FALSE(isValidIdentifier("$"));
-  EXPECT_FALSE(isValidIdentifier("1"));
-
-  EXPECT_TRUE(isValidIdentifier("_"));
-  EXPECT_TRUE(isValidIdentifier("a"));
-  EXPECT_TRUE(isValidIdentifier("z"));
-  EXPECT_TRUE(isValidIdentifier("A"));
-  EXPECT_TRUE(isValidIdentifier("Z"));
-  EXPECT_TRUE(isValidIdentifier("$", /*AllowDollar=*/true));
+  EXPECT_FALSE(isValidAsciiIdentifier("."));
+  EXPECT_FALSE(isValidAsciiIdentifier("\n"));
+  EXPECT_FALSE(isValidAsciiIdentifier(" "));
+  EXPECT_FALSE(isValidAsciiIdentifier("\x80"));
+  EXPECT_FALSE(isValidAsciiIdentifier("\xc2"));
+  EXPECT_FALSE(isValidAsciiIdentifier("\xff"));
+  EXPECT_FALSE(isValidAsciiIdentifier("$"));
+  EXPECT_FALSE(isValidAsciiIdentifier("1"));
+
+  EXPECT_TRUE(isValidAsciiIdentifier("_"));
+  EXPECT_TRUE(isValidAsciiIdentifier("a"));
+  EXPECT_TRUE(isValidAsciiIdentifier("z"));
+  EXPECT_TRUE(isValidAsciiIdentifier("A"));
+  EXPECT_TRUE(isValidAsciiIdentifier("Z"));
+  EXPECT_TRUE(isValidAsciiIdentifier("$", /*AllowDollar=*/true));
 
   // 2 characters, '_' suffix
-  EXPECT_FALSE(isValidIdentifier("._"));
-  EXPECT_FALSE(isValidIdentifier("\n_"));
-  EXPECT_FALSE(isValidIdentifier(" _"));
-  EXPECT_FALSE(isValidIdentifier("\x80_"));
-  EXPECT_FALSE(isValidIdentifier("\xc2_"));
-  EXPECT_FALSE(isValidIdentifier("\xff_"));
-  EXPECT_FALSE(isValidIdentifier("$_"));
-  EXPECT_FALSE(isValidIdentifier("1_"));
-
-  EXPECT_TRUE(isValidIdentifier("__"));
-  EXPECT_TRUE(isValidIdentifier("a_"));
-  EXPECT_TRUE(isValidIdentifier("z_"));
-  EXPECT_TRUE(isValidIdentifier("A_"));
-  EXPECT_TRUE(isValidIdentifier("Z_"));
-  EXPECT_TRUE(isValidIdentifier("$_", /*AllowDollar=*/true));
+  EXPECT_FALSE(isValidAsciiIdentifier("._"));
+  EXPECT_FALSE(isValidAsciiIdentifier("\n_"));
+  EXPECT_FALSE(isValidAsciiIdentifier(" _"));
+  EXPECT_FALSE(isValidAsciiIdentifier("\x80_"));
+  EXPECT_FALSE(isValidAsciiIdentifier("\xc2_"));
+  EXPECT_FALSE(isValidAsciiIdentifier("\xff_"));
+  EXPECT_FALSE(isValidAsciiIdentifier("$_"));
+  EXPECT_FALSE(isValidAsciiIdentifier("1_"));
+
+  EXPECT_TRUE(isValidAsciiIdentifier("__"));
+  EXPECT_TRUE(isValidAsciiIdentifier("a_"));
+  EXPECT_TRUE(isValidAsciiIdentifier("z_"));
+  EXPECT_TRUE(isValidAsciiIdentifier("A_"));
+  EXPECT_TRUE(isValidAsciiIdentifier("Z_"));
+  EXPECT_TRUE(isValidAsciiIdentifier("$_", /*AllowDollar=*/true));
 
   // 2 characters, '_' prefix
-  EXPECT_FALSE(isValidIdentifier("_."));
-  EXPECT_FALSE(isValidIdentifier("_\n"));
-  EXPECT_FALSE(isValidIdentifier("_ "));
-  EXPECT_FALSE(isValidIdentifier("_\x80"));
-  EXPECT_FALSE(isValidIdentifier("_\xc2"));
-  EXPECT_FALSE(isValidIdentifier("_\xff"));
-  EXPECT_FALSE(isValidIdentifier("_$"));
-  EXPECT_TRUE(isValidIdentifier("_1"));
-
-  EXPECT_TRUE(isValidIdentifier("__"));
-  EXPECT_TRUE(isValidIdentifier("_a"));
-  EXPECT_TRUE(isValidIdentifier("_z"));
-  EXPECT_TRUE(isValidIdentifier("_A"));
-  EXPECT_TRUE(isValidIdentifier("_Z"));
-  EXPECT_TRUE(isValidIdentifier("_$", /*AllowDollar=*/true));
+  EXPECT_FALSE(isValidAsciiIdentifier("_."));
+  EXPECT_FALSE(isValidAsciiIdentifier("_\n"));
+  EXPECT_FALSE(isValidAsciiIdentifier("_ "));
+  EXPECT_FALSE(isValidAsciiIdentifier("_\x80"));
+  EXPECT_FALSE(isValidAsciiIdentifier("_\xc2"));
+  EXPECT_FALSE(isValidAsciiIdentifier("_\xff"));
+  EXPECT_FALSE(isValidAsciiIdentifier("_$"));
+  EXPECT_TRUE(isValidAsciiIdentifier("_1"));
+
+  EXPECT_TRUE(isValidAsciiIdentifier("__"));
+  EXPECT_TRUE(isValidAsciiIdentifier("_a"));
+  EXPECT_TRUE(isValidAsciiIdentifier("_z"));
+  EXPECT_TRUE(isValidAsciiIdentifier("_A"));
+  EXPECT_TRUE(isValidAsciiIdentifier("_Z"));
+  EXPECT_TRUE(isValidAsciiIdentifier("_$", /*AllowDollar=*/true));
 
   // 3 characters, '__' prefix
-  EXPECT_FALSE(isValidIdentifier("__."));
-  EXPECT_FALSE(isValidIdentifier("__\n"));
-  EXPECT_FALSE(isValidIdentifier("__ "));
-  EXPECT_FALSE(isValidIdentifier("__\x80"));
-  EXPECT_FALSE(isValidIdentifier("__\xc2"));
-  EXPECT_FALSE(isValidIdentifier("__\xff"));
-  EXPECT_FALSE(isValidIdentifier("__$"));
-  EXPECT_TRUE(isValidIdentifier("__1"));
-
-  EXPECT_TRUE(isValidIdentifier("___"));
-  EXPECT_TRUE(isValidIdentifier("__a"));
-  EXPECT_TRUE(isValidIdentifier("__z"));
-  EXPECT_TRUE(isValidIdentifier("__A"));
-  EXPECT_TRUE(isValidIdentifier("__Z"));
-  EXPECT_TRUE(isValidIdentifier("__$", /*AllowDollar=*/true));
+  EXPECT_FALSE(isValidAsciiIdentifier("__."));
+  EXPECT_FALSE(isValidAsciiIdentifier("__\n"));
+  EXPECT_FALSE(isValidAsciiIdentifier("__ "));
+  EXPECT_FALSE(isValidAsciiIdentifier("__\x80"));
+  EXPECT_FALSE(isValidAsciiIdentifier("__\xc2"));
+  EXPECT_FALSE(isValidAsciiIdentifier("__\xff"));
+  EXPECT_FALSE(isValidAsciiIdentifier("__$"));
+  EXPECT_TRUE(isValidAsciiIdentifier("__1"));
+
+  EXPECT_TRUE(isValidAsciiIdentifier("___"));
+  EXPECT_TRUE(isValidAsciiIdentifier("__a"));
+  EXPECT_TRUE(isValidAsciiIdentifier("__z"));
+  EXPECT_TRUE(isValidAsciiIdentifier("__A"));
+  EXPECT_TRUE(isValidAsciiIdentifier("__Z"));
+  EXPECT_TRUE(isValidAsciiIdentifier("__$", /*AllowDollar=*/true));
 
   // 3 characters, '_' prefix and suffix
-  EXPECT_FALSE(isValidIdentifier("_._"));
-  EXPECT_FALSE(isValidIdentifier("_\n_"));
-  EXPECT_FALSE(isValidIdentifier("_ _"));
-  EXPECT_FALSE(isValidIdentifier("_\x80_"));
-  EXPECT_FALSE(isValidIdentifier("_\xc2_"));
-  EXPECT_FALSE(isValidIdentifier("_\xff_"));
-  EXPECT_FALSE(isValidIdentifier("_$_"));
-  EXPECT_TRUE(isValidIdentifier("_1_"));
-
-  EXPECT_TRUE(isValidIdentifier("___"));
-  EXPECT_TRUE(isValidIdentifier("_a_"));
-  EXPECT_TRUE(isValidIdentifier("_z_"));
-  EXPECT_TRUE(isValidIdentifier("_A_"));
-  EXPECT_TRUE(isValidIdentifier("_Z_"));
-  EXPECT_TRUE(isValidIdentifier("_$_", /*AllowDollar=*/true));
+  EXPECT_FALSE(isValidAsciiIdentifier("_._"));
+  EXPECT_FALSE(isValidAsciiIdentifier("_\n_"));
+  EXPECT_FALSE(isValidAsciiIdentifier("_ _"));
+  EXPECT_FALSE(isValidAsciiIdentifier("_\x80_"));
+  EXPECT_FALSE(isValidAsciiIdentifier("_\xc2_"));
+  EXPECT_FALSE(isValidAsciiIdentifier("_\xff_"));
+  EXPECT_FALSE(isValidAsciiIdentifier("_$_"));
+  EXPECT_TRUE(isValidAsciiIdentifier("_1_"));
+
+  EXPECT_TRUE(isValidAsciiIdentifier("___"));
+  EXPECT_TRUE(isValidAsciiIdentifier("_a_"));
+  EXPECT_TRUE(isValidAsciiIdentifier("_z_"));
+  EXPECT_TRUE(isValidAsciiIdentifier("_A_"));
+  EXPECT_TRUE(isValidAsciiIdentifier("_Z_"));
+  EXPECT_TRUE(isValidAsciiIdentifier("_$_", /*AllowDollar=*/true));
 }