[Lex] Warn about invisible Hangul whitespace

[modocache]

Summary

On Twitter @LunarLambda pointed out that Clang allows Hangul whitespace Unicode
characters in identifiers, which allows users to write very confusing
programs: https://twitter.com/LunarLambda/status/1110097030423240705

Clang warns about similar whitespace Unicode characters. Add the Hangul
half-width and full-width whitespace characters to the set that Clang
warns about.

N.B.: Clang warns about Japanese space character <U+3000>, but in a
different way, because that character is not a valid identifier
character according to the C++11 standard. So Clang emits a warning that
it will treat the Japanese <U+3000> as whitespace. This is different
from the Korean Hangul whitespace character, which is a valid identifier
character according to the C++11 standard. For this reason, Clang warns
the character will be treated as an identifier character, not as a
whitespace character -- so in sum, Clang's behavior is slightly
different for the Japanese whitespace character compared to the Korean
Hangul one.

[modocache]

Remove unneeded change to test identifier 'xx'.

[ruiu]

I wonder if we should handle Unicode codepoints that are in the whitespace category as a whole, instead of handling each codepoint individually.

[modocache]

I'm not super interested in this patch anymore, someone else feel free to work on this! :)

lib/Lex/Lexer.cpp

static constexpr HomoglyphPair SortedHomoglyphs[] = {
     {U'\u2212', '-'}, // MINUS SIGN
     {U'\u2215', '/'}, // DIVISION SLASH
     {U'\u2216', '\\'}, // SET MINUS
     {U'\u2217', '*'}, // ASTERISK OPERATOR
     {U'\u2223', '|'}, // DIVIDES
     {U'\u2227', '^'}, // LOGICAL AND
     {U'\u2236', ':'}, // RATIO
     {U'\u223c', '~'}, // TILDE OPERATOR
+    {U'\u3164', 0},   // HANGUL FILLER
     {U'\ua789', ':'}, // MODIFIER LETTER COLON
     {U'\ufeff', 0},   // ZERO WIDTH NO-BREAK SPACE
     {U'\uff01', '!'}, // FULLWIDTH EXCLAMATION MARK
     {U'\uff03', '#'}, // FULLWIDTH NUMBER SIGN
     {U'\uff04', '$'}, // FULLWIDTH DOLLAR SIGN
     {U'\uff05', '%'}, // FULLWIDTH PERCENT SIGN
     {U'\uff06', '&'}, // FULLWIDTH AMPERSAND
     {U'\uff08', '('}, // FULLWIDTH LEFT PARENTHESIS
static constexpr HomoglyphPair SortedHomoglyphs[] = {
     {U'\uff3b', '['}, // FULLWIDTH LEFT SQUARE BRACKET
     {U'\uff3c', '\\'}, // FULLWIDTH REVERSE SOLIDUS
     {U'\uff3d', ']'}, // FULLWIDTH RIGHT SQUARE BRACKET
     {U'\uff3e', '^'}, // FULLWIDTH CIRCUMFLEX ACCENT
     {U'\uff5b', '{'}, // FULLWIDTH LEFT CURLY BRACKET
     {U'\uff5c', '|'}, // FULLWIDTH VERTICAL LINE
     {U'\uff5d', '}'}, // FULLWIDTH RIGHT CURLY BRACKET
     {U'\uff5e', '~'}, // FULLWIDTH TILDE
+    {U'\uffa0', 0},   // HALFWIDTH HANGUL FILLER
     {0, 0}
   };
   auto Homoglyph =
       std::lower_bound(std::begin(SortedHomoglyphs),
                        std::end(SortedHomoglyphs) - 1, HomoglyphPair{C, '\0'});
   if (Homoglyph->Character == C) {
     llvm::SmallString<5> CharBuf;
     {

test/Lexer/unicode.c

int main () {
   return 🌷;
 }
 
 int n; = 3; // expected-warning {{treating Unicode character <U+037E> as identifier character rather than as ';' symbol}}
 int *n꞉꞉v = &n;; // expected-warning 2{{treating Unicode character <U+A789> as identifier character rather than as ':' symbol}}
                  // expected-warning@-1 {{treating Unicode character <U+037E> as identifier character rather than as ';' symbol}}
 int v＝［＝］（auto）｛return～x；｝（）; // expected-warning 12{{treating Unicode character}}
 
-int ⁠xx‍;
+int ⁠xx‍ㅤﾠ;
 // expected-warning@-1 {{identifier contains Unicode character <U+2060> that is invisible in some environments}}
 // expected-warning@-2 {{identifier contains Unicode character <U+FEFF> that is invisible in some environments}}
 // expected-warning@-3 {{identifier contains Unicode character <U+200D> that is invisible in some environments}}
+// expected-warning@-4 {{identifier contains Unicode character <U+3164> that is invisible in some environments}}
+// expected-warning@-5 {{identifier contains Unicode character <U+FFA0> that is invisible in some environments}}
 int foo!bar = 0; // expected-warning {{identifier contains Unicode character <U+200B> that is invisible in some environments}}
 int x = foobar; // expected-error {{undeclared identifier}}
 
 int ∣foo; // expected-error {{non-ASCII character}}
 #ifndef PP_ONLY
 #define ∶ x // expected-error {{macro name must be an identifier}}
 #endif