Implement delimited escape sequences.

[cor3ntin]

Summary

This is a proposed C++ paper (P2290R1) that has not been accepted yet

\x{XXXX} \u{XXXX} and \o{OOOO} are accepted in all languages mode
in characters and string literals.

[cor3ntin]

Fix tests & build

[aaron.ballman]

Because this proposal hasn't been accepted by WG21 or WG14, but the syntax is not defined by either standard, I think it's reasonable to support this but with appropriate warnings about it being a Clang extension. Can you add those diagnostics? Also, this is lacking test coverage for C.

clang/include/clang/Basic/DiagnosticLexKinds.td

131

Wouldn't this always be covered by expected '}'?

133

Empty isn't the only problem though -- what happens if the user specifies too many digits? e.g. \xFFFFFFFFFFFFFFFFF (that's 0xFFFF'FFFF'FFFF'FFFF'F)

135

clang/lib/Lex/LiteralSupport.cpp

151–156

Can't this be determined once we've read the } by looking at the string length?

154

This argument doesn't do anything, btw. Same happens in a few other places.

clang/test/Lexer/char-escapes-delimited.cpp

6 ↗

(On Diff #359544)

This diagnostic highlights the need to tell the user *which* character in the escape sequence is wrong. Consider: \u{0000O111I} as an example.

14 ↗

(On Diff #359544)

Can you add test cases that show whether digit separators are allowed or prohibited? (FWIW, I sort of think they should be allowed -- at least, I'm not certain I see a reason to disallow them and they do help with readability, especially for longer hex escapes.)

[cor3ntin]

clang/include/clang/Basic/DiagnosticLexKinds.td

131

I guess, I didn't think of that

133

That's covered by existing errors for overflowing the char type. There is no limit in the number of digits in the escape per say.

clang/lib/Lex/LiteralSupport.cpp

154

Indeed!

clang/test/Lexer/char-escapes-delimited.cpp

6 ↗

(On Diff #359544)

Make sense!

[cor3ntin]

• Format
• Add warning for extension
• Add negative tests for delimited sequences split

between 2 concatenated strings, and sequences containing digit
separators

• Improve diagnostic messages for invalid characters
• Stop passing unused parameters to diagnostics

clang/lib/Lex/LiteralSupport.cpp

151–156

I suppose we could. Do you think it would be clearer?

[aaron.ballman]

clang/include/clang/Basic/DiagnosticLexKinds.td

130–131

133

Ah, good to know, thanks!

clang/lib/Lex/LiteralSupport.cpp

151–156

I thought it could be cleaner if we also had to check lengths, but because we don't, I think this is fine as-is.

173

Comment here explaining why we're breaking when not delimited would be useful for other readers.

179

Can we pass the char directly, or get away with using a StringRef? It'd be nice to avoid the allocation from std::string.

289

Same here as above.

421

Same here as above.

497

This return looks incorrect because we'll miss diagnosing using the extension in this case even though the user used it.

clang/test/Lexer/char-escapes-delimited.c

44

Can you test that it's not out of range for L'\o{777}'?

60

I think this should be accepted rather than rejected. Long hex values are somewhat anticipated for this feature (given that you often need 5 or more characters to spell the interesting Unicode hex values), and digit separators help with readability in those cases. ... ... However, I suppose those *can't* work because of character literals. '\x{1234'5}' would be ambiguous to lex. Ouch, okay, so I guess I don't expect that to work, but having a test case with a character literal that tries to use a digit separator would be good.

[cor3ntin]

clang/lib/Lex/LiteralSupport.cpp

179

I can use StringRef yep - but I'm not concerned about allocation because of sbo - I used a pattern used elsewhere in the file

497

Indeed!

[cor3ntin]

• Add tests for short wchar overflow
• use string ref instead of string
• fix diagnostic

[cor3ntin]

Format

[cor3ntin]

• Fix tests

[cor3ntin]

• Add comment to explain why we break out during hex escape parsing

[cor3ntin]

• Fix diagnostic message

[cor3ntin]

clang/test/Lexer/char-escapes-delimited.c

60

I mean, that just explodes

[aaron.ballman]

clang/test/Lexer/char-escapes-delimited.c

79–81

I think the test is a bit off because you couldn't have a trailing digit separator so 12'3' wouldn't be valid anyway. What's the behavior if you use 12'3 instead?

[cor3ntin]

• Add delimited escape sequences in identifiers

[cor3ntin]

When preprocessing, treat incomplete or empty
delimited UCNs as warning, in the spirit of
[lex.pptoken]/3.1

Overflow is still treated as an error.

[aaron.ballman]

Mostly some small nits.

clang/include/clang/Basic/DiagnosticLexKinds.td

134

This describes what the code is, but not why it's an issue. How about: delimited escape sequence cannot be empty or something along those lines?

146–151

I think we can get away with only one diagnostic here.

clang/lib/Lex/Lexer.cpp

3091

3129

Should fix this formatting issue. You can also elide the braces per our usual coding conventions.

3136

Should this be using Diagnose instead?

3153

3174

clang/lib/Lex/LiteralSupport.cpp

494–501

499–502

[cor3ntin]

Address most of the comments

I left warn_delimited_ucn and err_ucn_escape_incomplete
as separate warnings as I think it's easier to understand,
and could not find a way to escape a closing brace inside
a %select{}.

[cor3ntin]

Fix tests

[aaron.ballman]

In D105737#2891349, @aaron.ballman wrote:

Because this proposal hasn't been accepted by WG21 or WG14, but the syntax is not defined by either standard, I think it's reasonable to support this but with appropriate warnings about it being a Clang extension. Can you add those diagnostics? Also, this is lacking test coverage for C.

This proposal has now been seen by both WG14 and WG21 and has sentiment for support in both committees. I think the extension diagnostic is still reasonable (we'll want to support this syntax in older language modes when both committees adopt it for a newer language mode).

LGTM, I'm down to only some minor nits with the patch. @rsmith, do you have any concerns you'd like to see addressed before landing this?

clang/include/clang/Basic/DiagnosticLexKinds.td

130

We usually capitalize Clang for this kind of diagnostic.

clang/lib/Lex/Lexer.cpp

3174

clang/lib/Lex/LiteralSupport.cpp

166

337–352

Better to use something that can't be confused with a type name (clang::Type), also drops the top-level const.

[cor3ntin]

In D105737#2989571, @aaron.ballman wrote:

In D105737#2891349, @aaron.ballman wrote:

Because this proposal hasn't been accepted by WG21 or WG14, but the syntax is not defined by either standard, I think it's reasonable to support this but with appropriate warnings about it being a Clang extension. Can you add those diagnostics? Also, this is lacking test coverage for C.

This proposal has now been seen by both WG14 and WG21 and has sentiment for support in both committees. I think the extension diagnostic is still reasonable (we'll want to support this syntax in older language modes when both committees adopt it for a newer language mode).

LGTM, I'm down to only some minor nits with the patch. @rsmith, do you have any concerns you'd like to see addressed before landing this?

Just to clarify, I don't think this should be allowed in C++20 or C17.
Only C23 and C++23 modes. It make sense to add an extension warning there until the proposal are approved imo

Do you agree?

[aaron.ballman]

In D105737#2989591, @cor3ntin wrote:

In D105737#2989571, @aaron.ballman wrote:

In D105737#2891349, @aaron.ballman wrote:

Because this proposal hasn't been accepted by WG21 or WG14, but the syntax is not defined by either standard, I think it's reasonable to support this but with appropriate warnings about it being a Clang extension. Can you add those diagnostics? Also, this is lacking test coverage for C.

This proposal has now been seen by both WG14 and WG21 and has sentiment for support in both committees. I think the extension diagnostic is still reasonable (we'll want to support this syntax in older language modes when both committees adopt it for a newer language mode).

LGTM, I'm down to only some minor nits with the patch. @rsmith, do you have any concerns you'd like to see addressed before landing this?

Just to clarify, I don't think this should be allowed in C++20 or C17.
Only C23 and C++23 modes. It make sense to add an extension warning there until the proposal are approved imo

Do you agree?

I could agree, but I'm wondering why you don't want to allow in older language modes? Clang typically allows conforming extensions in older language modes (with appropriate diagnostics), and this would qualify. e.g., https://godbolt.org/z/x5Ws6YdGc

[cor3ntin]

Address Aaron's comments

[cor3ntin]

Rebase

[cor3ntin]

Fix "clang" case in tests

[aaron.ballman]

Thank you for the new feature! I've landed on your behalf in 274adcb866343cb505408d8f401840ea814522c8.

clang/include/clang/Basic/DiagnosticLexKinds.td

def ext_unicode_whitespace : ExtWarn<
   InGroup<DiagGroup<"unicode-whitespace">>;
 def warn_utf8_symbol_homoglyph : Warning<
   "treating Unicode character <U+%0> as identifier character rather than "
   "as '%1' symbol">, InGroup<DiagGroup<"unicode-homoglyph">>;
 def warn_utf8_symbol_zero_width : Warning<
   "identifier contains Unicode character <U+%0> that is invisible in "
   "some environments">, InGroup<DiagGroup<"unicode-zero-width">>;
 
+def ext_delimited_escape_sequence : Extension<
+  "delimited escape sequences are a Clang extension">,
+  InGroup<DiagGroup<"delimited-escape-sequence-extension">>;
+def err_delimited_escape_empty : Error<
+  "delimited escape sequence cannot be empty">;
+def err_delimited_escape_missing_brace: Error<
+  "expected '{' after '\\%0' escape sequence">;
+def err_delimited_escape_invalid : Error<
+  "invalid digit '%0' in escape sequence">;
 def err_hex_escape_no_digits : Error<
   "\\%0 used with no following hex digits">;
 def warn_ucn_escape_no_digits : Warning<
   "\\%0 used with no following hex digits; "
   "treating as '\\' followed by identifier">, InGroup<Unicode>;
 def err_ucn_escape_incomplete : Error<
   "incomplete universal character name">;
+def warn_delimited_ucn_incomplete : Warning<
+  "incomplete delimited universal character name; "
+  "treating as '\\' 'u' '{' identifier">, InGroup<Unicode>;
+def warn_delimited_ucn_empty : Warning<
+  "empty delimited universal character name; "
+  "treating as '\\' 'u' '{' '}'">, InGroup<Unicode>;
 def warn_ucn_escape_incomplete : Warning<
   "incomplete universal character name; "
   "treating as '\\' followed by identifier">, InGroup<Unicode>;
 def note_ucn_four_not_eight : Note<"did you mean to use '\\u'?">;
 
 def err_ucn_escape_basic_scs : Error<
   "character '%0' cannot be specified by a universal character name">;
 def err_ucn_control_character : Error<

clang/lib/Lex/Lexer.cpp

bool Lexer::isCodeCompletionPoint(const char *CurPtr) const {
 
   return false;
 }
 
 uint32_t Lexer::tryReadUCN(const char *&StartPtr, const char *SlashLoc,
                            Token *Result) {
   unsigned CharSize;
   char Kind = getCharAndSize(StartPtr, CharSize);
+  bool Delimited = false;
+  bool FoundEndDelimiter = false;
+  unsigned Count = 0;
+  bool Diagnose = Result && !isLexingRawMode();
 
   unsigned NumHexDigits;
   if (Kind == 'u')
     NumHexDigits = 4;
   else if (Kind == 'U')
     NumHexDigits = 8;
   else
     return 0;
 
   if (!LangOpts.CPlusPlus && !LangOpts.C99) {
-    if (Result && !isLexingRawMode())
+    if (Diagnose)
       Diag(SlashLoc, diag::warn_ucn_not_valid_in_c89);
     return 0;
   }
 
   const char *CurPtr = StartPtr + CharSize;
   const char *KindLoc = &CurPtr[-1];
 
   uint32_t CodePoint = 0;
-  for (unsigned i = 0; i < NumHexDigits; ++i) {
+  while (Count != NumHexDigits || Delimited) {
     char C = getCharAndSize(CurPtr, CharSize);
+    if (!Delimited && C == '{') {
+      Delimited = true;
+      CurPtr += CharSize;
+      continue;
+    }
+
+    if (Delimited && C == '}') {
+      CurPtr += CharSize;
+      FoundEndDelimiter = true;
+      break;
+    }
 
     unsigned Value = llvm::hexDigitValue(C);
     if (Value == -1U) {
-      if (Result && !isLexingRawMode()) {
-        if (i == 0) {
-          Diag(BufferPtr, diag::warn_ucn_escape_no_digits)
-            << StringRef(KindLoc, 1);
-        } else {
-          Diag(BufferPtr, diag::warn_ucn_escape_incomplete);
+      if (!Delimited)
+        break;
+      if (Diagnose)
+        Diag(BufferPtr, diag::warn_delimited_ucn_incomplete)
+            << StringRef(&C, 1);
+      return 0;
+    }
+
+    if (CodePoint & 0xF000'0000) {
+      if (Diagnose)
+        Diag(KindLoc, diag::err_escape_too_large) << 0;
+      return 0;
+    }
+
+    CodePoint <<= 4;
+    CodePoint |= Value;
+    CurPtr += CharSize;
+    Count++;
+  }
+
+  if (Count == 0) {
+    if (Diagnose)
+      Diag(StartPtr, FoundEndDelimiter ? diag::warn_delimited_ucn_empty
+                                       : diag::warn_ucn_escape_no_digits)
+    return 0;
+  }
 
-          // If the user wrote \U1234, suggest a fixit to \u.
-          if (i == 4 && NumHexDigits == 8) {
-            CharSourceRange URange = makeCharRange(*this, KindLoc, KindLoc + 1);
-            Diag(KindLoc, diag::note_ucn_four_not_eight)
-              << FixItHint::CreateReplacement(URange, "u");
-          }
-        }
-      }
-
-      return 0;
-    }
+  if (!Delimited && Count != NumHexDigits) {
+    if (Diagnose) {
+      Diag(BufferPtr, diag::warn_ucn_escape_incomplete);
+      if (Count == 4 && NumHexDigits == 8) {
 
-    CodePoint <<= 4;
-    CodePoint += Value;
-
-    CurPtr += CharSize;
+  if (Delimited && PP) {
+    Diag(BufferPtr, diag::ext_delimited_escape_sequence);
   }
 
   if (Result) {
     Result->setFlag(Token::HasUCN);
-    if (CurPtr - StartPtr == (ptrdiff_t)NumHexDigits + 2)
+    if (CurPtr - StartPtr == (ptrdiff_t)(Count + 2 + (Delimited ? 2 : 0)))
       StartPtr = CurPtr;
     else
       while (StartPtr != CurPtr)
         (void)getAndAdvanceChar(StartPtr, *Result);
   } else {
     StartPtr = CurPtr;
   }
 

clang/lib/Lex/LiteralSupport.cpp

 /// either a character or a string literal.
 static unsigned ProcessCharEscape(const char *ThisTokBegin,
                                   const char *&ThisTokBuf,
                                   const char *ThisTokEnd, bool &HadError,
                                   FullSourceLoc Loc, unsigned CharWidth,
                                   DiagnosticsEngine *Diags,
                                   const LangOptions &Features) {
   const char *EscapeBegin = ThisTokBuf;
+  bool Delimited = false;
+  bool EndDelimiterFound = false;
 
   // Skip the '\' char.
   ++ThisTokBuf;
 
   // We know that this character can't be off the end of the buffer, because
   // that would have been \", which would not have been the end of string.
   unsigned ResultChar = *ThisTokBuf++;
   switch (ResultChar) {
static unsigned ProcessCharEscape(const char *ThisTokBegin,
   case 't':
     ResultChar = 9;
     break;
   case 'v':
     ResultChar = 11;
     break;
   case 'x': { // Hex escape.
     ResultChar = 0;
-    if (ThisTokBuf == ThisTokEnd || !isHexDigit(*ThisTokBuf)) {
+    if (ThisTokBuf != ThisTokEnd && *ThisTokBuf == '{') {
+      Delimited = true;
+      ThisTokBuf++;
+      if (*ThisTokBuf == '}') {
+        Diag(Diags, Features, Loc, ThisTokBegin, EscapeBegin, ThisTokBuf,
+             diag::err_delimited_escape_empty);
+        return ResultChar;
+      }
+    } else if (ThisTokBuf == ThisTokEnd || !isHexDigit(*ThisTokBuf)) {
       if (Diags)
         Diag(Diags, Features, Loc, ThisTokBegin, EscapeBegin, ThisTokBuf,
              diag::err_hex_escape_no_digits) << "x";
-      HadError = true;
-      break;
+      return ResultChar;
     }
 
     // Hex escapes are a maximal series of hex digits.
     bool Overflow = false;
     for (; ThisTokBuf != ThisTokEnd; ++ThisTokBuf) {
-      int CharVal = llvm::hexDigitValue(ThisTokBuf[0]);
-      if (CharVal == -1) break;
+      if (Delimited && *ThisTokBuf == '}') {
+        ThisTokBuf++;
+        EndDelimiterFound = true;
+        break;
+      }
+      int CharVal = llvm::hexDigitValue(*ThisTokBuf);
+      if (CharVal == -1) {
+        // Non delimited hex escape sequences stop at the first non-hex digit.
+        if (!Delimited)
+          break;
+        HadError = true;
+        if (Diags)
+          Diag(Diags, Features, Loc, ThisTokBegin, EscapeBegin, ThisTokBuf,
+               diag::err_delimited_escape_invalid)
+              << StringRef(ThisTokBuf, 1);
+        continue;
+      }
       // About to shift out a digit?
       if (ResultChar & 0xF0000000)
         Overflow = true;
       ResultChar <<= 4;
       ResultChar |= CharVal;
     }
-
     // See if any bits will be truncated when evaluated as a character.
     if (CharWidth != 32 && (ResultChar >> CharWidth) != 0) {
       Overflow = true;
       ResultChar &= ~0U >> (32-CharWidth);
     }
 
     // Check for overflow.
-    if (Overflow && Diags)   // Too many digits to fit in
+    if (!HadError && Overflow) { // Too many digits to fit in
+      HadError = true;
+      if (Diags)
         Diag(Diags, Features, Loc, ThisTokBegin, EscapeBegin, ThisTokBuf,
-           diag::err_escape_too_large) << 0;
+             diag::err_escape_too_large)
+            << 0;
+    }
     break;
   }
   case '0': case '1': case '2': case '3':
   case '4': case '5': case '6': case '7': {
     // Octal escapes.
     --ThisTokBuf;
     ResultChar = 0;
 
case '4': case '5': case '6': case '7': {
     if (CharWidth != 32 && (ResultChar >> CharWidth) != 0) {
       if (Diags)
         Diag(Diags, Features, Loc, ThisTokBegin, EscapeBegin, ThisTokBuf,
              diag::err_escape_too_large) << 1;
       ResultChar &= ~0U >> (32-CharWidth);
     }
     break;
   }
+  case 'o': {
+    bool Overflow = false;
+    if (ThisTokBuf == ThisTokEnd || *ThisTokBuf != '{') {
+      HadError = true;
+      if (Diags)
+        Diag(Diags, Features, Loc, ThisTokBegin, EscapeBegin, ThisTokBuf,
+             diag::err_delimited_escape_missing_brace);
+
+      break;
+    }
+    ResultChar = 0;
+    Delimited = true;
+    ++ThisTokBuf;
+    if (*ThisTokBuf == '}') {
+      Diag(Diags, Features, Loc, ThisTokBegin, EscapeBegin, ThisTokBuf,
+           diag::err_delimited_escape_empty);
+      return ResultChar;
+    }
+
+    while (ThisTokBuf != ThisTokEnd) {
+      if (*ThisTokBuf == '}') {
+        EndDelimiterFound = true;
+        ThisTokBuf++;
+        break;
+      }
+      if (*ThisTokBuf < '0' || *ThisTokBuf > '7') {
+        HadError = true;
+        if (Diags)
+          Diag(Diags, Features, Loc, ThisTokBegin, EscapeBegin, ThisTokBuf,
+               diag::err_delimited_escape_invalid)
+              << StringRef(ThisTokBuf, 1);
+        ThisTokBuf++;
+        continue;
+      }
+      if (ResultChar & 0x020000000)
+        Overflow = true;
 
+      ResultChar <<= 3;
+      ResultChar |= *ThisTokBuf++ - '0';
+    }
+    // Check for overflow.  Reject '\777', but not L'\777'.
+    if (!HadError &&
+        (Overflow || (CharWidth != 32 && (ResultChar >> CharWidth) != 0))) {
+      HadError = true;
+      if (Diags)
+        Diag(Diags, Features, Loc, ThisTokBegin, EscapeBegin, ThisTokBuf,
+             diag::err_escape_too_large)
+            << 1;
+      ResultChar &= ~0U >> (32 - CharWidth);
+    }
+    break;
+  }
     // Otherwise, these are not valid escapes.
   case '(': case '{': case '[': case '%':
     // GCC accepts these as extensions.  We warn about them as such though.
     if (Diags)
       Diag(Diags, Features, Loc, ThisTokBegin, EscapeBegin, ThisTokBuf,
            diag::ext_nonstandard_escape)
         << std::string(1, ResultChar);
     break;
   default:
     if (!Diags)
       break;
 
     if (isPrintable(ResultChar))
       Diag(Diags, Features, Loc, ThisTokBegin, EscapeBegin, ThisTokBuf,
            diag::ext_unknown_escape)
         << std::string(1, ResultChar);
     else
       Diag(Diags, Features, Loc, ThisTokBegin, EscapeBegin, ThisTokBuf,
            diag::ext_unknown_escape)
         << "x" + llvm::utohexstr(ResultChar);
     break;
   }
 
+  if (Delimited && Diags) {
+    if (!EndDelimiterFound)
+      Diag(Diags, Features, Loc, ThisTokBegin, EscapeBegin, ThisTokBuf,
+           diag::err_expected)
+          << tok::r_brace;
+    else if (!HadError) {
+      Diag(Diags, Features, Loc, ThisTokBegin, EscapeBegin, ThisTokBuf,
+           diag::ext_delimited_escape_sequence);
+    }
+  }
+
   return ResultChar;
 }
 
 static void appendCodePoint(unsigned Codepoint,
                             llvm::SmallVectorImpl<char> &Str) {
   char ResultBuf[4];
   char *ResultPtr = ResultBuf;
   bool Res = llvm::ConvertCodePointToUTF8(Codepoint, ResultPtr);
   (void)Res;
   assert(Res && "Unexpected conversion failure");
   Str.append(ResultBuf, ResultPtr);
 }
 
 void clang::expandUCNs(SmallVectorImpl<char> &Buf, StringRef Input) {
   for (StringRef::iterator I = Input.begin(), E = Input.end(); I != E; ++I) {
     if (*I != '\\') {
       Buf.push_back(*I);
       continue;
     }
 
     ++I;
-    assert(*I == 'u' || *I == 'U');
+    char Kind = *I;
+    ++I;
+
+    assert(Kind == 'u' || Kind == 'U');
+    uint32_t CodePoint = 0;
+
+    if (Kind == 'u' && *I == '{') {
+      for (++I; *I != '}'; ++I) {
+        unsigned Value = llvm::hexDigitValue(*I);
+        assert(Value != -1U);
+        CodePoint <<= 4;
+        CodePoint += Value;
+      }
+      appendCodePoint(CodePoint, Buf);
+      continue;
+    }
 
     unsigned NumHexDigits;
-    if (*I == 'u')
+    if (Kind == 'u')
       NumHexDigits = 4;
     else
       NumHexDigits = 8;
 
     assert(I + NumHexDigits <= E);
 
-    uint32_t CodePoint = 0;
-    for (++I; NumHexDigits != 0; ++I, --NumHexDigits) {
+    for (; NumHexDigits != 0; ++I, --NumHexDigits) {
       unsigned Value = llvm::hexDigitValue(*I);
       assert(Value != -1U);
 
       CodePoint <<= 4;
       CodePoint += Value;
     }
 
     appendCodePoint(CodePoint, Buf);
static bool ProcessUCNEscape(const char *ThisTokBegin, const char *&ThisTokBuf,
                              FullSourceLoc Loc, DiagnosticsEngine *Diags,
                              const LangOptions &Features,
                              bool in_char_string_literal = false) {
   const char *UcnBegin = ThisTokBuf;
 
   // Skip the '\u' char's.
   ThisTokBuf += 2;
 
-  if (ThisTokBuf == ThisTokEnd || !isHexDigit(*ThisTokBuf)) {
+  bool Delimited = false;
+  bool EndDelimiterFound = false;
+  bool HasError = false;
+
+  if (UcnBegin[1] == 'u' && in_char_string_literal &&
+      ThisTokBuf != ThisTokEnd && *ThisTokBuf == '{') {
+    Delimited = true;
+    ThisTokBuf++;
+  } else if (ThisTokBuf == ThisTokEnd || !isHexDigit(*ThisTokBuf)) {
     if (Diags)
       Diag(Diags, Features, Loc, ThisTokBegin, UcnBegin, ThisTokBuf,
            diag::err_hex_escape_no_digits) << StringRef(&ThisTokBuf[-1], 1);
     return false;
   }
   UcnLen = (ThisTokBuf[-1] == 'u' ? 4 : 8);
-  unsigned short UcnLenSave = UcnLen;
-  for (; ThisTokBuf != ThisTokEnd && UcnLenSave; ++ThisTokBuf, UcnLenSave--) {
-    int CharVal = llvm::hexDigitValue(ThisTokBuf[0]);
-    if (CharVal == -1) break;
+
+  bool Overflow = false;
+  unsigned short Count = 0;
+  for (; ThisTokBuf != ThisTokEnd && (Delimited || Count != UcnLen);
+       ++ThisTokBuf) {
+    if (Delimited && *ThisTokBuf == '}') {
+      ++ThisTokBuf;
+      EndDelimiterFound = true;
+      break;
+    }
+    int CharVal = llvm::hexDigitValue(*ThisTokBuf);
+    if (CharVal == -1) {
+      HasError = true;
+      if (!Delimited)
+        break;
+      if (Diags) {
+        Diag(Diags, Features, Loc, ThisTokBegin, UcnBegin, ThisTokBuf,
+             diag::err_delimited_escape_invalid)
+            << StringRef(ThisTokBuf, 1);
+      }
+      Count++;
+      continue;
+    }
+    if (UcnVal & 0xF0000000) {
+      Overflow = true;
+      continue;
+    }
     UcnVal <<= 4;
     UcnVal |= CharVal;
+    Count++;
+  }
+
+  if (Overflow) {
+    if (Diags)
+      Diag(Diags, Features, Loc, ThisTokBegin, UcnBegin, ThisTokBuf,
+           diag::err_escape_too_large)
+          << 0;
+    return false;
+  }
+
+  if (Delimited && !EndDelimiterFound) {
+    if (Diags) {
+      Diag(Diags, Features, Loc, ThisTokBegin, UcnBegin, ThisTokBuf,
+           diag::err_expected)
+          << tok::r_brace;
+    }
+    return false;
   }
+
   // If we didn't consume the proper number of digits, there is a problem.
-  if (UcnLenSave) {
+  if (Count == 0 || (!Delimited && Count != UcnLen)) {
     if (Diags)
       Diag(Diags, Features, Loc, ThisTokBegin, UcnBegin, ThisTokBuf,
-           diag::err_ucn_escape_incomplete);
+           Delimited ? diag::err_delimited_escape_empty
+                     : diag::err_ucn_escape_incomplete);
     return false;
   }
 
+  if (HasError)
+    return false;
+
   // Check UCN constraints (C99 6.4.3p2) [C++11 lex.charset p2]
   if ((0xD800 <= UcnVal && UcnVal <= 0xDFFF) || // surrogate codepoints
       UcnVal > 0x10FFFF) {                      // maximum legal UTF32 value
     if (Diags)
       Diag(Diags, Features, Loc, ThisTokBegin, UcnBegin, ThisTokBuf,
            diag::err_ucn_escape_invalid);
     return false;
   }
if (Diags) {
         Diag(Diags, Features, Loc, ThisTokBegin, UcnBegin, ThisTokBuf,
              IsError ? diag::err_ucn_control_character :
                        diag::warn_cxx98_compat_literal_ucn_control_character);
     }
     if (IsError)
       return false;
   }
 
   if (!Features.CPlusPlus && !Features.C99 && Diags)
     Diag(Diags, Features, Loc, ThisTokBegin, UcnBegin, ThisTokBuf,
          diag::warn_ucn_not_valid_in_c89_literal);
 
+  if (Delimited && Diags)
+    Diag(Diags, Features, Loc, ThisTokBegin, UcnBegin, ThisTokBuf,
+         diag::ext_delimited_escape_sequence);
+
   return true;
 }
 
 /// MeasureUCNEscape - Determine the number of bytes within the resulting string
 /// which this UCN will occupy.
 static int MeasureUCNEscape(const char *ThisTokBegin, const char *&ThisTokBuf,
                             const char *ThisTokEnd, unsigned CharByteWidth,
                             const LangOptions &Features, bool &HadError) {
   // UTF-32: 4 bytes per escape.

clang/test/Lexer/char-escapes-delimited.c

+// RUN: %clang_cc1 -fsyntax-only -pedantic -verify %s
+// RUN: %clang_cc1 -x c -fsyntax-only -pedantic -verify %s
+// RUN: %clang_cc1 -fwchar-type=short -fno-signed-wchar -fsyntax-only -pedantic -verify %s
+// RUN: %clang_cc1 -x c -fwchar-type=short -fno-signed-wchar -fsyntax-only -pedantic -verify %s
+
+const char *errors =
+    "\u{}"  //expected-error {{delimited escape sequence cannot be empty}}
+    "\u{"   //expected-error {{expected '}'}}
+    "\u{h}" //expected-error {{invalid digit 'h' in escape sequence}}
+    "\x{}"  //expected-error {{delimited escape sequence cannot be empty}}
+    "\x{"   //expected-error {{expected '}'}}
+    "\x{h}" //expected-error {{invalid digit 'h' in escape sequence}}
+    "\o{}"  //expected-error {{delimited escape sequence cannot be empty}}
+    "\o{"   //expected-error {{expected '}'}}
+    "\o{8}" //expected-error {{invalid digit '8' in escape sequence}}
+    ;
+
+void ucn() {
+  char a = '\u{1234}'; // expected-error {{character too large for enclosing character literal type}}
+                       // expected-warning@-1 {{delimited escape sequences are a Clang extension}}
+
+  unsigned b = U'\u{1234}'; // expected-warning {{extension}}
+
+#ifdef __cplusplus
+  unsigned b2 = U'\u{1}'; // expected-warning {{extension}}
+#else
+  unsigned b2 = U'\u{1}';     //expected-error {{universal character name refers to a control character}}
+#endif
+
+  unsigned c = U'\u{000000000001234}'; // expected-warning {{extension}}
+  unsigned d = U'\u{111111111}';       //expected-error {{hex escape sequence out of range}}
+}
+
+void hex() {
+  char a = '\x{1}';             // expected-warning {{extension}}
+  char b = '\x{abcdegggggabc}'; // expected-error 5{{invalid digit 'g' in escape sequence}}
+  char c = '\x{ff1}';           // expected-error {{hex escape sequence out of range}}
+
+#if __WCHAR_MAX__ > 0xFFFF
+  unsigned d = L'\x{FFFFFFFF}';  // expected-warning {{extension}}
+  unsigned e = L'\x{100000000}'; // expected-error {{hex escape sequence out of range}}
+#else
+  unsigned f = L'\x{FFFF}';   // expected-warning {{extension}}
+  unsigned g = L'\x{10000}';  // expected-error {{hex escape sequence out of range}}
+#endif
+  unsigned h = U'\x{FFFFFFFF}';  // expected-warning {{extension}}
+  unsigned i = U'\x{100000000}'; // expected-error {{hex escape sequence out of range}}
+}
+
+void octal() {
+  char a = '\o{1}';              // expected-warning {{extension}}
+  char b = '\o{12345678881238}'; // expected-error 4{{invalid digit '8' in escape sequence}}
+  char c = '\o{777}';            // //expected-error {{octal escape sequence out of range}}
+#if __WCHAR_MAX__ > 0xFFFF
+  unsigned d = L'\o{37777777777}'; // expected-warning {{extension}}
+  unsigned e = L'\o{40000000000}'; // expected-error {{octal escape sequence out of range}}
+#else
+  unsigned d = L'\o{177777}'; // expected-warning {{extension}}
+  unsigned e = L'\o{200000}'; // expected-error {{octal escape sequence out of range}}
+#endif
+}
+
+void concat() {
+  (void)"\x{" "12}"; // expected-error {{expected '}'}}
+  (void)"\u{" "12}"; // expected-error {{expected '}'}}
+  (void)"\o{" "12}"; // expected-error {{expected '}'}}
+
+  (void)"\x{12" "}"; // expected-error {{expected '}'}}
+  (void)"\u{12" "}"; // expected-error {{expected '}'}}
+  (void)"\o{12" "}"; // expected-error {{expected '}'}}
+}
+
+void separators() {
+  (void)"\x{12'3}"; // expected-error {{invalid digit ''' in escape sequence}}
+  (void)"\u{12'3}"; // expected-error {{invalid digit ''' in escape sequence}}
+  (void)"\o{12'3}"; // expected-error {{invalid digit ''' in escape sequence}}
+
+  '\x{12'3'}';   // expected-error {{expected '}'}}
+                 // expected-error@-1 2{{expected ';'}}
+                 // expected-warning@-2 3{{expression result unused}}
+}

clang/test/Parser/cxx11-user-defined-literals.cpp

 int operator""_𐀀(char) {} // expected-note {{previous}}
 int 𐀀 = '4'_𐀀 + '2'_\U00010000;
 int operator""_\U00010000(char) {} // expected-error {{redefinition of 'operator""_𐀀'}}
 
 // These all declare the same function.
 int operator""_℮""_\u212e""_\U0000212e""(const char*, size_t);
 int operator""_\u212e""_\U0000212e""_℮""(const char*, size_t);
 int operator""_\U0000212e""_℮""_\u212e""(const char*, size_t);
+
+int operator""_\u{212f}(char);
+
 int mix_ucn_utf8 = ""_℮""_\u212e""_\U0000212e"";
 
 void operator""_℮""_ℯ(unsigned long long) {} // expected-error {{differing user-defined suffixes ('_℮' and '_ℯ') in string literal concatenation}}
 void operator""_℮""_\u212f(unsigned long long) {} // expected-error {{differing user-defined suffixes ('_℮' and '_ℯ') in string literal concatenation}}
 void operator""_\u212e""_ℯ(unsigned long long) {} // expected-error {{differing user-defined suffixes ('_℮' and '_ℯ') in string literal concatenation}}
 void operator""_\u212e""_\u212f(unsigned long long) {} // expected-error {{differing user-defined suffixes ('_℮' and '_ℯ') in string literal concatenation}}
 
 void operator""_℮""_℮(unsigned long long) {} // expected-note {{previous}}
 void operator""_\u212e""_\u212e(unsigned long long) {} // expected-error {{redefinition}}
 
 #define ¢ *0.01 // expected-error {{macro name must be an identifier}}
 constexpr int operator""_¢(long double d) { return d * 100; }  // expected-error {{character <U+00A2> not allowed in an identifier}}
 constexpr int operator""_¢(unsigned long long n) { return n; } // expected-error {{character <U+00A2> not allowed in an identifier}}
 static_assert(0.02_¢ == 2_¢, "");                              // expected-error 2{{character <U+00A2> not allowed in an identifier}}

clang/test/Preprocessor/ucn-pp-identifier.c

 #if a\u00FD()
 #error "This should never happen"
 #endif
 
 #if a\U000000FD()
 #error "This should never happen"
 #endif
 
+#if a\u{FD}() //expected-warning {{Clang extension}}
+#error "This should never happen"
+#endif
+
 #if \uarecool // expected-warning{{incomplete universal character name; treating as '\' followed by identifier}} expected-error {{invalid token at start of a preprocessor expression}}
 #endif
 #if \uwerecool // expected-warning{{\u used with no following hex digits; treating as '\' followed by identifier}} expected-error {{invalid token at start of a preprocessor expression}}
 #endif
 #if \U0001000  // expected-warning{{incomplete universal character name; treating as '\' followed by identifier}} expected-error {{invalid token at start of a preprocessor expression}}
 #endif
 
 // Make sure we reject disallowed UCNs
 #define \ufffe // expected-error {{macro name must be an identifier}}
 #define \U10000000  // expected-error {{macro name must be an identifier}}
 #define \u0061  // expected-error {{character 'a' cannot be specified by a universal character name}} expected-error {{macro name must be an identifier}}
+#define \u{fffe} // expected-error {{macro name must be an identifier}} expected-warning {{Clang extension}}
 
 #define a\u0024
 
 #if \u0110 // expected-warning {{is not defined, evaluates to 0}}
 #endif
 
 
 #define \u0110 1 / 0

 
 
 #define capital_u_\U00FC
 // expected-warning@-1 {{incomplete universal character name}} expected-note@-1 {{did you mean to use '\u'?}} expected-warning@-1 {{whitespace}}
 // CHECK: note: did you mean to use '\u'?
 // CHECK-NEXT:   #define capital_u_\U00FC
 // CHECK-NEXT: {{^                   \^}}
 // CHECK-NEXT: {{^                   u}}
+
+#define \u{}           // expected-warning {{empty delimited universal character name; treating as '\' 'u' '{' '}'}} expected-error {{macro name must be an identifier}}
+#define \u{123456789}  // expected-error {{hex escape sequence out of range}} expected-error {{macro name must be an identifier}}
+#define \u{            // expected-warning {{incomplete delimited universal character name; treating as '\' 'u' '{' identifier}} expected-error {{macro name must be an identifier}}
+#define \u{fgh}        // expected-warning {{incomplete delimited universal character name; treating as '\' 'u' '{' identifier}} expected-error {{macro name must be an identifier}}

clang/test/Sema/ucn-identifiers.c

 // expected-note@-4 + {{declared here}}
 #endif
 
 void goodCalls() {
   \u00FCber(0);
   \u00fcber(1);
   über(2);
   \U000000FCber(3);
+  \u{FC}ber(4); // expected-warning {{Clang extension}}
 }
 
 void badCalls() {
   \u00FCber(0.5); // expected-warning{{implicit conversion from 'double' to 'int'}}
   \u00fcber = 0; // expected-error{{non-object type 'void (int)' is not assignable}}
 
   über(1, 2);
-  \U000000FCber(); 
+  \U000000FCber();
 #ifdef __cplusplus
   // expected-error@-3 {{no matching function}}
   // expected-error@-3 {{no matching function}}
 #else
   // expected-error@-6 {{too many arguments to function call, expected 1, have 2}}
   // expected-error@-6 {{too few arguments to function call, expected 1, have 0}}
 #endif
 }