[clang] Allow lexer to handle string_view literals

[AntonBikineev]

Summary

This is to address p0403r0 proposal.

[EricWF]

Please add tests to cxx1y-user-defined-literals.cpp (and probably rename it).

Other than that this LGTM, but I don't feel comfortable approving clang patches.

[malcolm.parsons]

@mclow.lists is working on this in D26667.
The review comments from that apply here too.

[EricWF]

Should this review be closed then?

[AntonBikineev]

Just added a small test case

[malcolm.parsons]

Does Sema::CheckLiteralOperatorDeclaration need to check StringLiteralParser::isValidUDSuffix?

[AntonBikineev]

In D26829#599906, @malcolm.parsons wrote:

Does Sema::CheckLiteralOperatorDeclaration need to check StringLiteralParser::isValidUDSuffix?

Thanks, nice point! Just addressed it.

[rsmith]

include/clang/Basic/DiagnosticLexKinds.td

189–190 ↗

(On Diff #78602)

I don't see a need for a custom diagnostic for this. Using our normal "reserved ud-suffix" diagnostic seems fine, and it doesn't make sense to treat this suffix as a hard error but accept all the rest with a warning.

lib/Lex/LiteralSupport.cpp

1716–1717

Just make this call NumericLiteralParser::isValidUDSuffix and then check for the sv case. All the numeric suffixes are also valid string literal suffixes for the form operator""suffix.

[AntonBikineev]

lib/Lex/LiteralSupport.cpp

1716–1717

This makes sense for the call sites we currently have.

All the numeric suffixes are also valid string literal suffixes for the form operator""suffix.

Don't really understand this part. It seems inconsistent if one calls, say,

StringLiteralParser::isValidUDSuffix(LangOpts(), "if")

and gets

true

[AntonBikineev]

Richard, thanks, I addressed your comments.

[rsmith]

LGTM, thanks!

lib/Lex/Lexer.cpp

1717

I don't think the LangOpts variable is worthwhile; remove and replace the use of it below with getLangOpts().

[AntonBikineev]

@rsmith Richard, any plans to merge this or is there anything left?

[EricWF]

Committed on behalf of @AntonBikineev in r290744.

include/clang/Lex/LiteralSupport.h

     return UDSuffixOffset;
   }
 
+  static bool isValidUDSuffix(const LangOptions &LangOpts, StringRef Suffix);
+
 private:
   void init(ArrayRef<Token> StringToks);
   bool CopyStringFragment(const Token &Tok, const char *TokBegin,

lib/Lex/Lexer.cpp

                                          getLangOpts());
         if (!isIdentifierBody(Next)) {
           // End of suffix. Check whether this is on the whitelist.
-          IsUDSuffix = (Chars == 1 && Buffer[0] == 's') ||
-                       NumericLiteralParser::isValidUDSuffix(
-                           getLangOpts(), StringRef(Buffer, Chars));
+          const StringRef CompleteSuffix(Buffer, Chars);
+          IsUDSuffix = StringLiteralParser::isValidUDSuffix(getLangOpts(),
+                                                            CompleteSuffix);
           break;
         }
 

lib/Lex/LiteralSupport.cpp

 
   return SpellingPtr-SpellingStart;
 }
+
+/// Determine whether a suffix is a valid ud-suffix. We avoid treating reserved
+/// suffixes as ud-suffixes, because the diagnostic experience is better if we
+/// treat it as an invalid suffix.
+bool StringLiteralParser::isValidUDSuffix(const LangOptions &LangOpts,
+                                          StringRef Suffix) {
+  return NumericLiteralParser::isValidUDSuffix(LangOpts, Suffix) ||
+         Suffix == "sv";
+}

lib/Sema/SemaDeclCXX.cpp

     //   Literal suffix identifiers that do not start with an underscore
     //   are reserved for future standardization.
     Diag(FnDecl->getLocation(), diag::warn_user_literal_reserved)
-      << NumericLiteralParser::isValidUDSuffix(getLangOpts(), LiteralName);
+      << StringLiteralParser::isValidUDSuffix(getLangOpts(), LiteralName);
   }
 
   return false;

test/SemaCXX/cxx1z-user-defined-literals.cpp

+// RUN: %clang_cc1 -std=c++1z %s -include %s -verify
+
+#ifndef INCLUDED
+#define INCLUDED
+
+#pragma clang system_header
+namespace std {
+  using size_t = decltype(sizeof(0));
+
+  struct string_view {};
+  string_view operator""sv(const char*, size_t);
+}
+
+#else
+
+using namespace std;
+string_view s = "foo"sv;
+const char* p = "bar"sv; // expected-error {{no viable conversion}}
+char error = 'x'sv; // expected-error {{invalid suffix}} expected-error {{expected ';'}}
+
+#endif