[clang-cl] Enable concatenation of predefined identifiers

[RIscRIpt]

GitHub issue: https://github.com/llvm/llvm-project/issues/63563

[RIscRIpt]

Rebased onto main

[RIscRIpt]

Added @aaron.ballman as reviewer, because he was reviewer of related patch: 878e590503dff

[aaron.ballman]

Adding some other reviewers for more opinions while I ponder the changes. Sema seems like the wrong place to perform such concatenations, but given that adjacent string literals are concatenated during phase 6, I'm not certain it's actually observable.

[cor3ntin]

clang/include/clang/Parse/Parser.h

578–582 ↗

(On Diff #538025)

Unless you find a better name, I think it's preferable to keep isTokenConcatenable and isTokenPredefinedMsMacro as separate functions.
Also, this seems like a weird place to check for getLangOpts().MicrosoftExt

clang/lib/Parse/ParseExpr.cpp

3288–3296

I think I'd prefer ConsumeAnyToken with an assert that checks it's a stringLiteral or a predefined ms exppression

clang/lib/Sema/SemaExpr.cpp

2033–2069

Can we put that logic in a separate function?

2042

can you assert it's a string literal otherwise?

2050–2065

I think it might be easier to create a string_literal token directly here. I'm also not sure we need to use Lexer::Stringify

clang/test/Sema/ms_predefined_expr.cpp

12

do we have any tests that look at the values of these things?

[cor3ntin]

clang/lib/Parse/ParseExpr.cpp

1300–1301

Can we instead, look at NextToken() and if next token is a StringLiteral or a MS predefined extension we fallthrough?
That would avoid having duplicated logic in ParsePredefinedExpression and ActOnStringLiteral which would be a nice simplification

[RIscRIpt]

Addressed review comments.

In D153914#4497696, @aaron.ballman wrote:

Sema seems like the wrong place to perform such concatenations, but given that adjacent string literals are concatenated during phase 6...

I agree, but I couldn't come up with anything better (having limited knowledge about Clang).
Other possible ugly solution: make actual concatenation in StringLiteralParser, but this would require either of:

1. Pass Decl* into StringLiteralParser - this requires dependency (and linkage) of Lex with AST.
2. Pre-calculate function names for all possible types of tokens (__FUNCTION__, __FUNCDNAME__, etc.) in ActOnStringLiteral and pass into StringLiteralParser
3. Find other way of obtaining current function name in Lex.

One may wish to expand these macros during phase (4), but these macros are predefined identifiers in non-MSVC compilers, and MSVC don't expand them as macros (-E compiler flag); and we don't have knowledge about AST (not to mention the function name) during phase (4).

In D153914#4497696, @aaron.ballman wrote:

I'm not certain it's actually observable.

I didn't understand this part.

clang/include/clang/Parse/Parser.h

578–582 ↗

(On Diff #538025)

Regarding getLangOpts().MicrosoftExt. If you are talking about it's presence in a function which name is meant to be used as a predicate, I agree. If you are talking about class Parser, then there're other places with references to getLangOpts().

Without such function ParseStringLiteralExpression implementation would be too verbose.
Let's decide what we can do after I address other comments.

Meanwhile, I renamed it to isTokenLikeStringLiteral().

clang/lib/Parse/ParseExpr.cpp

1300–1301

I thought of that, but I was afraid that playing with fallthrough wasn't appreciated.
Thanks, fixed.

3288–3296

Done. But do we really need an assertion here? We have one in the function preamble and strict condition in while.

clang/lib/Sema/SemaExpr.cpp

2033–2069

Done. Tho, I couldn't make it const (for the same reason I couldn't make getCurScopeDecl() const). And I wasn't sure about the interface:

std::vector<Token> ExpandMSPredefinedMacros(ArrayRef<Token> Toks);

vs

void ExpandMSPredefinedMacros(MutableArrayRef<Token> Toks);

2042

Done.

2050–2065

I think it might be easier to create a string_literal token directly here.

What do you mean? Is there a function which creates Token object from StringRef? Or is there a way to associate string literal value with a Token without PP? I would like to simplify it, but I haven't found other ways of achieving the same result.

I'm also not sure we need to use Lexer::Stringify

Well, as far as I can see StringLiteralParser expands escape sequences. So, I am just being too careful here.
If not using Lexer::Stringify do we want to assert that function name does not contain neither " not \ (which should not happen™)?

clang/test/Sema/ms_predefined_expr.cpp

12

clang/test/Analysis/eval-predefined-exprs.cpp
clang/test/AST/Interp/literals.cpp
clang/test/SemaCXX/source_location.cpp
clang/test/SemaCXX/predefined-expr.cpp

[RIscRIpt]

Rebased onto main

[tahonermann]

clang/include/clang/Basic/DiagnosticSemaKinds.td

118–120

I think it makes since to be more explicit that the concatenation is with regard to string literals.

clang/include/clang/Basic/TokenKinds.h

89–95

clang/include/clang/Parse/Parser.h

578–582 ↗

(On Diff #538025)

I suggest passing getLangOpts() to isFunctionLocalPredefinedMacro (isUnexpandableMsMacro) and letting it determine whether the token is or is not one of these special not-really-a-string-literal macro things. This will facilitate additional such macros controlled by different options while also colocating the option check with the related tokens.

clang/include/clang/Sema/Sema.h

5705

clang/lib/Parse/ParseExpr.cpp

1301–1312

Since the conditional code is only relevant for some of the tokens here, I would prefer to see the other token kinds (kw___func__, kw___PRETTY_FUNCTION__) handled separately to avoid the conditional fallthrough. That means duplicating the calls to Actions.ActOnPredefinedExpr() and ConsumeToken() between the blocks, but that bothers me less than having to understand the complicated condition.

3282–3285

I think __func__ is not considered a string literal in Microsoft mode, so I think this comment needs some adjusting.

3292–3295

What is the reason for requiring non-concatenated predefined function local macros to be handled by ActOnPredefinedExpr()?

clang/lib/Sema/Sema.cpp

1494

getCurScopeDecl isn't a great name since this doesn't handle class or namespace scopes. How about getCurLocalScopeDecl? That name isn't quite right either since this can return a TranslationUnitDecl, but I think it matches the intent fairly well.

None of the callers need the actual TranslationUnitDecl that is returned. I think this function can return nullptr is the current scope isn't function local and callers can issue the relevant diagnostic in that case (thus avoiding the need to check that a translation unit decl was returned).

clang/lib/Sema/SemaExpr.cpp

1975–1978

2005–2008

A diagnostic is issued if the call to getCurScopeDecl() returned a translation unit declaration (at least in Microsoft mode). Does it make sense to pass a pointer to a TranslationUnitDecl here?

3708

This can just be a nullptr check with my suggested change to getCurScopeDecl().

clang/test/Sema/ms_predefined_expr.cpp

170

If we don't already have such tests, please add tests that exercise these macros at global, namespace, and class scope.

[RIscRIpt]

Addressed review comments, rebased onto main.

clang/include/clang/Basic/TokenKinds.h

89–95

Thanks, I like the name. But shouldn't we reflect that we are referring to only Microsoft (unexpandable) macros? How about isFunctionLocalPredefinedMsMacro?

clang/include/clang/Parse/Parser.h

578–582 ↗

(On Diff #538025)

@tahonermann, in theory it sounds good, but if I understood you correctly, the implementation seem weird to me:

inline bool isFunctionLocalPredefinedMsMacro(TokenKind K, const LangOptions& langOpts) {
  if (!langOpts.MicrosoftExt)
    return false;
  return K == tok::kw___FUNCTION__ || K == tok::kw___FUNCSIG__ ||
         K == tok::kw_L__FUNCTION__ || K == tok::kw_L__FUNCSIG__ ||
         K == tok::kw___FUNCDNAME__;
}

And I would have to add #include "LangOptions.h" in TokenKinds.h.

578–582 ↗

(On Diff #538025)

If you are still concerned about usage of LangOptions in this context, and at the same time you are not against of having such function, I can offer the following alternatives:

1. Make it a local lambda function in ParseStringLiteralExpression
2. Make it a TU local function in ParseExpr.cpp

I went with (2) to do simplification of condition in switch statement below. Marking thread resolved, as it's no longer relevant.

clang/include/clang/Sema/Sema.h

5705

Renamed to ExpandFunctionLocalPredefinedMsMacros. If you think my addition of Ms is redundant, let me know.

clang/lib/Parse/ParseExpr.cpp

1301–1312

That's a valid point. And by handling two tokens separately allows simplification of the condition. Adjusted code.

3282–3285

Right, removed mention of __func__.

3292–3295

To ensure generation of the same AST as before my changes (with PredefinedExpr):

|   | `-VarDecl <col:2, col:19> col:13 a 'const char[2]' cinit
|   |   `-PredefinedExpr <col:19> 'const char[2]' __FUNCTION__
|   |     `-StringLiteral <col:19> 'const char[2]' "f"

This actually brings a question: do we want to wrap StringLiterals (into PredefinedExpr) which were formed from concatenation of string-literals and some predefined identifier(s)? But it does not really make any sense: concatenated string does not represent any of PredefinedExpr.

clang/lib/Sema/Sema.cpp

1494

Sounds good to me. The only alternative I can come up with is getCurFunctionScopeDecl, but is it any better?

Adjusted code as per your suggestions.

clang/lib/Sema/SemaExpr.cpp

1975–1978

Changed, thank you!

2005–2008

Shortly, yes, kind of. ComputeName can handle TranslationUnitDecl in which case it returns an empty string. This way we implicitly (without additional code) create empty string-literal Tokens which can be handled in StringLiteralParser. And we cannot pass non-string-literal tokens into StringLiteralParser.

3708

Done, but I had to undo removal of

currentDecl = Context.getTranslationUnitDecl();

but it is not a big deal.

[cor3ntin]

clang/lib/Parse/ParseExpr.cpp

3288–3296

Looking at that again, i think you can remove the assert (the one at the top should stay). which you did, excellent. sorry about that!

3292–3295

Yes, this make sense. I think it's better than adding more complexity / memory footprint to StringLiteral.

clang/lib/Sema/SemaExpr.cpp

2033–2069

I think the later let you use a small vector so it's probably more efficient. I'm find with either

2050–2065

Thanks! I really would get rid of Stringify here - I struggle to see how a function could be produced that has characters that cannot appear in an identifier. even asserting isn't very useful.

clang/test/Sema/ms_predefined_expr.cpp

12

I think it's worth add a few more tests in clang/test/SemaCXX/predefined-expr.cpp checking concatenations

[RIscRIpt]

Addressed review comments, rebased onto main

clang/lib/Sema/SemaExpr.cpp

2050–2065

Replaced Stringify with enclosure into raw-string-literal. Despite raw-string-literals are available starting from C++11, as far as I can see StringLiteralParser does not check for the standard.
Let me know, if you believe we cannot abuse this fact, and we should keep using "regular string-literal" for this purpose.

clang/test/Sema/ms_predefined_expr.cpp

12

Added tests to check values of these string literals, as well as moved ms_wide_predefined_expr.cpp tests into ms_predefined_expr.cpp.

[tahonermann]

clang/include/clang/Basic/TokenKinds.h

89–95

I don't think the Microsoft association is meaningful. Sure, some of the predefined macros are only treated differently when compiling in Microsoft mode, but some of those macros aren't Microsoft specific. Also, there might be macros provided by other implementations that we'll want to emulate some day.

clang/include/clang/Sema/Sema.h

5705

I don't think the Ms is redundant, but I do think it is unnecessary and slightly confusing (__FUNCTION__) is not a Microsoft macro.

clang/lib/Parse/ParseExpr.cpp

1307–1308

TokenIsLikeStringLiteral() checks MicrosoftExt, so the check here is redundant. This is an example of why I would like to see the MicrosoftExt checking pushed down to isFunctionLocalPredefinedMsMacro(); that really seems where that checking belongs to me.

3293–3294

Is there a missing check for MicrosoftExt here? This is another example of why I would prefer to see those checks pushed down to isFunctionLocalPredefinedMsMacro().

clang/lib/Sema/SemaExpr.cpp

1983–1984

This could use a comment since it isn't obvious why the translation unit declaration is used when not in a local scope declaration of some kind.

2005–2008

Ah, ok. And I see it even differentiates what name is produced for __PRETTY_FUNCTION__. That leaves me wondering what the right behavior should be at class and namespace scope, but maybe I'm better off not asking questions I don't want to know the answer to.

[cor3ntin]

clang/include/clang/Basic/TokenKinds.h

89–95

I think it is, there is currently no way to link isFunctionLocalPredefinedMacro to the MS feature. "MSPredefinedMacro" is pretty self explanatory, same reason we most often - but not always - use GNU in the name of function related to GNU extensions.
There are enough similar-sounding features and extensions that we should try to make it clear which feature we are talking about.

[tahonermann]

clang/include/clang/Basic/TokenKinds.h

89–95

Perhaps we still haven't found the right name then. With the name that I've been advocating, this function should also return true for tok::kw___PRETTY_FUNCTION__ even though that macro should not expand to something that can be concatenated with other string literals (whether compiling in Microsoft mode or not).

The Microsoft extension is the localized expansion to a string literal that can be concatenated with other string literals. We probably should isolate the conditions for that behavior to one place and if we do that, then I guess naming this function as specific to Microsoft mode is ok; we can always rename it later if it gets used for non-Microsoft extensions.

Here is my suggestion then:

// Return true if the token corresponds to a function local predefined
// macro that, in Microsoft modes, expands to a string literal (that can
// then be concatenated with other string literals).
inline bool isMsFunctionLocalStringLiteralMacro(TokenKind K, const LangOptions &langOpts) {
  return langOpts.MicrosoftExt && (
      K == tok::kw___FUNCTION__ || K == tok::kw_L__FUNCTION__ ||
      K == tok::kw___FUNCSIG__ || K == tok::kw_L__FUNCSIG__ ||
      K == tok::kw___FUNCDNAME__);
}

This will avoid the need for callers to check for MicrosoftExt; that is what I really want to avoid since it is easy to forget to do that check.

[RIscRIpt]

clang/include/clang/Basic/TokenKinds.h

89–95

1. By requiring user to pass LangOptions I think we can remove MS association (implying that there could be other non-msft cases in the future)
2. We would have to include a LangOptions.h in TokenKinds.h, are we ok with this? Alternatively while this function is for msft cases only, we could pass bool MicrosoftExt

Personally, I like version with LangOptions and removal of MS.

[RIscRIpt]

Addressed review comments, rebased onto main

clang/lib/Parse/ParseExpr.cpp

1301–1312

Having TokenIsLikeStringLiteral I think we can return to single case. I believe 3 values in condition is not difficult.

1307–1308

Let's consider this code:

if (!TokenIsLikeStringLiteral(NextToken(), getLangOpts())) {
  Res = Actions.ActOnPredefinedExpr(Tok.getLocation(), SavedKind);
  ConsumeToken();
  break;
}

The condition can be expanded as follows: !(isStringLiteral || (MsExt && NeededToken)). Let's consider MsExt is false, then we have: !isStringLiteral. So, if next token is StringLiteral we will try to concatenate it even without MsExt, which is invalid. Thus this seemingly redundant check is needed.

3293–3294

My intention for this assert is to ensure that we keep treating single predefined identifiers as PredefinedExpr (when we don't need to concat them) to preserve AST. I've adjusted the check to make it more clear.

[tahonermann]

I think this is looking good. The only big thing I noticed is some code that looks like it should have been removed with the last set of changes. Otherwise, just some minor concerns and suggestions.

clang/lib/Parse/ParseExpr.cpp

1294–1296

I believe this code should be removed now; it is no longer reachable.

1301–1312

I think this is good, but how about a comment to make the intent more clear?

clang/lib/Sema/SemaExpr.cpp

1903

2014–2017

"EFLPM"? I'm assuming this is an attempt to avoid an accidental clash with the computed name. Does this suffice to ensure a clash can't happen? If not, can we do better? Per http://eel.is/c++draft/lex.string#nt:r-char and http://eel.is/c++draft/lex.charset#1, we have a lot of flexibility regarding which characters to use.

clang/test/Sema/ms_predefined_expr.cpp

115

How about testing some other encoding prefix combinations?

u"" __FUNCTION // Ok; UTF-16 string literal
u"" L__FUNCTION__ // ill-formed.

[RIscRIpt]

Address review comments, rebase onto main

[RIscRIpt]

Thanks to your suggestion of testing different types, I realized clang does not support MSVC macros with u, u8, and U prefixes in addition to L__FUNCDNAME. By the way, clang replicates MSVC behavior a little bit incorrectly: L__FUNCTION__ is not a valid token for MSVC, however it becomes valid if used via macro concatenation (see #define WIDE).
I think addition of support of new macros as well as bug fix (if possible) should be submitted separately. I'll open-up an issue, and submit a PR later.

clang/lib/Sema/SemaExpr.cpp

2014–2017

At first I thought you were hinting me to use non-ascii characters for d-char-sequence. However, when I looked closely d-char-sequence is not as flexible as r-chars that you referenced: http://eel.is/c++draft/lex.string#nt:d-char and http://eel.is/c++draft/lex.charset#2

Here're my thoughts:

1. I was afraid that there could be a possibility of a function name contain either quote (") or a backslash (\) despite it seemed impossible.
2. At first I used Lexer::Stringify to make it bullet-proof. I didn't like it, neither did you (reviewers).
3. I replaced Lexer::Stringify with raw-string-literal (d-char-sequence was chosen as acronym of current function name).
4. Current EFLPM looks weird and cryptic. And we are limited to [lex.charset.basic] with exceptions (space and earlier chars are not allowed). I think, using a raw-string-literal without d-char-sequence would be enough, because problem could be caused only by two consecutive characters )", neither of which can appear in a function name.

clang/test/Sema/ms_predefined_expr.cpp

115

Good idea. I am not sure whether I should specify -std in the test command. I'll leave it without, because current default standard is C++17, and I believe it's not going to be decreased.

And I think there are enough tests of checking values of these macros. So, in tests for encoding I am going to check types.

[cor3ntin]

clang/include/clang/Basic/TokenKinds.h

89–95

By requiring user to pass LangOptions I think we can remove MS association

I don't think there is any motivation to future proof against hypotheticals, we can always refactor later

We would have to include a LangOptions.h in TokenKinds.h

This makes me uneasy, but i think we can move the function to LiteralSupport.h and include that in ParseExpr.cpp.

[tahonermann]

clang/lib/Sema/SemaExpr.cpp

2014–2017

Sorry about that; you are absolutely right that d-char-sequence is (much) more constrained than r-char-sequence.

For __FUNCSIG__, the generated text can include arbitrary text. For example, given this declaration:

void f(decltype(sizeof("()"))*)

the macro value is:

void __cdecl f(decltype(sizeof ("()")) *)

which does contain )".

I think we should do more to avoid any possibility of the resulting string being unparseable because the resulting diagnostics will be completely inscrutable. The best way to do that would be to avoid trying to produce a parseable string literal in the first place. Based on that and given that Sema::ActOnStringLiteral() immediately uses StringLiteralParser to produce an expanded string literal, I think we would be better off moving this expansion there such that these macros can be expanded to already cooked string literals that don't need to be parsed at all. This will mean having to modify the StringLiteralParser constructor to accept a Decl* pointer that will be passed getCurLocalScopeDecl() in Sema::ActOnStringLiteral(). StringLiteralParser can then assert if it ever encounters one of these function local predefined tokens when it hasn't been provided a corresponding Decl* to use with them.

clang/test/Sema/ms_predefined_expr.cpp

115

Thank you for adding those. I think we should check values for a couple of cases as well. Consider a function name that contains 𐀀 (U+10000 LINEAR B SYLLABLE B008 A). In UTF-8, that character requires four code units (F0 90 80 80) while in UTF-16 it requires two (D800 DC00). We'll want to ensure that such characters are properly converted between encodings.

void test_encoding𐀀() {
  ASSERT_EQ(u"" __FUNCTION__, u"test_encoding𐀀");
}

[tahonermann]

clang/test/Sema/ms_predefined_expr.cpp

170

Here is another case to test. MSVC accepts this.

void test_static_assert() {
  static_assert(true, __FUNCTION__);
}

[cor3ntin]

clang/lib/Sema/SemaExpr.cpp

2014–2017

I didn't think about that scenario but in that case the original idea to use Lexer::Stringify seems like a cleaner solution to me.
Adding that kind of logic to LiteralSupport creates a lot of coupling for not much gain that I can see.

In that light, Lexer::Stringify seems like the least worse option.

[tahonermann]

clang/lib/Sema/SemaExpr.cpp

2014–2017

Yeah, that might be the simpler solution. That matches how __FILE__, __BASE_FILE__, and __FILE_NAME__ are handled in Preprocessor::ExpandBuiltinMacro().

[RIscRIpt]

Addressed review comments, rebased onto main

Noticable changes:

1. Diagnostics message
2. Support of expansion of these attributes in 2nd argument of static_assert
3. Moved helper-functions to Lex/LiteralSupport
4. Changed back to usage of Lexer::Stringify for function name expansion

Local lit clang/test succeed for both Debug and Release builds.

clang/include/clang/Basic/TokenKinds.h

89–95

Sounds good to me. Due to other changes I also moved TokenIsLikeStringLiteral there.

I don't think there is any motivation to future proof against hypotheticals, we can always refactor later

Yes, not a strong argument, but I'd like to keep current name if you are not against.

clang/lib/Sema/SemaExpr.cpp

2014–2017

Thanks for pointing out good counter-example.

In that light, Lexer::Stringify seems like the least worse option.

Unfortunately, yes.

In regards other possible implementations, I commented previously here: https://reviews.llvm.org/D153914#4500807

to modify the StringLiteralParser constructor to accept a Decl*

This requires dependency (and linkage) of Lex with AST.

I am going to add your counter-examples to test, and change implementation back to Lexer::Stringify

clang/test/Sema/ms_predefined_expr.cpp

170

Thanks for finding this! I had to do some adjustments to support it. In addition I changed diagnostics message because it no longer made sense, and removed assert about "single predefined identifiers shall be handled by ActOnPredefinedExpr". Other code around references of err_expected_string_literal do not seem to require support of this MS ext.

[RIscRIpt]

Removed irrelevant changes

clang/lib/Sema/SemaExpr.cpp

1981

I think I should remove this assertion so this function would be usable without MS ext, on the other hand it would be a noop without MS ext.

[tahonermann]

I think I'm happy with this. I noted two code changes that I think are no longer needed and offered some final suggestions on some names.

clang/include/clang/Basic/DiagnosticSemaKinds.td

118–120

For consistency with other names, I'll suggest some alternative names here. I think these better reflect the actual diagnostic message.

• ext_expand_predef_ms -> ext_string_literal_from_predefined
• MicrosoftConcatPredefined -> MicrosoftStringLiteralFromPredefined

clang/include/clang/Basic/TokenKinds.h

22–23

It looks like this change is no longer needed.

clang/lib/Basic/TokenKinds.cpp

14–15

This change appears to no longer be needed.

[tahonermann]

clang/lib/Sema/SemaExpr.cpp

1981

I suggest leaving it in solely because use of the function does impose some overhead in the construction of the std::vector that is returned. If that overhead can be avoided (I guess by passing the container to populate by reference as a separate argument and then return either the original ArrayRef or a new one referencing the populated container), then this could be made a no-op and the checks for MicrosoftExt at the call sites could be removed.

[RIscRIpt]

Rename diagnostic messages constants

[tahonermann]

I'm happy with this. I noted one last rename suggestion to maintain consistency, but am accepting. Please give Corentin a final chance to raise any concerns. Thank you for being so responsive through all the review iterations!

clang/include/clang/Basic/DiagnosticGroups.td

1210

We should probably name the diag group to match.

• microsoft-concat-predefined -> microsoft-string-literal-from-predefined

[RIscRIpt]

Rename MicrosoftStringLiteralFromPredefined

[RIscRIpt]

Thank you for the review! I apologize for missing the small details; I should have noticed and addressed them myself.

[RIscRIpt]

Rebased onto main, run local lit clang/tests. (bump for @cor3ntin)
As far as I understand this shall be merged into main by maintainers.
Please let me know if something else is expected from me.

[cor3ntin]

In D153914#4570109, @RIscRIpt wrote:

Rebased onto main, run local lit clang/tests. (bump for @cor3ntin)
As far as I understand this shall be merged into main by maintainers.
Please let me know if something else is expected from me.

We will commit on your behalf, what name/email do you want us to use?
Thanks!

[aaron.ballman]

clang/lib/Sema/Sema.cpp

1494–1505

Made the function const because it's not writing to any fields, removed else because of preceding return.

clang/lib/Sema/SemaExpr.cpp

1986

Matching our usual naming conventions.

1998–2000

This will miss the diagnostic if the current declaration is a namespace instead of at file scope, right?

2033–2035

Probably worth a comment here explaining that ExpandedToks acts as the backing store for StringToks, which is why both are being assigned to here.

[RIscRIpt]

Address review comments, rebase onto main

[RIscRIpt]

In D153914#4570148, @cor3ntin wrote:

We will commit on your behalf, what name/email do you want us to use?
Thanks!

Thanks for asking; keep the name/email as-is in the commit you see: Author: Richard Dzenis <[e-mail]>

clang/lib/Sema/Sema.cpp

1494–1505

I would love to, and I tried. But unfortunately it's not possible without introducing more changes: all the member functions (except getCurFunctionOrMethodDecl()) are non-const.
Removed else.

clang/lib/Sema/SemaExpr.cpp

1998–2000

Right. But getCurLocalScopeDecl() returns function scope at most. See Note in a code comment above.

2005–2008

A diagnostic is issued if the call to getCurScopeDecl() returned a translation unit declaration (at least in Microsoft mode). Does it make sense to pass a pointer to a TranslationUnitDecl here?

[aaron.ballman]

clang/lib/Sema/Sema.cpp

1494–1505

Ah, thanks!

clang/lib/Sema/SemaExpr.cpp

1998–2000

Ah, exactly right, thank you!

2033–2034

[RIscRIpt]

Updated comment; rebased onto main

clang/lib/Sema/SemaExpr.cpp

2033–2034

Thanks, that's better. Changed in both places.

clang/include/clang/Basic/DiagnosticGroups.td

 def MicrosoftConstInit : DiagGroup<"microsoft-const-init">;
 def MicrosoftVoidPseudoDtor : DiagGroup<"microsoft-void-pseudo-dtor">;
 def MicrosoftAnonTag : DiagGroup<"microsoft-anon-tag">;
 def MicrosoftCommentPaste : DiagGroup<"microsoft-comment-paste">;
 def MicrosoftEndOfFile : DiagGroup<"microsoft-end-of-file">;
 def MicrosoftInaccessibleBase : DiagGroup<"microsoft-inaccessible-base">;
 def MicrosoftStaticAssert : DiagGroup<"microsoft-static-assert">;
 def MicrosoftInitFromPredefined : DiagGroup<"microsoft-init-from-predefined">;
+def MicrosoftConcatPredefined : DiagGroup<"microsoft-concat-predefined">;
 
 // Aliases.
 def : DiagGroup<"msvc-include", [MicrosoftInclude]>;
                 // -Wmsvc-include = -Wmicrosoft-include
 
 // Warnings group for warnings about Microsoft extensions.
 def Microsoft : DiagGroup<"microsoft",
     [MicrosoftCharize, MicrosoftDrectveSection, MicrosoftInclude,
      MicrosoftCppMacro, MicrosoftFixedEnum, MicrosoftSealed, MicrosoftAbstract,
      MicrosoftUnqualifiedFriend, MicrosoftExceptionSpec, MicrosoftUsingDecl,
      MicrosoftMutableReference, MicrosoftPureDefinition,
      MicrosoftUnionMemberReference, MicrosoftExplicitConstructorCall,
      MicrosoftEnumValue, MicrosoftDefaultArgRedefinition, MicrosoftTemplate,
      MicrosoftRedeclareStatic, MicrosoftEnumForwardReference, MicrosoftGoto,
      MicrosoftFlexibleArray, MicrosoftExtraQualification, MicrosoftCast,
      MicrosoftConstInit, MicrosoftVoidPseudoDtor, MicrosoftAnonTag,
      MicrosoftCommentPaste, MicrosoftEndOfFile, MicrosoftStaticAssert,
-     MicrosoftInitFromPredefined, MicrosoftInconsistentDllImport]>;
+     MicrosoftInitFromPredefined, MicrosoftConcatPredefined,
+     MicrosoftInconsistentDllImport]>;
 
 def ClangClPch : DiagGroup<"clang-cl-pch">;
 
 def ObjCNonUnifiedException : DiagGroup<"objc-nonunified-exceptions">;
 
 def ObjCProtocolMethodImpl : DiagGroup<"objc-protocol-method-implementation">;
 
 def ObjCNoPropertyAutoSynthesis : DiagGroup<"objc-property-synthesis">;

clang/include/clang/Basic/DiagnosticSemaKinds.td

 
 // Semantic analysis of constant literals.
 def ext_predef_outside_function : Warning<
   "predefined identifier is only valid inside function">,
   InGroup<DiagGroup<"predefined-identifier-outside-function">>;
 def ext_init_from_predefined : ExtWarn<
   "initializing an array from a '%0' predefined identifier is a Microsoft extension">,
   InGroup<MicrosoftInitFromPredefined>;
+def ext_expand_predef_ms : ExtWarn<
+  "expansion of predefined identifier '%0' to a string literal is a Microsoft extension">,
+  InGroup<MicrosoftConcatPredefined>;
 def warn_float_overflow : Warning<
   "magnitude of floating-point constant too large for type %0; maximum is %1">,
    InGroup<LiteralRange>;
 def warn_float_underflow : Warning<
   "magnitude of floating-point constant too small for type %0; minimum is %1">,
   InGroup<LiteralRange>;
 def warn_float_compare_literal : Warning<
   "floating-point comparison is always %select{true|false}0; "

clang/include/clang/Basic/TokenKinds.h

 #ifndef LLVM_CLANG_BASIC_TOKENKINDS_H
 #define LLVM_CLANG_BASIC_TOKENKINDS_H
 
 #include "llvm/ADT/DenseMapInfo.h"
 #include "llvm/Support/Compiler.h"
 
 namespace clang {
 
+class LangOptions;
+
 namespace tok {
 
 /// Provides a simple uniform namespace for tokens from all C languages.
 enum TokenKind : unsigned short {
 #define TOK(X) X,
 #include "clang/Basic/TokenKinds.def"
   NUM_TOKENS
 };

 /// Returns the spelling of preprocessor keywords, such as "else".
 const char *getPPKeywordSpelling(PPKeywordKind Kind) LLVM_READNONE;
 
 /// Return true if this is a raw identifier or an identifier kind.
 inline bool isAnyIdentifier(TokenKind K) {
   return (K == tok::identifier) || (K == tok::raw_identifier);
 }
 
 /// Return true if this is a C or C++ string-literal (or
 /// C++11 user-defined-string-literal) token.
 inline bool isStringLiteral(TokenKind K) {
   return K == tok::string_literal || K == tok::wide_string_literal ||
          K == tok::utf8_string_literal || K == tok::utf16_string_literal ||
          K == tok::utf32_string_literal;
 }
 
 /// Return true if this is a "literal" kind, like a numeric
 /// constant, string, etc.
 inline bool isLiteral(TokenKind K) {
   return K == tok::numeric_constant || K == tok::char_constant ||
          K == tok::wide_char_constant || K == tok::utf8_char_constant ||
          K == tok::utf16_char_constant || K == tok::utf32_char_constant ||
          isStringLiteral(K) || K == tok::header_name;

clang/include/clang/Lex/LiteralSupport.h

 class SourceLocation;
 class TargetInfo;
 class SourceManager;
 class LangOptions;
 
 /// Copy characters from Input to Buf, expanding any UCNs.
 void expandUCNs(SmallVectorImpl<char> &Buf, StringRef Input);
 
+/// Return true if the token corresponds to a function local predefined macro,
+/// which expands to a string literal, that can be concatenated with other
+/// string literals (only in Microsoft mode).
+bool isFunctionLocalStringLiteralMacro(tok::TokenKind K, const LangOptions &LO);
+
+/// Return true if the token is a string literal, or a function local
+/// predefined macro, which expands to a string literal.
+bool tokenIsLikeStringLiteral(const Token &Tok, const LangOptions &LO);
+
 /// NumericLiteralParser - This performs strict semantic analysis of the content
 /// of a ppnumber, classifying it as either integer, floating, or erroneous,
 /// determines the radix of the value and can convert it to a useful value.
 class NumericLiteralParser {
   const SourceManager &SM;
   const LangOptions &LangOpts;
   DiagnosticsEngine &Diags;
 

clang/include/clang/Sema/Sema.h

public:
   /// in a 'block', this returns the containing context.
   ObjCMethodDecl *getCurMethodDecl();
 
   /// getCurFunctionOrMethodDecl - Return the Decl for the current ObjC method
   /// or C function we're in, otherwise return null.  If we're currently
   /// in a 'block', this returns the containing context.
   NamedDecl *getCurFunctionOrMethodDecl() const;
 
+  /// getCurLocalScopeDecl - Return the Decl for either of:
+  /// block, lambda, captured statement, function, or nullptr.
+  Decl *getCurLocalScopeDecl();
+
   /// Add this decl to the scope shadowed decl chains.
   void PushOnScopeChains(NamedDecl *D, Scope *S, bool AddToContext = true);
 
   /// isDeclInScope - If 'Ctx' is a function/method, isDeclInScope returns true
   /// if 'D' is in Scope 'S', otherwise 'S' is ignored and isDeclInScope returns
   /// true if 'D' belongs to the given declaration context.
   ///
   /// \param AllowInlineNamespace If \c true, allow the declaration to be in the
ExprResult BuildDeclarationNameExpr(
       bool AcceptInvalidDecl = false);
 
   ExprResult BuildLiteralOperatorCall(LookupResult &R,
                       DeclarationNameInfo &SuffixInfo,
                       ArrayRef<Expr *> Args,
                       SourceLocation LitEndLoc,
                       TemplateArgumentListInfo *ExplicitTemplateArgs = nullptr);
 
+  // ExpandFunctionLocalPredefinedMacros - Returns a new vector of Tokens,
+  // where Tokens representing function local predefined macros (such as
+  // __FUNCTION__) are replaced (expanded) with string-literal Tokens.
+  std::vector<Token> ExpandFunctionLocalPredefinedMacros(ArrayRef<Token> Toks);
+
   ExprResult BuildPredefinedExpr(SourceLocation Loc,
                                  PredefinedExpr::IdentKind IK);
   ExprResult ActOnPredefinedExpr(SourceLocation Loc, tok::TokenKind Kind);
   ExprResult ActOnIntegerConstant(SourceLocation Loc, uint64_t Val);
 
   ExprResult BuildSYCLUniqueStableNameExpr(SourceLocation OpLoc,
                                            SourceLocation LParen,
                                            SourceLocation RParen,

clang/lib/Basic/TokenKinds.cpp

 //===--- TokenKinds.cpp - Token Kinds Support -----------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 //  This file implements the TokenKind enum and support functions.
 //
 //===----------------------------------------------------------------------===//
 
 #include "clang/Basic/TokenKinds.h"
+
+#include "clang/Basic/LangOptions.h"
 #include "llvm/Support/ErrorHandling.h"
 using namespace clang;
 
 static const char * const TokNames[] = {
 #define TOK(X) #X,
 #define KEYWORD(X,Y) #X,
 #include "clang/Basic/TokenKinds.def"
   nullptr

clang/lib/Lex/LiteralSupport.cpp

for (; NumHexDigits != 0; ++I, --NumHexDigits) {
       CodePoint += Value;
     }
 
     appendCodePoint(CodePoint, Buf);
     --I;
   }
 }
 
+bool clang::isFunctionLocalStringLiteralMacro(tok::TokenKind K,
+                                              const LangOptions &LO) {
+  return LO.MicrosoftExt &&
+         (K == tok::kw___FUNCTION__ || K == tok::kw_L__FUNCTION__ ||
+          K == tok::kw___FUNCSIG__ || K == tok::kw_L__FUNCSIG__ ||
+          K == tok::kw___FUNCDNAME__);
+}
+
+bool clang::tokenIsLikeStringLiteral(const Token &Tok, const LangOptions &LO) {
+  return tok::isStringLiteral(Tok.getKind()) ||
+         isFunctionLocalStringLiteralMacro(Tok.getKind(), LO);
+}
+
 static bool ProcessNumericUCNEscape(const char *ThisTokBegin,
                                     const char *&ThisTokBuf,
                                     const char *ThisTokEnd, uint32_t &UcnVal,
                                     unsigned short &UcnLen, bool &Delimited,
                                     FullSourceLoc Loc, DiagnosticsEngine *Diags,
                                     const LangOptions &Features,
                                     bool in_char_string_literal = false) {
   const char *UcnBegin = ThisTokBuf;

clang/lib/Parse/ParseDeclCXX.cpp

 #include "clang/AST/DeclTemplate.h"
 #include "clang/AST/PrettyDeclStackTrace.h"
 #include "clang/Basic/AttributeCommonInfo.h"
 #include "clang/Basic/Attributes.h"
 #include "clang/Basic/CharInfo.h"
 #include "clang/Basic/OperatorKinds.h"
 #include "clang/Basic/TargetInfo.h"
 #include "clang/Basic/TokenKinds.h"
+#include "clang/Lex/LiteralSupport.h"
 #include "clang/Parse/ParseDiagnostic.h"
 #include "clang/Parse/Parser.h"
 #include "clang/Parse/RAIIObjectsForParser.h"
 #include "clang/Sema/DeclSpec.h"
 #include "clang/Sema/EnterExpressionEvaluationContext.h"
 #include "clang/Sema/ParsedTemplate.h"
 #include "clang/Sema/Scope.h"
 #include "llvm/ADT/SmallString.h"
if (Tok.is(tok::r_paren)) {
     Diag(Tok, DiagVal) << getStaticAssertNoMessageFixIt(AssertExpr.get(),
                                                         Tok.getLocation());
   } else {
     if (ExpectAndConsume(tok::comma)) {
       SkipUntil(tok::semi);
       return nullptr;
     }
 
-    if (isTokenStringLiteral())
+    if (tokenIsLikeStringLiteral(Tok, getLangOpts()))
       AssertMessage = ParseUnevaluatedStringLiteralExpression();
     else if (getLangOpts().CPlusPlus26)
       AssertMessage = ParseConstantExpressionInExprEvalContext();
     else {
       Diag(Tok, diag::err_expected_string_literal)
           << /*Source='static_assert'*/ 1;
       SkipMalformedDecl();
       return nullptr;

clang/lib/Parse/ParseExpr.cpp

 /// handled by ParseCastExpression, the higher level pieces are handled by
 /// ParseBinaryExpression.
 ///
 //===----------------------------------------------------------------------===//
 
 #include "clang/AST/ASTContext.h"
 #include "clang/AST/ExprCXX.h"
 #include "clang/Basic/PrettyStackTrace.h"
+#include "clang/Lex/LiteralSupport.h"
 #include "clang/Parse/Parser.h"
 #include "clang/Parse/RAIIObjectsForParser.h"
 #include "clang/Sema/DeclSpec.h"
 #include "clang/Sema/EnterExpressionEvaluationContext.h"
 #include "clang/Sema/ParsedTemplate.h"
 #include "clang/Sema/Scope.h"
 #include "clang/Sema/TypoCorrection.h"
 #include "llvm/ADT/SmallVector.h"
#undef RTT_JOIN
   case tok::char_constant:     // constant: character-constant
   case tok::wide_char_constant:
   case tok::utf8_char_constant:
   case tok::utf16_char_constant:
   case tok::utf32_char_constant:
     Res = Actions.ActOnCharacterConstant(Tok, /*UDLScope*/getCurScope());
     ConsumeToken();
     break;
   case tok::kw___func__:       // primary-expression: __func__ [C99 6.4.2.2]
   case tok::kw___FUNCTION__:   // primary-expression: __FUNCTION__ [GNU]
   case tok::kw___FUNCDNAME__:   // primary-expression: __FUNCDNAME__ [MS]
   case tok::kw___FUNCSIG__:     // primary-expression: __FUNCSIG__ [MS]
   case tok::kw_L__FUNCTION__:   // primary-expression: L__FUNCTION__ [MS]
   case tok::kw_L__FUNCSIG__:    // primary-expression: L__FUNCSIG__ [MS]
   case tok::kw___PRETTY_FUNCTION__:  // primary-expression: __P..Y_F..N__ [GNU]
+    // Function local predefined macros are represented by PredefinedExpr except
+    // when Microsoft extensions are enabled and one of these macros is adjacent
+    // to a string literal or another one of these macros.
+    if (!(getLangOpts().MicrosoftExt &&
+          tokenIsLikeStringLiteral(Tok, getLangOpts()) &&
+          tokenIsLikeStringLiteral(NextToken(), getLangOpts()))) {
       Res = Actions.ActOnPredefinedExpr(Tok.getLocation(), SavedKind);
       ConsumeToken();
       break;
+    }
+    [[fallthrough]]; // treat MS function local macros as concatenable strings
   case tok::string_literal:    // primary-expression: string-literal
   case tok::wide_string_literal:
   case tok::utf8_string_literal:
   case tok::utf16_string_literal:
   case tok::utf32_string_literal:
     Res = ParseStringLiteralExpression(true);
     break;
   case tok::kw__Generic:   // primary-expression: generic-selection [C11 6.5.1]
     Res = ParseGenericSelectionExpression();

 
 ExprResult Parser::ParseUnevaluatedStringLiteralExpression() {
   return ParseStringLiteralExpression(/*AllowUserDefinedLiteral=*/false,
                                       /*Unevaluated=*/true);
 }
 
 ExprResult Parser::ParseStringLiteralExpression(bool AllowUserDefinedLiteral,
                                                 bool Unevaluated) {
-  assert(isTokenStringLiteral() && "Not a string literal!");
+  assert(tokenIsLikeStringLiteral(Tok, getLangOpts()) &&
+         "Not a string-literal-like token!");
 
-  // String concat.  Note that keywords like __func__ and __FUNCTION__ are not
-  // considered to be strings for concatenation purposes.
+  // String concatenation.
+  // Note: some keywords like __FUNCTION__ are not considered to be strings
+  // for concatenation purposes, unless Microsoft extensions are enabled.
   SmallVector<Token, 4> StringToks;
 
   do {
     StringToks.push_back(Tok);
-    ConsumeStringToken();
-  } while (isTokenStringLiteral());
+    ConsumeAnyToken();
+  } while (tokenIsLikeStringLiteral(Tok, getLangOpts()));
 
   if (Unevaluated) {
     assert(!AllowUserDefinedLiteral && "UDL are always evaluated");
     return Actions.ActOnUnevaluatedStringLiteral(StringToks);
   }
 
   // Pass the set of string tokens, ready for concatenation, to the actions.
   return Actions.ActOnStringLiteral(StringToks,
                                     AllowUserDefinedLiteral ? getCurScope()
                                                             : nullptr);
 }
 
 /// ParseGenericSelectionExpression - Parse a C11 generic-selection
 /// [C11 6.5.1.1].

clang/lib/Sema/Sema.cpp

 
 NamedDecl *Sema::getCurFunctionOrMethodDecl() const {
   DeclContext *DC = getFunctionLevelDeclContext();
   if (isa<ObjCMethodDecl>(DC) || isa<FunctionDecl>(DC))
     return cast<NamedDecl>(DC);
   return nullptr;
 }
 
+Decl *Sema::getCurLocalScopeDecl() {
+  if (const BlockScopeInfo *BSI = getCurBlock())
+    return BSI->TheDecl;
+  else if (const LambdaScopeInfo *LSI = getCurLambda())
+    return LSI->CallOperator;
+  else if (const CapturedRegionScopeInfo *CSI = getCurCapturedRegion())
+    return CSI->TheCapturedDecl;
+  else if (NamedDecl *ND = getCurFunctionOrMethodDecl())
+    return ND;
+  else
+    return nullptr;
+}
+
 LangAS Sema::getDefaultCXXMethodAddrSpace() const {
   if (getLangOpts().OpenCL)
     return getASTContext().getDefaultOpenCLPointeeAddrSpace();
   return LangAS::Default;
 }
 
 void Sema::EmitCurrentDiagnostic(unsigned DiagID) {
   // FIXME: It doesn't make sense to me that DiagID is an incoming argument here

clang/lib/Sema/SemaExpr.cpp

return GenericSelectionExpr::Create(
         Context, KeyLoc, ControllingExpr, Types, Exprs, DefaultLoc, RParenLoc,
         ContainsUnexpandedParameterPack, ResultIndex);
   }
   return GenericSelectionExpr::Create(
       Context, KeyLoc, ControllingType, Types, Exprs, DefaultLoc, RParenLoc,
       ContainsUnexpandedParameterPack, ResultIndex);
 }
 
+static PredefinedExpr::IdentKind getPredefinedExprKind(tok::TokenKind Kind) {
+  switch (Kind) {
+  default:
+    llvm_unreachable("unexpected TokenKind");
+  case tok::kw___func__:
+    return PredefinedExpr::Func; // [C99 6.4.2.2]
+  case tok::kw___FUNCTION__:
+    return PredefinedExpr::Function;
+  case tok::kw___FUNCDNAME__:
+    return PredefinedExpr::FuncDName; // [MS]
+  case tok::kw___FUNCSIG__:
+    return PredefinedExpr::FuncSig; // [MS]
+  case tok::kw_L__FUNCTION__:
+    return PredefinedExpr::LFunction; // [MS]
+  case tok::kw_L__FUNCSIG__:
+    return PredefinedExpr::LFuncSig; // [MS]
+  case tok::kw___PRETTY_FUNCTION__:
+    return PredefinedExpr::PrettyFunction; // [GNU]
+  }
+}
+
 /// getUDSuffixLoc - Create a SourceLocation for a ud-suffix, given the
 /// location of the token and the offset of the ud-suffix within it.
 static SourceLocation getUDSuffixLoc(Sema &S, SourceLocation TokLoc,
                                      unsigned Offset) {
   return Lexer::AdvanceToTokenCharacter(TokLoc, Offset, S.getSourceManager(),
                                         S.getLangOpts());
 }
 
if (S.LookupLiteralOperator(Scope, R, llvm::ArrayRef(ArgTy, Args.size()),
                               /*AllowStringTemplatePack*/ false,
                               /*DiagnoseMissing*/ true) == Sema::LOLR_Error)
     return ExprError();
 
   return S.BuildLiteralOperatorCall(R, OpNameInfo, Args, LitEndLoc);
 }
 
 ExprResult Sema::ActOnUnevaluatedStringLiteral(ArrayRef<Token> StringToks) {
+  std::vector<Token> ExpandedToks;
+  if (getLangOpts().MicrosoftExt)
+    StringToks = ExpandedToks = ExpandFunctionLocalPredefinedMacros(StringToks);
+
   StringLiteralParser Literal(StringToks, PP,
                               StringLiteralEvalMethod::Unevaluated);
   if (Literal.hadError)
     return ExprError();
 
   SmallVector<SourceLocation, 4> StringTokLocs;
   for (const Token &Tok : StringToks)
     StringTokLocs.push_back(Tok.getLocation());
 
   StringLiteral *Lit = StringLiteral::Create(
       Context, Literal.GetString(), StringLiteral::Unevaluated, false, {},
       &StringTokLocs[0], StringTokLocs.size());
 
   if (!Literal.getUDSuffix().empty()) {
     SourceLocation UDSuffixLoc =
         getUDSuffixLoc(*this, StringTokLocs[Literal.getUDSuffixToken()],
                        Literal.getUDSuffixOffset());
     return ExprError(Diag(UDSuffixLoc, diag::err_invalid_string_udl));
   }
 
   return Lit;
 }
 
+std::vector<Token>
+Sema::ExpandFunctionLocalPredefinedMacros(ArrayRef<Token> Toks) {
+  // MSVC treats some predefined identifiers (e.g. __FUNCTION__) as function
+  // local macros that expand to string literals that may be concatenated.
+  // These macros are expanded here (in Sema), because StringLiteralParser
+  // (in Lex) doesn't know the enclosing function (because it hasn't been
+  // parsed yet).
+  assert(getLangOpts().MicrosoftExt);
+
+  // Note: Although function local macros are defined only inside functions,
+  // we ensure a valid `currentDecl` even outside of a function. This allows
+  // expansion of macros into empty string literals without additional checks.
+  Decl *currentDecl = getCurLocalScopeDecl();
+  if (!currentDecl)
+    currentDecl = Context.getTranslationUnitDecl();
+
+  std::vector<Token> ExpandedToks;
+  ExpandedToks.reserve(Toks.size());
+  for (const Token &Tok : Toks) {
+    if (!isFunctionLocalStringLiteralMacro(Tok.getKind(), getLangOpts())) {
+      assert(tok::isStringLiteral(Tok.getKind()));
+      ExpandedToks.emplace_back(Tok);
+      continue;
+    }
+    if (isa<TranslationUnitDecl>(currentDecl)) {
+      Diag(Tok.getLocation(), diag::ext_predef_outside_function);
+    }
+    // Stringify predefined expression
+    Diag(Tok.getLocation(), diag::ext_expand_predef_ms) << Tok.getKind();
+    SmallString<64> Str;
+    llvm::raw_svector_ostream OS(Str);
+    Token &Exp = ExpandedToks.emplace_back();
+    Exp.startToken();
+    if (Tok.getKind() == tok::kw_L__FUNCTION__ ||
+        Tok.getKind() == tok::kw_L__FUNCSIG__) {
+      OS << 'L';
+      Exp.setKind(tok::wide_string_literal);
+    } else {
+      Exp.setKind(tok::string_literal);
+    }
+    OS << '"'
+       << Lexer::Stringify(PredefinedExpr::ComputeName(
+              getPredefinedExprKind(Tok.getKind()), currentDecl))
+       << '"';
+    PP.CreateString(OS.str(), Exp, Tok.getLocation(), Tok.getEndLoc());
+  }
+  return ExpandedToks;
+}
+
 /// ActOnStringLiteral - The specified tokens were lexed as pasted string
 /// fragments (e.g. "foo" "bar" L"baz").  The result string has to handle string
 /// concatenation ([C99 5.1.1.2, translation phase #6]), so it may come from
 /// multiple tokens.  However, the common case is that StringToks points to one
 /// string.
 ///
 ExprResult
 Sema::ActOnStringLiteral(ArrayRef<Token> StringToks, Scope *UDLScope) {
   assert(!StringToks.empty() && "Must have at least one string!");
 
+  std::vector<Token> ExpandedToks;
+  if (getLangOpts().MicrosoftExt)
+    StringToks = ExpandedToks = ExpandFunctionLocalPredefinedMacros(StringToks);
+
   StringLiteralParser Literal(StringToks, PP);
   if (Literal.hadError)
     return ExprError();
 
   SmallVector<SourceLocation, 4> StringTokLocs;
   for (const Token &Tok : StringToks)
     StringTokLocs.push_back(Tok.getLocation());
 
   QualType CharTy = Context.CharTy;
   StringLiteral::StringKind Kind = StringLiteral::Ordinary;
   if (Literal.isWide()) {
     CharTy = Context.getWideCharType();
     Kind = StringLiteral::Wide;
   } else if (Literal.isUTF8()) {
     if (getLangOpts().Char8)
       CharTy = Context.Char8Ty;
     Kind = StringLiteral::UTF8;
   } else if (Literal.isUTF16()) {
     CharTy = Context.Char16Ty;
     Kind = StringLiteral::UTF16;
   } else if (Literal.isUTF32()) {
     CharTy = Context.Char32Ty;
     Kind = StringLiteral::UTF32;
   } else if (Literal.isPascal()) {
     CharTy = Context.UnsignedCharTy;
   }
 
   // Warn on initializing an array of char from a u8 string literal; this
   // becomes ill-formed in C++2a.
   if (getLangOpts().CPlusPlus && !getLangOpts().CPlusPlus20 &&
       !getLangOpts().Char8 && Kind == StringLiteral::UTF8) {
     Diag(StringTokLocs.front(), diag::warn_cxx20_compat_utf8_string);
 
     // Create removals for all 'u8' prefixes in the string literal(s). This
     // ensures C++2a compatibility (but may change the program behavior when
     // built by non-Clang compilers for which the execution character set is
     // not always UTF-8).
     auto RemovalDiag = PDiag(diag::note_cxx20_compat_utf8_string_remove_u8);
     SourceLocation RemovalDiagLoc;
     for (const Token &Tok : StringToks) {
       if (Tok.getKind() == tok::utf8_string_literal) {
bool success =
       llvm::ConvertUTF8toWide(CharByteWidth, Source, ResultPtr, ErrorPtr);
   (void)success;
   assert(success);
   Target.resize(ResultPtr - &Target[0]);
 }
 
 ExprResult Sema::BuildPredefinedExpr(SourceLocation Loc,
                                      PredefinedExpr::IdentKind IK) {
-  // Pick the current block, lambda, captured statement or function.
-  Decl *currentDecl = nullptr;
-  if (const BlockScopeInfo *BSI = getCurBlock())
-    currentDecl = BSI->TheDecl;
-  else if (const LambdaScopeInfo *LSI = getCurLambda())
-    currentDecl = LSI->CallOperator;
-  else if (const CapturedRegionScopeInfo *CSI = getCurCapturedRegion())
-    currentDecl = CSI->TheCapturedDecl;
-  else
-    currentDecl = getCurFunctionOrMethodDecl();
-
+  Decl *currentDecl = getCurLocalScopeDecl();
   if (!currentDecl) {
     Diag(Loc, diag::ext_predef_outside_function);
     currentDecl = Context.getTranslationUnitDecl();
   }
 
   QualType ResTy;
   StringLiteral *SL = nullptr;
   if (cast<DeclContext>(currentDecl)->isDependentContext())
     ResTy = Context.DependentTy;
   else {
if (Ty.isNull())
     return ExprError();
   if (!TSI)
     TSI = Context.getTrivialTypeSourceInfo(Ty, LParen);
 
   return BuildSYCLUniqueStableNameExpr(OpLoc, LParen, RParen, TSI);
 }
 
 ExprResult Sema::ActOnPredefinedExpr(SourceLocation Loc, tok::TokenKind Kind) {
-  PredefinedExpr::IdentKind IK;
-
-  switch (Kind) {
-  default: llvm_unreachable("Unknown simple primary expr!");
-  case tok::kw___func__: IK = PredefinedExpr::Func; break; // [C99 6.4.2.2]
-  case tok::kw___FUNCTION__: IK = PredefinedExpr::Function; break;
-  case tok::kw___FUNCDNAME__: IK = PredefinedExpr::FuncDName; break; // [MS]
-  case tok::kw___FUNCSIG__: IK = PredefinedExpr::FuncSig; break; // [MS]
-  case tok::kw_L__FUNCTION__: IK = PredefinedExpr::LFunction; break; // [MS]
-  case tok::kw_L__FUNCSIG__: IK = PredefinedExpr::LFuncSig; break; // [MS]
-  case tok::kw___PRETTY_FUNCTION__: IK = PredefinedExpr::PrettyFunction; break;
-  }
-
-  return BuildPredefinedExpr(Loc, IK);
+  return BuildPredefinedExpr(Loc, getPredefinedExprKind(Kind));
 }
 
 ExprResult Sema::ActOnCharacterConstant(const Token &Tok, Scope *UDLScope) {
   SmallString<16> CharBuffer;
   bool Invalid = false;
   StringRef ThisTok = PP.getSpelling(Tok, CharBuffer, &Invalid);
   if (Invalid)
     return ExprError();

clang/test/Sema/ms_predefined_expr.cpp

 // RUN: %clang_cc1 %s -fsyntax-only -Wmicrosoft -verify -fms-extensions
 
-void f() {
+using size_t = __SIZE_TYPE__;
+
+// Test array initialization
+void array_init() {
  const char a[] = __FUNCTION__; // expected-warning{{initializing an array from a '__FUNCTION__' predefined identifier is a Microsoft extension}}
  const char b[] = __FUNCDNAME__; // expected-warning{{initializing an array from a '__FUNCDNAME__' predefined identifier is a Microsoft extension}}
  const char c[] = __FUNCSIG__; // expected-warning{{initializing an array from a '__FUNCSIG__' predefined identifier is a Microsoft extension}}
  const char d[] = __func__; // expected-warning{{initializing an array from a '__func__' predefined identifier is a Microsoft extension}}
  const char e[] = __PRETTY_FUNCTION__; // expected-warning{{initializing an array from a '__PRETTY_FUNCTION__' predefined identifier is a Microsoft extension}}
+ const wchar_t f[] = L__FUNCTION__; // expected-warning{{initializing an array from a 'L__FUNCTION__' predefined identifier is a Microsoft extension}}
+ const wchar_t g[] = L__FUNCSIG__; // expected-warning{{initializing an array from a 'L__FUNCSIG__' predefined identifier is a Microsoft extension}}
+}
+
+// Test function local identifiers outside of a function
+const char* g_function = __FUNCTION__;            // expected-warning{{predefined identifier is only valid inside function}}
+const char* g_function_lconcat = "" __FUNCTION__; // expected-warning{{predefined identifier is only valid inside function}} \
+                                                  // expected-warning{{expansion of predefined identifier '__FUNCTION__' to a string literal is a Microsoft extension}}
+const char* g_function_rconcat = __FUNCTION__ ""; // expected-warning{{predefined identifier is only valid inside function}} \
+                                                  // expected-warning{{expansion of predefined identifier '__FUNCTION__' to a string literal is a Microsoft extension}}
+
+namespace NS
+{
+  const char* function = __FUNCTION__;            // expected-warning{{predefined identifier is only valid inside function}}
+  const char* function_lconcat = "" __FUNCTION__; // expected-warning{{predefined identifier is only valid inside function}} \
+                                                  // expected-warning{{expansion of predefined identifier '__FUNCTION__' to a string literal is a Microsoft extension}}
+  const char* function_rconcat = __FUNCTION__ ""; // expected-warning{{predefined identifier is only valid inside function}} \
+                                                  // expected-warning{{expansion of predefined identifier '__FUNCTION__' to a string literal is a Microsoft extension}}
+
+  struct S
+  {
+    static constexpr const char* function = __FUNCTION__;            // expected-warning{{predefined identifier is only valid inside function}}
+    static constexpr const char* function_lconcat = "" __FUNCTION__; // expected-warning{{predefined identifier is only valid inside function}} \
+                                                                     // expected-warning{{expansion of predefined identifier '__FUNCTION__' to a string literal is a Microsoft extension}}
+    static constexpr const char* function_rconcat = __FUNCTION__ ""; // expected-warning{{predefined identifier is only valid inside function}} \
+                                                                     // expected-warning{{expansion of predefined identifier '__FUNCTION__' to a string literal is a Microsoft extension}}
+  };
+}
+
+template<class T, class U>
+constexpr bool is_same = false;
+template<class T>
+constexpr bool is_same<T, T> = true;
+
+template<typename T, size_t N>
+constexpr bool equal(const T (&a)[N], const T (&b)[N]) {
+  for (size_t i = 0; i < N; i++)
+    if (a[i] != b[i])
+      return false;
+  return true;
+}
+
+#define ASSERT_EQ(X, Y) static_assert(equal(X, Y))
+#define ASSERT_EQ_TY(X, Y) static_assert(is_same<decltype((X)[0]), decltype((Y)[0])>)
+
+#define _WIDE(s) L##s
+#define WIDE(s)  _WIDE(s)
+
+// Test value
+extern "C" void test_value() {
+  ASSERT_EQ(__FUNCTION__, "test_value");
+  ASSERT_EQ(__FUNCSIG__, "void __cdecl test_value(void)");
+
+  ASSERT_EQ(WIDE(__FUNCTION__), L"test_value");
+  ASSERT_EQ(WIDE(__FUNCSIG__), L"void __cdecl test_value(void)");
+}
+
+namespace PR13206 {
+  template<class T> class A {
+    public:
+      void method() {
+        ASSERT_EQ(L__FUNCTION__, L"method");
+      }
+  };
+
+  void test_template_value() {
+    A<int> x;
+    x.method();
+  }
+}
+
+// Test concatenation
+extern "C" void test_concat() {
+  ASSERT_EQ("left_" __FUNCTION__, "left_test_concat");                   // expected-warning{{expansion of predefined identifier '__FUNCTION__' to a string literal is a Microsoft extension}}
+  ASSERT_EQ("left_" __FUNCDNAME__, "left_test_concat");                  // expected-warning{{expansion of predefined identifier '__FUNCDNAME__' to a string literal is a Microsoft extension}}
+  ASSERT_EQ("left " __FUNCSIG__, "left void __cdecl test_concat(void)"); // expected-warning{{expansion of predefined identifier '__FUNCSIG__' to a string literal is a Microsoft extension}}
+
+  ASSERT_EQ(__FUNCTION__ "_right", "test_concat_right");                   // expected-warning{{expansion of predefined identifier '__FUNCTION__' to a string literal is a Microsoft extension}}
+  ASSERT_EQ(__FUNCDNAME__ "_right", "test_concat_right");                  // expected-warning{{expansion of predefined identifier '__FUNCDNAME__' to a string literal is a Microsoft extension}}
+  ASSERT_EQ(__FUNCSIG__ " right", "void __cdecl test_concat(void) right"); // expected-warning{{expansion of predefined identifier '__FUNCSIG__' to a string literal is a Microsoft extension}}
+
+  ASSERT_EQ("left_" __FUNCTION__ "_right", "left_test_concat_right");                   // expected-warning{{expansion of predefined identifier '__FUNCTION__' to a string literal is a Microsoft extension}}
+  ASSERT_EQ("left_" __FUNCDNAME__ "_right", "left_test_concat_right");                  // expected-warning{{expansion of predefined identifier '__FUNCDNAME__' to a string literal is a Microsoft extension}}
+  ASSERT_EQ("left " __FUNCSIG__ " right", "left void __cdecl test_concat(void) right"); // expected-warning{{expansion of predefined identifier '__FUNCSIG__' to a string literal is a Microsoft extension}}
+
+  ASSERT_EQ(__FUNCTION__ "/" __FUNCSIG__, "test_concat/void __cdecl test_concat(void)"); // expected-warning{{expansion of predefined identifier '__FUNCTION__' to a string literal is a Microsoft extension}} \
+                                                                                         // expected-warning{{expansion of predefined identifier '__FUNCSIG__' to a string literal is a Microsoft extension}}
+}
+
+extern "C" void test_wide_concat() {
+  // test L"" + ""
+  ASSERT_EQ(L"" __FUNCTION__, L__FUNCTION__); // expected-warning{{expansion of predefined identifier '__FUNCTION__' to a string literal is a Microsoft extension}}
+  ASSERT_EQ(L"" __FUNCSIG__, L__FUNCSIG__);   // expected-warning{{expansion of predefined identifier '__FUNCSIG__' to a string literal is a Microsoft extension}}
+
+  // test Lx + ""
+  ASSERT_EQ(L__FUNCTION__, L__FUNCTION__ ""); // expected-warning{{expansion of predefined identifier 'L__FUNCTION__' to a string literal is a Microsoft extension}}
+  ASSERT_EQ(L__FUNCSIG__, L__FUNCSIG__ "");   // expected-warning{{expansion of predefined identifier 'L__FUNCSIG__' to a string literal is a Microsoft extension}}
+
+  ASSERT_EQ(L"left_" L__FUNCTION__, L"left_test_wide_concat");                   // expected-warning{{expansion of predefined identifier 'L__FUNCTION__' to a string literal is a Microsoft extension}}
+  ASSERT_EQ(L"left " L__FUNCSIG__, L"left void __cdecl test_wide_concat(void)"); // expected-warning{{expansion of predefined identifier 'L__FUNCSIG__' to a string literal is a Microsoft extension}}
+
+  ASSERT_EQ(L__FUNCTION__ L"_right", L"test_wide_concat_right");                   // expected-warning{{expansion of predefined identifier 'L__FUNCTION__' to a string literal is a Microsoft extension}}
+  ASSERT_EQ(L__FUNCSIG__ L" right", L"void __cdecl test_wide_concat(void) right"); // expected-warning{{expansion of predefined identifier 'L__FUNCSIG__' to a string literal is a Microsoft extension}}
+
+  ASSERT_EQ(L"left_" L__FUNCTION__ L"_right", L"left_test_wide_concat_right");                   // expected-warning{{expansion of predefined identifier 'L__FUNCTION__' to a string literal is a Microsoft extension}}
+  ASSERT_EQ(L"left " L__FUNCSIG__ L" right", L"left void __cdecl test_wide_concat(void) right"); // expected-warning{{expansion of predefined identifier 'L__FUNCSIG__' to a string literal is a Microsoft extension}}
+
+  ASSERT_EQ(L__FUNCTION__ L"/" L__FUNCSIG__, L"test_wide_concat/void __cdecl test_wide_concat(void)"); // expected-warning{{expansion of predefined identifier 'L__FUNCTION__' to a string literal is a Microsoft extension}} \
+                                                                                                       // expected-warning{{expansion of predefined identifier 'L__FUNCSIG__' to a string literal is a Microsoft extension}}
+}
+
+void test_encoding() {
+  ASSERT_EQ_TY("" __FUNCTION__, "");     // expected-warning{{expansion of predefined identifier '__FUNCTION__' to a string literal is a Microsoft extension}}
+  ASSERT_EQ_TY(L"" __FUNCTION__, L"");   // expected-warning{{expansion of predefined identifier '__FUNCTION__' to a string literal is a Microsoft extension}}
+  ASSERT_EQ_TY(L"" L__FUNCTION__, L"");  // expected-warning{{expansion of predefined identifier 'L__FUNCTION__' to a string literal is a Microsoft extension}}
+  ASSERT_EQ_TY("" L__FUNCTION__, L"");   // expected-warning{{expansion of predefined identifier 'L__FUNCTION__' to a string literal is a Microsoft extension}}
+  ASSERT_EQ_TY(u8"" __FUNCTION__, u8""); // expected-warning{{expansion of predefined identifier '__FUNCTION__' to a string literal is a Microsoft extension}}
+  ASSERT_EQ_TY(u"" __FUNCTION__, u"");   // expected-warning{{expansion of predefined identifier '__FUNCTION__' to a string literal is a Microsoft extension}}
+  ASSERT_EQ_TY(U"" __FUNCTION__, U"");   // expected-warning{{expansion of predefined identifier '__FUNCTION__' to a string literal is a Microsoft extension}}
+                                         //
+  ASSERT_EQ_TY(__FUNCTION__ L"", L"");   // expected-warning{{expansion of predefined identifier '__FUNCTION__' to a string literal is a Microsoft extension}}
+  ASSERT_EQ_TY(__FUNCTION__ u8"", u8""); // expected-warning{{expansion of predefined identifier '__FUNCTION__' to a string literal is a Microsoft extension}}
+  ASSERT_EQ_TY(__FUNCTION__ u"", u"");   // expected-warning{{expansion of predefined identifier '__FUNCTION__' to a string literal is a Microsoft extension}}
+  ASSERT_EQ_TY(__FUNCTION__ U"", U"");   // expected-warning{{expansion of predefined identifier '__FUNCTION__' to a string literal is a Microsoft extension}}
+}
+
+extern "C" void test_𐀀() {
+  ASSERT_EQ(U"" __FUNCTION__, U"test_𐀀");   // expected-warning{{expansion of predefined identifier '__FUNCTION__' to a string literal is a Microsoft extension}}
+  ASSERT_EQ(u"" __FUNCTION__, u"test_𐀀");   // expected-warning{{expansion of predefined identifier '__FUNCTION__' to a string literal is a Microsoft extension}}
+  ASSERT_EQ(u8"" __FUNCTION__, u8"test_𐀀"); // expected-warning{{expansion of predefined identifier '__FUNCTION__' to a string literal is a Microsoft extension}}
+}
+
+template<typename T>
+void unused(T);
+
+void test_invalid_encoding() {
+  unused(u8"" L__FUNCTION__); // expected-warning{{expansion of predefined identifier 'L__FUNCTION__' to a string literal is a Microsoft extension}} \
+                              // expected-error{{unsupported non-standard concatenation of string literals}}
+  unused(u"" L__FUNCTION__);  // expected-warning{{expansion of predefined identifier 'L__FUNCTION__' to a string literal is a Microsoft extension}} \
+                              // expected-error{{unsupported non-standard concatenation of string literals}}
+  unused(U"" L__FUNCTION__);  // expected-warning{{expansion of predefined identifier 'L__FUNCTION__' to a string literal is a Microsoft extension}} \
+                              // expected-error{{unsupported non-standard concatenation of string literals}}
+}
+
+constexpr size_t operator""_len(const char*, size_t len) {
+  return len;
+}
+
+void test_udliteral() {
+  static_assert(__FUNCTION__ ""_len == 14); // expected-warning{{expansion of predefined identifier '__FUNCTION__' to a string literal is a Microsoft extension}}
+}
+
+void test_static_assert() {
+  static_assert(true, __FUNCTION__); // expected-warning{{expansion of predefined identifier '__FUNCTION__' to a string literal is a Microsoft extension}}
+}
+
+void test_char_injection(decltype(sizeof('"')), decltype(sizeof("()"))) {
+  unused("" __FUNCSIG__); // expected-warning{{expansion of predefined identifier '__FUNCSIG__' to a string literal is a Microsoft extension}}
 }

clang/test/Sema/ms_wide_predefined_expr.cpp

-// RUN: %clang_cc1 %s -fsyntax-only -Wno-unused-value -Wmicrosoft -verify -fms-extensions
-// expected-no-diagnostics
-
-// Wide character predefined identifiers
-#define _STR2WSTR(str) L##str
-#define STR2WSTR(str) _STR2WSTR(str)
-void abcdefghi12(void) {
- const wchar_t (*ss)[12] = &STR2WSTR(__FUNCTION__);
- static int arr[sizeof(STR2WSTR(__FUNCTION__))==12*sizeof(wchar_t) ? 1 : -1];
- const wchar_t (*ss2)[31] = &STR2WSTR(__FUNCSIG__);
- static int arr2[sizeof(STR2WSTR(__FUNCSIG__))==31*sizeof(wchar_t) ? 1 : -1];
-}
-
-namespace PR13206 {
-void foo(const wchar_t *);
-
-template<class T> class A {
-public:
- void method() {
-  foo(L__FUNCTION__);
- }
-};
-
-void bar() {
- A<int> x;
- x.method();
-}
-}