This is an archive of the discontinued LLVM Phabricator instance.

[clang] Implement Change scope of lambda trailing-return-type
ClosedPublic

Authored by cor3ntin on Feb 7 2022, 6:44 AM.

Details

Summary

Implement P2036R3.

Captured variables by copy (explicitely or not), are deduced
correctly at the point we know whether the lambda is mutable,
and ill-formed before that.

Up until now, the entire lambda declaration up to the start of the body would be parsed in the parent scope, such that capture would not be available to look up.

The scoping is changed to have an outer lambda scope, followed by the lambda prototype and body.

The lambda scope is necessary because there may be a template scope between the start of the lambda (to which we want to attach the captured variable) and the prototype scope.

We also need to introduce a declaration context to attach the captured variable to (and several parts of clang assume captures are handled from the call operator context), before we know the type of the call operator.

The order of operations is as follow:

  • Parse the init capture in the lambda's parent scope
  • Introduce a lambda scope
  • Create the lambda class and call operator
  • Add the init captures to the call operator context and the lambda scope. But the variables are not capured yet (because we don't know their type).

Instead, explicit captures are stored in a temporary map that conserves the order of capture (for the purpose of having a stable order in the ast dumps).

  • A flag is set on LambdaScopeInfo to indicate that we have not yet injected the captures.
  • The parameters are parsed (in the parent context, as lambda mangling recurses in the parent context, we couldn't mangle a lambda that is attached to the context of a lambda whose type is not yet known).
  • The lambda qualifiers are parsed, at this point We can switch (for the second time) inside the lambda context, unset the flag indicating that we have not parsed the lambda qualifiers,

record the lambda is mutable and capture the explicit variables.

  • We can parse the rest of the lambda type, transform the lambda and call operator's types and also transform the call operator to a template function decl where necessary.

At this point, both captures and parameters can be injected in the body's scope. When trying to capture an implicit variable, if we are before the qualifiers of a lambda, we need to remember that the variables are still in the parent's context (rather than in the call operator's).

Diff Detail

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
cor3ntin added a reviewer: Restricted Project.Mar 30 2022, 11:07 AM

I've just skimmed over the patch and it is a little bit hard to understand for people who don't have a strong background. Is it possible to split this one into several parts?

clang/lib/Sema/Scope.cpp
71

This looks like:

clang/lib/Sema/SemaExpr.cpp
18545

If this is only used in current file, we should mark it as static or put in anonymous namespace. And I think this lacks a comment

cor3ntin updated this revision to Diff 419658.Apr 1 2022, 12:55 AM
  • Add missing static
cor3ntin marked an inline comment as done.Apr 1 2022, 12:56 AM
cor3ntin added inline comments.
clang/lib/Sema/SemaExpr.cpp
18545

Good catch, thanks

cor3ntin marked an inline comment as done.Apr 1 2022, 12:58 AM

I've just skimmed over the patch and it is a little bit hard to understand for people who don't have a strong background. Is it possible to split this one into several parts?

Unfortunately, I don't thing that would be very practical. The patch is not small but what it changes is rather small, and if we tried to split it the intermediate patches would be inconsistent and untested

I've just skimmed over the patch and it is a little bit hard to understand for people who don't have a strong background. Is it possible to split this one into several parts?

Unfortunately, I don't thing that would be very practical. The patch is not small but what it changes is rather small, and if we tried to split it the intermediate patches would be inconsistent and untested

Yeah, but I indeed found some NFC changes... I think it might be helpful to split it as NFC parts and other parts.

clang/include/clang/AST/DeclCXX.h
1816–1821

It looks like this one is not used, is it?

clang/include/clang/Sema/ScopeInfo.h
840

I suggest a comment to describe this one.

842

I think the name Before* is a little bit confusing. From the context, it would be set to true in ActOnLambdaIntroducerand be set to false in ActOnLambdaClosureQualifiers. So it looks like something IsInRangeOf.... The name before implies that it should be true by default and be false after some point...

I am not sure if my point is clear...

clang/include/clang/Sema/Sema.h
6870

The comment looks like a draft.

clang/lib/Parse/ParseExprCXX.cpp
1435

Unintended change?

1486

It looks a little bit. It looks like if the code is () and HasParameters would still be true.

clang/lib/Sema/Scope.cpp
71

This isn't addressed. I mean the flags would always be true if the last clause get evaluated.

clang/lib/Sema/SemaExpr.cpp
18550

Could we declare it as a boolean.

clang/lib/Sema/SemaLambda.cpp
358–364

The comment says Start while the function name starts with finish. It looks odd...

512–514

How about:

892–979

Add a static qualifier.

cor3ntin updated this revision to Diff 421227.Apr 7 2022, 8:30 AM
cor3ntin marked 4 inline comments as done.

Address ChuanqiXu's feedback

cor3ntin updated this revision to Diff 421228.Apr 7 2022, 8:31 AM

Formatting.

clang/include/clang/AST/DeclCXX.h
1816–1821

Thanks., you are right, i wrote that before realizing i did not need it.

clang/include/clang/Sema/ScopeInfo.h
842

I struggle to find a better name, but I added a comment. Does that help?

clang/include/clang/Sema/Sema.h
6870

Thanks for noticing that, I added a longer comment

clang/lib/Parse/ParseExprCXX.cpp
1486

Which is what we want to track. But it was not clear, I renamed the variable to HasParentheses

clang/lib/Sema/Scope.cpp
71

I'm not sure I understand your point. We are checking that it's a function scope that is not a lambda scope.
They are probably other ways to write that but they aren't clearer

clang/lib/Sema/SemaLambda.cpp
358–364

The comment appertain to the startLambdaDefinition below, i fixed it.
I also renamed finishLambdaMethodType to buildTypeForLambdaCallOperator` which is clearer`

ChuanqiXu added inline comments.Apr 7 2022, 8:11 PM
clang/include/clang/Sema/ScopeInfo.h
842

I'm not good at naming too... I am wondering a name with words like range could be fine.

843

I hope the comment explains the meaning of unsigned here. If it means index, I think SmallVector would make code simpler.

clang/lib/Sema/Scope.cpp
71

I mean the suggested change is equal to the original one and it is shorter. I don't think it would lost any information since they are all in the same line.

clang/lib/Sema/SemaExpr.cpp
18557
clang/lib/Sema/SemaLambda.cpp
892–979

I guess it is needed to explain the meaning of Unsafe

894–897
1075–1080

I suggest to add a assertion for Var to check it is not null

1136–1140
1178–1231

If DelayedCaptures is SmallVector, we could write:


I observed there a lot continue in the loop. So we could hoist the definition of Var to the start of the loop. And we could fill DelayedCaptures by RAII.

1189

Do you think it is better to replace I with std:distance(Intro.Captures.begin(), C)? I prefer it since we could reduce the scope of I in the style.

1190

It looks like we could define PrevCaptureLoc in the loop.

1210

We prefer C++20 than C++2a now.

1308–1309
cor3ntin updated this revision to Diff 421454.Apr 8 2022, 1:48 AM
cor3ntin marked 6 inline comments as done.

Address ChuanqiXu's comment

cor3ntin updated this revision to Diff 421456.Apr 8 2022, 1:57 AM
  • Address ChuanqiXu's comments
  • Add an entry in ReleaseNotes
clang/lib/Sema/Scope.cpp
71

LambdaScope is a constant here
Both FunctionPrototypeScope and FunctionPrototypeScope | LambdaScope are valid values, and we only want the condition to be true when flags & FunctionPrototypeScope is true and flags & LambdaScope is false.
Maybe we misunderstand each other

clang/lib/Sema/SemaLambda.cpp
892–979

Agreed, I added a comment

1190

I think that would be worse, as this is the location of the previous capture.
it would looks something like.
SourceLocation PrevCaptureLoc = C == Intro.Captures.begin() ? {} : (C-1)->Loc

cor3ntin updated this revision to Diff 421461.Apr 8 2022, 2:13 AM

Add dependent tests

Some drive by comments, I plan to do a more thorough review when I have the chance.

clang/docs/ReleaseNotes.rst
111

Unintended whitespace change?

258
clang/include/clang/Basic/DiagnosticSemaKinds.td
7714
clang/include/clang/Sema/Scope.h
47–51

Unintended whitespace changes?

145–147
clang/include/clang/Sema/ScopeInfo.h
840

Can you re-flow the comments to 80 cols?

clang/lib/Parse/ParseExprCXX.cpp
1289

This call surprises me here -- I would expect to see this called from ParseLambdaIntroducer() so that anyone who parses a lambda introducer gets the action.

clang/lib/Sema/Scope.cpp
70–73

Should re-flow comments with the above line.

clang/lib/Sema/SemaDecl.cpp
14352 ↗(On Diff #421461)

Unintended whitespace change?

cor3ntin updated this revision to Diff 421596.Apr 8 2022, 11:19 AM
cor3ntin marked 6 inline comments as done.

Address Aaron's comments.

cor3ntin updated this revision to Diff 421598.Apr 8 2022, 11:21 AM

Missed a comment in Scope.h

ChuanqiXu accepted this revision.Apr 11 2022, 7:19 PM

I've tried my best to review this one and it looks good to me basically. Since @aaron.ballman plans to review this one. I think it'd better to wait for this respond.

clang/lib/Sema/Scope.cpp
71

Oh, sorry, I misread the code. The code now should be correct.

clang/lib/Sema/SemaLambda.cpp
1136–1140

I feel like my suggested change is simpler.

This revision is now accepted and ready to land.Apr 11 2022, 7:19 PM
aaron.ballman added inline comments.Apr 12 2022, 6:20 AM
clang/include/clang/Basic/DiagnosticSemaKinds.td
7714

It looks like this comment may have been missed.

clang/lib/Parse/ParseExprCXX.cpp
1289

I'd still like to hear why this is being called here instead of when parsing the lambda introducer itself.

cor3ntin added inline comments.Apr 12 2022, 6:31 AM
clang/include/clang/Basic/DiagnosticSemaKinds.td
7714

That doesn't work

[a]<typename x = decltype(a)>(decltype(a) )
                      ^ nope            ^ double nope
clang/lib/Parse/ParseExprCXX.cpp
1289

This is preexisting that sema for the introducer was done in this function.
The only thing that change is that we are doing it before parsiong the parameter instead of after, in the same action.
And we need to do it here, because of the scope introduction

aaron.ballman added inline comments.Apr 12 2022, 7:58 AM
clang/include/clang/Basic/DiagnosticSemaKinds.td
7714

Thank you for this! I think we should go with %select to name where it cannot be used. e.g., cannot be used in a lambda %select{parameter list|template parameter list|whatever}1

clang/test/SemaCXX/lambda-capture-type-deduction.cpp
62

Can you also add test coverage for when the requires comes *before* the parens as in: [x] requires(is_same<const int &, decltype((x))>)(){}();?

I think you should also look at the grammar for lambda-expression to make sure we're not missing other coverage (use in attributes also appears to be untested).

aaron.ballman added inline comments.Apr 12 2022, 7:59 AM
clang/lib/Parse/ParseExprCXX.cpp
1289

This is preexisting that sema for the introducer was done in this function. The only thing that change is that we are doing it before parsiong the parameter instead of after, in the same action. And we need to do it here, because of the scope introduction

Ah, okay, that's a good reason not to muck with it in this patch. I still think that design is a bit suspect, but it's not a new issue (thank you for pointing that out).

cor3ntin updated this revision to Diff 422282.Apr 12 2022, 10:21 AM

After offline discussion with Aaron, we decided that "can not appear here" was a
sufficient diagnostic, so I have implemented that.
A better solution would be to track which captures fail, and collect the diagnostics
to emit them after the parameter list, but this seem to add a lot of complexity
for something few people are likely to encounter.

  • Add tests
  • A diagnostic for use before capture could be emited twice when parsing

a template parameter list, this was fixed.

aaron.ballman accepted this revision.Apr 13 2022, 9:46 AM

LGTM! When landing, please revert the changes to SemaDecl.cpp (no need to churn that file) and apply the one suggested refactoring (I commented to make it more clear which one I mean).

clang/lib/Sema/SemaLambda.cpp
1136–1140

+1 to this suggestion being a bit easier to read.

cor3ntin added inline comments.Apr 13 2022, 10:00 AM
clang/lib/Sema/SemaLambda.cpp
1136–1140

With that changes, a capture that is not in the delayed capture list forces us to go through the list of capture anyway, which seems like a waste.
So this is not just about readability. it should still work, and only really matter for error cases, but it's still not identical code.

cor3ntin added inline comments.Apr 13 2022, 10:03 AM
clang/lib/Sema/SemaLambda.cpp
1136–1140

Alternatively, I guess I can change the order of operation to ensure Var is never null here, and that would probably be cleaner!

aaron.ballman added inline comments.Apr 13 2022, 10:07 AM
clang/lib/Sema/SemaLambda.cpp
1136–1140

Ah, thank you for pointing out that this was also about performance. If you're able to clean it up by changing the order of operations, that'd be good, but if it turns into a slog, I don't insist.

cor3ntin updated this revision to Diff 422557.Apr 13 2022, 10:18 AM

Cleanup SemaLambda as discussed with Aaron

This revision was landed with ongoing or failed builds.Apr 13 2022, 11:00 AM
This revision was automatically updated to reflect the committed changes.

Seems like this is breaking clang bootstrapping?

llvm/include/llvm/CodeGen/LiveInterval.h:630:53: error: captured variable 'Idx' cannot appear here
              [=](std::remove_reference_t<decltype(*Idx)> V,
                                                    ^

Yes, working on a fix as we speak.
The meaning of that code changed (in all c++ language modes), I'm
currently trying to find if we have any other issue of that sort and will
commit a fix in a few hours

Yes, working on a fix as we speak.
The meaning of that code changed (in all c++ language modes), I'm
currently trying to find if we have any other issue of that sort and will
commit a fix in a few hours

OK, I'll revert in the meantime to unbreak bots.

rsmith added a subscriber: rsmith.Apr 13 2022, 9:37 PM

Hey @cor3ntin, sorry, I reverted this again. It's breaking thread-safety annotations on lambdas in a pretty severe way:

Mutex m;
auto lambda = [=]() EXCLUSIVE_LOCKS_REQUIRED(m) {...}

doesn't compile any more. Presumably we should be parsing the attribute in the enclosing context rather than in the half-way-inside-the-lambda context in which we parse parameters?

cor3ntin reopened this revision.Apr 13 2022, 11:56 PM

@rsmith Sorry about that, I will investigate. We found preexisting issues - attributes seem to ODR use when they should maybe not and I could not thing of attributes that would refer to capture, so i didn't spent enough time looking at attributes.

This revision is now accepted and ready to land.Apr 13 2022, 11:56 PM

Presumably we should be parsing the attribute in the enclosing context rather than in the half-way-inside-the-lambda context in which we parse parameters?

That's a good question.

void f() {
int y;
(void)[=, x = 1]() 
    __attribute__((diagnose_if(!is_same<decltype((y)), int&>, "wrong type", "warning")))  // do we want this to work?
    __attribute__((diagnose_if(x != 1, "wrong type", "warning"))) // do we want this to work?
    mutable {}();
}

I think we *probably* want both of this to compile and evaluate the conditions consistently with the new rule for trailing return type/requires/noexcept specifier (in that case triggering no warning).
So the consistent approach would be to parse them in the context of the call operator, once the captures and their type is known.

Is there any reason we should not be consistent with the spirit of the c++ spec here?

cor3ntin updated this revision to Diff 422806.Apr 14 2022, 4:11 AM
Properly support GNU Attributes

Perform delayed parsing of GNU attributes (using code that already
existed to delay the parsing of class methods), so that
gnu attributes can refer to captured variables,
with the same semantics as decltype/noexcept/requires.

In order we:
 * Lex the attributes
 * Parse the mutable keyword
 * Inject parameter names in the scope of the call operator
   so they get found during attribute parsing
 * Parse the attributes and attach them to the declarator
   being parsed.
aaron.ballman added inline comments.Apr 14 2022, 10:16 AM
clang/lib/Parse/ParseExprCXX.cpp
1234

I think this should likely be named ParseLambdaLexedGNUAttributeArgs() instead, because this is pretty specific to GNU-style attribute argument lists and shouldn't be used for other attribute syntaxes, correct?

1253–1256

This comment reads a bit oddly to me. I think it may be better as "After parsing attribute arguments, we've either reached the EOF token (signaling that parsing was successful) or we have tokens we need to consume until we reach the EOF."

1258

Then I think we can assert that the token is EOF here and consume it always.

1371
1401

I think you should add an assert before this call to ensure that all attributes are GNU ones.

1490

Do we have test coverage for this change? (It seems orthogonal to your patch, so this might be worth splitting out.)

cor3ntin marked an inline comment as done.Apr 14 2022, 10:29 AM
cor3ntin added inline comments.
clang/lib/Parse/ParseExprCXX.cpp
1253–1256

Agreed

1258

Agreed, that seems better

1371

It's more like there is no __declspec that can refer to expression so this was never implemented

1401

I'm not sure that gains much?
In the future if late parsing is implemented for __declspec this would just work , unless we add an assert. I'd be fine adding it though.

1490

I'm happy doing that, or to add a test somewhere.

aaron.ballman added inline comments.Apr 14 2022, 10:40 AM
clang/lib/Parse/ParseExprCXX.cpp
1401

Because ParseLambdaLexedAttribute() specifically parses GNU attribute args and not declspec ones, I think that function should be renamed. Once that function is renamed to say GNU explicitly, then there's a question of what happens when we hand it late parsed declspec attribute. Which is why I was suggesting a comment above to mention that declspec attributes don't late parse anyway.

Another way to handle this that's more general would be to change ParseLambdaLexedAttribute() to parse different args depending on the syntax of the attribute (then it can keep its generic name).

cor3ntin marked an inline comment as done.Apr 14 2022, 10:45 AM
cor3ntin added inline comments.
clang/lib/Parse/ParseExprCXX.cpp
1490

This was actually a _very_ stupid change on my part, the previous code was correct. ie, msvc does not support [] __declspec(deprecated) {};
https://godbolt.org/z/cEKP44rY1
and so the omission of __declspec here, which i thought was a bug, is actually very purposeful

aaron.ballman added inline comments.Apr 14 2022, 10:48 AM
clang/lib/Parse/ParseExprCXX.cpp
1490

Thank you for double-checking that; I thought that might be the case, but I hadn't tested it myself yet.

cor3ntin updated this revision to Diff 422920.EditedApr 14 2022, 11:08 AM
cor3ntin marked 2 inline comments as done.

Address Aaron's comments

cor3ntin added inline comments.Apr 14 2022, 11:13 AM
clang/lib/Parse/ParseExprCXX.cpp
1401

Okay, after actually reading the code, we can't actually assert on that.

LateParsedAttribute is really LateParsedGNUAttribute. And maybe we want to rename that after this patch. It can never contain anything else, and because __attribute__ was eaten, we don't really have anything to check for.

aaron.ballman added inline comments.Apr 14 2022, 11:19 AM
clang/lib/Parse/ParseExprCXX.cpp
1401

Oh, well TIL. :-D

Yeah, don't bother with the assert then. Any probably don't bother with the rename I suggested either.

(Someday) we'll either rename LateParsedAttribute to be more correct, or we'll extend it to support more syntaxes and keep the same name. Either way, the interfaces in your patch will be sufficiently clear.

aaron.ballman accepted this revision.Apr 14 2022, 11:41 AM

LGTM, thank you!

This revision was landed with ongoing or failed builds.Apr 15 2022, 7:51 AM
This revision was automatically updated to reflect the committed changes.

I note that building on Ubuntu, I am now seeing failures like:

env/gbuildtojson/gbuildtojson.cxx:10:
In file included from /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/algorithm:74:
In file included from /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/pstl/glue_algorithm_defs.h:13:
In file included from /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:61:
In file included from /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/unordered_map:52:
In file included from /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/debug/unordered_map:49:
/usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/debug/safe_unordered_container.h:71:28: error: captured variable '__local_end' cannot appear here

[__local_end](__decltype(__local_end) __it)
                         ^

/usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/debug/safe_unordered_container.h:71:4: note: variable '__local_end' is explicitly captured here

[__local_end](__decltype(__local_end) __it)
 ^

/usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/debug/safe_unordered_container.h:69:2: note: '__local_end' declared here

auto __local_end = _M_cont()._M_base().cend(0);
^

/usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/debug/safe_unordered_container.h:79:44: error: captured variable '__end' cannot appear here

this->_M_invalidate_if([__end](__decltype(__end) __it)
                                          ^

/usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/debug/safe_unordered_container.h:79:26: note: variable '__end' is explicitly captured here

this->_M_invalidate_if([__end](__decltype(__end) __it)
                        ^

/usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/debug/safe_unordered_container.h:78:2: note: '__end' declared here

auto __end = _M_cont()._M_base().cend();
^

@grandinj Thanks for notifying me of this issue, and sorry for the inconvenience.

There is a fix here https://reviews.llvm.org/D123909
Should the review of that linger too long, I'll revert this patch.

The remediation has landed, sorry again for the trouble this caused, I'll keep an eye out for further issues

thanks for the quick response

rtrieu added a subscriber: rtrieu.Apr 18 2022, 10:03 AM

This seems to be acting weird in template instantations. Here's an example where the lambda only errors inside a template.

template <class>
int foo(int x = 0) {
    auto lambda = [x = x+1]() -> decltype(x) {
        return x;
    };
    return -1;
}

// no template
int bar(int x = 0) {
    auto lambda = [x = x+1]() -> decltype(x) {
        return x;
    };
    return -1;
}

int a = foo<int>();  // error
int b = bar();  // no  error
/tmp/lambda.cc:4:16: error: variable 'x' cannot be implicitly captured in a lambda with no capture-default specified
        return x;
               ^
/tmp/lambda.cc:17:9: note: in instantiation of function template specialization 'foo<int>' requested here
int a = foo<int>();  // error
        ^
/tmp/lambda.cc:3:20: note: 'x' declared here
    auto lambda = [x = x+1]() -> decltype(x) {
                   ^
/tmp/lambda.cc:3:19: note: lambda expression begins here
    auto lambda = [x = x+1]() -> decltype(x) {
                  ^
/tmp/lambda.cc:3:27: note: capture 'x' by value
    auto lambda = [x = x+1]() -> decltype(x) {
                          ^
                          , x
/tmp/lambda.cc:3:27: note: capture 'x' by reference
    auto lambda = [x = x+1]() -> decltype(x) {
                          ^
                          , &x
/tmp/lambda.cc:3:20: note: default capture by value
    auto lambda = [x = x+1]() -> decltype(x) {
                   ^
                   =, 
/tmp/lambda.cc:3:20: note: default capture by reference
    auto lambda = [x = x+1]() -> decltype(x) {
                   ^
                   &, 
1 error generated.
bgraur added a subscriber: bgraur.Apr 19 2022, 6:03 AM

We've also been seeing failures in our downstream Arm compiler when running the Perennial C++14 compliance tests related to this change. Specifically, the tests expect a diagnostic to be issued when the lambda capture variable is outside of the lambda's {} scope. Another tests uses the lambda capture variable in a decltype-style return type which is also outside of the scope.

Do these failures sound like what others have run into?

This seems to be acting weird in template instantations. Here's an example where the lambda only errors inside a template.

Thanks for notifying me of that. I will land a fix shortly https://reviews.llvm.org/D124012

Sorry but I've reverted this patch and all its fixups in c79e6007edef4b0044be93c4ffff64dc806ac687 and 0f5dbfd29ae0df215a01aff80d29255bb799fed0 .
See https://reviews.llvm.org/D123909#3461716 for another case not considered.

Sorry but I've reverted this patch and all its fixups in c79e6007edef4b0044be93c4ffff64dc806ac687 and 0f5dbfd29ae0df215a01aff80d29255bb799fed0 .
See https://reviews.llvm.org/D123909#3461716 for another case not considered.

Please double check with the patch author and reviewers before unilaterally reverting multiple commits with no notice and no failing build bots. I do not see a valid justification for these reverts -- the concerns have been true positives so far (or have generated core issues that WG21 is still discussing), and @cor3ntin has been highly responsive with addressing the fallout. This is a significant amount of churn that I don't think should have happened.

MaskRay added a comment.EditedApr 20 2022, 11:15 AM

Sorry but I've reverted this patch and all its fixups in c79e6007edef4b0044be93c4ffff64dc806ac687 and 0f5dbfd29ae0df215a01aff80d29255bb799fed0 .
See https://reviews.llvm.org/D123909#3461716 for another case not considered.

Please double check with the patch author and reviewers before unilaterally reverting multiple commits with no notice and no failing build bots. I do not see a valid justification for these reverts -- the concerns have been true positives so far (or have generated core issues that WG21 is still discussing), and @cor3ntin has been highly responsive with addressing the fallout. This is a significant amount of churn that I don't think should have happened.

Reply to both this message and https://reviews.llvm.org/D123909#3462560:

I am sorry but this thread has gained reports from multiple independent parties about breakage, and I just thought the followups did not fix all issues.
I confess I do not really understand the subtle areas in the language.
My decision to revert was mostly driven by fixups' descriptions like "but did not properly handled dependant context,"

  • which made me believe there were indeed bugs in the implementation and new corner cases just emerged.

I'd agree that there is invalid code which is not correctly rejected, and the new code correctly rejects it (if only for such cases there is no justification for reverts), but the fixups mixed with bugfixes made the whole picture unclear.

I apology if I did make wrong reverts.
At this point, the reverts have been made. I hope that starting from a clean state is not too bad. I am grateful to @cor3ntin who is highly responsive with addressing the fallout.

Sorry but I've reverted this patch and all its fixups in c79e6007edef4b0044be93c4ffff64dc806ac687 and 0f5dbfd29ae0df215a01aff80d29255bb799fed0 .
See https://reviews.llvm.org/D123909#3461716 for another case not considered.

Please double check with the patch author and reviewers before unilaterally reverting multiple commits with no notice and no failing build bots. I do not see a valid justification for these reverts -- the concerns have been true positives so far (or have generated core issues that WG21 is still discussing), and @cor3ntin has been highly responsive with addressing the fallout. This is a significant amount of churn that I don't think should have happened.

Reply to both this message and https://reviews.llvm.org/D123909#3462560:

I am sorry but this thread has gained reports from multiple independent parties about breakage, and I just thought the followups did not fix all issues.

AFAIK, all issues have either been addressed, are true positives, or are a matter for WG21 to figure out what to do with. However, I'm not certain it was clear from the reviews that this was the case, so it's understandable there's some amount of "did we get everything?" confusion.

I confess I do not really understand the subtle areas in the language.
My decision to revert was mostly driven by fixups' descriptions like "but did not properly handled dependant context,"

  • which made me believe there were indeed bugs in the implementation and new corner cases just emerged.

I'd agree that there is invalid code which is not correctly rejected, but the fixups mixed with bugfixes made the whole picture unclear.

The trouble here was: WG21 adopted a proposal intended as a bugfix, but it was never implemented before (to my knowledge). We went and did the implementation and discovered that the bugfix paper actually introduces different bugs (hence us breaking libstdc++; the diagnostic was valid, but the paper didn't likely intend to make that code invalid). CWG2569 was filed to try to address this, but the issue hasn't been resolved by WG21 yet. So what @cor3ntin has done is to address *only* the libstdc++ failure case (to keep important system headers working) until WG21 figures out whether we need to be relaxed in more cases.

This does mean that there may be some code broken that WG21 will later decide should not be broken, but that's a matter for the Core Issue; these changes are to implement P2036R3 which is a C++23 feature.

I apology if I did make wrong reverts.
At this point, the reverts have been made. I hope that starting from a clean state is not too bad. I am grateful to @cor3ntin who is highly responsive with addressing the fallout.

Thanks for the apology, but I still think the reverts were premature, so hopefully we don't do this again. The issue with reverting all of these is that you put the burden back onto a relatively new contributor to try to figure out how to reapply all these changes when there wasn't actually a need to do so.

In terms of where we go from here, there's a few options:

  1. revert the reverts; this is easiest, but causes the most churn
  2. merge all three patches into one (squashed) patch and commit that; this requires a little bit more work, but causes less churn
  3. assume that P2036R3 is not implementable until WG21 resolves CWG2569 because it breaks too much code

I think I have a preference for #2 as that will hopefully leave us with the most clear git history for when we do a blame later, but I could live with #1. If we find that there is significant broken code still, but it's not in system headers, then I think we should explore #3, but I don't think we have evidence that we're in this situation either.

amyk added a subscriber: amyk.Apr 20 2022, 11:44 AM

We've also been seeing failures in our downstream Arm compiler when running the Perennial C++14 compliance tests related to this change. Specifically, the tests expect a diagnostic to be issued when the lambda capture variable is outside of the lambda's {} scope. Another tests uses the lambda capture variable in a decltype-style return type which is also outside of the scope.

Do these failures sound like what others have run into?

Hi, I would just like to express that I am also seeing these two issues on Power, that are a result of the original patch (D119136) that @alanphipps has pointed out.

Sorry for the delayed reply.

Sorry but I've reverted this patch and all its fixups in c79e6007edef4b0044be93c4ffff64dc806ac687 and 0f5dbfd29ae0df215a01aff80d29255bb799fed0 .
See https://reviews.llvm.org/D123909#3461716 for another case not considered.

Please double check with the patch author and reviewers before unilaterally reverting multiple commits with no notice and no failing build bots. I do not see a valid justification for these reverts -- the concerns have been true positives so far (or have generated core issues that WG21 is still discussing), and @cor3ntin has been highly responsive with addressing the fallout. This is a significant amount of churn that I don't think should have happened.

Reply to both this message and https://reviews.llvm.org/D123909#3462560:

I am sorry but this thread has gained reports from multiple independent parties about breakage, and I just thought the followups did not fix all issues.

AFAIK, all issues have either been addressed, are true positives, or are a matter for WG21 to figure out what to do with. However, I'm not certain it was clear from the reviews that this was the case, so it's understandable there's some amount of "did we get everything?" confusion.

Yes, as far as I'm aware all the implementation issues with this patch has been addressed (or are undiscovered), so the remaining issues are that a C++ proposal (approved as a Defect Report in C++11 and up) do break existing code.
The issue has been raised in the C++ committee and clang implements https://github.com/cplusplus/CWG/issues/31 to mitigate the issue.
You will note that the resolution addresses decltype(id) but not decltype(*id) - there is no reasonable way to make the later work in a coherent fashion.

I think it would be tremendously helpful if you could share more of the reports you had internally, as it might affect how the C++ committee decides to resolves this.

In the meantime, clang could make that pattern well formed in older language modes, it something we have discussed, however I'm not sure you want to rely on code that will be made ill-formed when switching to C++23.
And the fact that decltype(*id) would have a different meaning in different part of the lambda expression ([id] (decltype(*id)) ->decltype(*id) may be 2 different types) makes me very uncomfortable - it's the reason the C++ committee decided to make that ill-formed to begin with.

It was the intent of WG21 for this change to be breaking. But as Aaron said, Clang is the first implementation to support it, so we are discovering unforeseen issues with it.
I think we should quantify the extent of the problem before taking drastic actions, as that will help both WG21 and Clang decide what are the most effective mitigation strategy, if any.

Thanks for the apology, but I still think the reverts were premature, so hopefully we don't do this again. The issue with reverting all of these is that you put the burden back onto a relatively new contributor to try to figure out how to reapply all these changes when there wasn't actually a need to do so.

In terms of where we go from here, there's a few options:

  1. revert the reverts; this is easiest, but causes the most churn
  2. merge all three patches into one (squashed) patch and commit that; this requires a little bit more work, but causes less churn
  3. assume that P2036R3 is not implementable until WG21 resolves CWG2569 because it breaks too much code

I think I have a preference for #2 as that will hopefully leave us with the most clear git history for when we do a blame later, but I could live with #1. If we find that there is significant broken code still, but it's not in system headers, then I think we should explore #3, but I don't think we have evidence that we're in this situation either.

I hope we don't fix bugs in production -- WG21 bug is also a bug.

  1. The decision made in the paper p2036r3 section 4.1 brings in a breaking change.
  2. The paper is applied as a DR on all previous C++ versions that apply.
  3. The paper has no Annex C entry.

These three things do not compose. The paper didn't do enough homework.

I would like to see One patch that allows other parties to try, but not necessarily in the trunk, since doing so couples the trial with other useful work in the trunk.

We've also been seeing failures in our downstream Arm compiler when running the Perennial C++14 compliance tests related to this change. Specifically, the tests expect a diagnostic to be issued when the lambda capture variable is outside of the lambda's {} scope. Another tests uses the lambda capture variable in a decltype-style return type which is also outside of the scope.

Do these failures sound like what others have run into?

Hi, I would just like to express that I am also seeing these two issues on Power, that are a result of the original patch (D119136) that @alanphipps has pointed out.

Do you have a code example, just to make sure we are talking about the same thing?

[i = 0]() -> decltype(i) {}

For example now is supposed to be valid with this change. I'm not sure how conformance test suits handle back-ported language changes and how long it takes them to be updated.

MaskRay added a comment.EditedApr 20 2022, 12:20 PM

Thanks for being willing to wait a bit and collect usage. Given current feedback, I am not going to simply undo the reverts.

@cor3ntin To get one patch other folks can try:

git revert -n c79e6007edef4b0044be93c4ffff64dc806ac687
git revert -n 0f5dbfd29ae0df215a01aff80d29255bb799fed0
git commit   # with a description incorporating this revision and all the following-ups. Make sure `Differential Revision: https://reviews.llvm.org/D119136` is included

On this webpage, click "Add Action" -> "Reopen Revision".
Upload a new diff either via web UI or arc diff 'HEAD^'.

Other folks can use curl -L 'https://reviews.llvm.org/D119136?download=1' | patch -p1 to apply the patch locally, or use arc patch D119136.

@rsmith @aaron.ballman Do you think it's worth resubmitting this patch with a look ahead of the mutable keyword (this seems to me a better strategy than other flimsy workaround suggested/tried), or would you rather wait for WG21 to come up with something?

When compile the following valid testcase:

void foo()
{
        int x = [x](int y[sizeof x]){return sizeof x;}(0);
}

It will complain:

error: captured variable 'x' cannot appear here
        int x = [x](int y[sizeof x]){return sizeof x;}(0);
                                 ^

The issue is also described in: https://cplusplus.github.io/CWG/issues/2569.html

@rsmith @aaron.ballman Do you think it's worth resubmitting this patch with a look ahead of the mutable keyword (this seems to me a better strategy than other flimsy workaround suggested/tried), or would you rather wait for WG21 to come up with something?

In principle, I think it would be reasonable to put this change behind a language option with a -f flag, and enable it by default only in C++23 mode, while WG21 sorts out what they want to do. But this patch is making some substantial changes in terms of how much code is moved around -- do you think it's feasible to support both modes without a lot of additional complexity?

@rsmith @aaron.ballman Do you think it's worth resubmitting this patch with a look ahead of the mutable keyword (this seems to me a better strategy than other flimsy workaround suggested/tried), or would you rather wait for WG21 to come up with something?

To directly answer your question: I'd want a reasonably strong signal from WG21 that that's the direction they want to pursue before we implement something like that, especially because it changes the meaning of some constructs while leaving them valid, which will create some churn for people living at head especially if we then roll that change back again. (I think we did have a strong enough signal to early-adopt the decltype(id) handling.)

@rsmith @aaron.ballman Do you think it's worth resubmitting this patch with a look ahead of the mutable keyword (this seems to me a better strategy than other flimsy workaround suggested/tried), or would you rather wait for WG21 to come up with something?

To directly answer your question: I'd want a reasonably strong signal from WG21 that that's the direction they want to pursue before we implement something like that, especially because it changes the meaning of some constructs while leaving them valid, which will create some churn for people living at head especially if we then roll that change back again. (I think we did have a strong enough signal to early-adopt the decltype(id) handling.)

I'll wait then, there are enough subtlety as it is, having different behavior in different language modes is not something i can commit time to at the moment (and one of the reason I was hoping we would still get this paper as a DR).
It's certainly doable - in particular getting everything before the end of the parameter declaration to behave as currently is trivial, but getting lookup in trailing return type etc to behave differently in different language mode would require some massaging.
I would also be concerned that putting that behind a flag would not give us meaningful data, so i don't know how useful it would be.

I was hoping to have some kind of implementation experience with look ahead as it's the most palatable fix, in my opinion, but it doesn't seem to be the one likely to be considered.
So I might implement it so that we can show that it's implementable, but no reason to attempt to upstream it - except that we won't get usage experience, nor will we know the extent of the breakage with CWG2569 applied

When compile the following valid testcase:

void foo()
{
        int x = [x](int y[sizeof x]){return sizeof x;}(0);
}

It will complain:

error: captured variable 'x' cannot appear here
        int x = [x](int y[sizeof x]){return sizeof x;}(0);
                                 ^

The issue is also described in: https://cplusplus.github.io/CWG/issues/2569.html

Is that something you encountered in existing code, or where you trying to write test against this change.
I only implemented the decltype part of the proposed resolution, for reason explained in that link (supporting sizeof(unqual-id) and `noexcept(unqual-id) seems extremely arbitrary).
But hopefully WG21 will come up with a more encompassing fix before we attempt to re-land this change

When compile the following valid testcase:

void foo()
{
        int x = [x](int y[sizeof x]){return sizeof x;}(0);
}

It will complain:

error: captured variable 'x' cannot appear here
        int x = [x](int y[sizeof x]){return sizeof x;}(0);
                                 ^

The issue is also described in: https://cplusplus.github.io/CWG/issues/2569.html

Is that something you encountered in existing code, or where you trying to write test against this change.
I only implemented the decltype part of the proposed resolution, for reason explained in that link (supporting sizeof(unqual-id) and `noexcept(unqual-id) seems extremely arbitrary).
But hopefully WG21 will come up with a more encompassing fix before we attempt to re-land this change

It is reduced from the existing code.