This is an archive of the discontinued LLVM Phabricator instance.

[Clang] Implement Change scope of lambda trailing-return-type
ClosedPublic

Authored by cor3ntin on Apr 24 2022, 2:16 PM.

Details

Summary

This implements P2036R3 and P2579R0.
That is, explicit, int, and implicit capture become visible
at the start of the parameter head.

Diff Detail

Event Timeline

cor3ntin created this revision.Apr 24 2022, 2:16 PM
Herald added a project: Restricted Project. · View Herald TranscriptApr 24 2022, 2:16 PM
cor3ntin requested review of this revision.Apr 24 2022, 2:16 PM
Herald added a project: Restricted Project. · View Herald Transcript
cor3ntin retitled this revision from [Clang][WIP] Implement Change scope of lambda trailing-return-type to [Clang][WIP] Implement Change scope of lambda trailing-return-type - Take 2.Apr 24 2022, 2:18 PM
cor3ntin edited reviewers, added: rsmith, aaron.ballman, erichkeane, MaskRay; removed: jdoerfert.

Hey folks.
I do not intend this to be merge but if there is anyway this can be battle tested against internal codebases, i think it could be helpful to help guide how WG21 proceed with this feature.
Here is a paper listing the failures encountered when we merged the previous attempt, and offers a more holistic mitigation - by looking a head for the mutable keyword https://isocpp.org/files/papers/D2579R0.pdf

Sorry again for the troubles the previous attempt caused but I'm glad we caught this problem sooner rather than later...

note to self.
@aaron.ballman noticed this patch doesn't handle leading attributes correctly

constexpr int asp = 5;
[=] [[gnu::regparm(asp)]] () {};

Now that Kona is over, I'm hoping to get back to that in the coming weeks. Rebasing will be... fun.
I want to make sure we are all okay making that a DR following WG21 guidance, given that not making it a DR would have potentially large impact on the PR.
Thanks!

aaron.ballman added a reviewer: Restricted Project.Nov 21 2022, 5:08 AM

Now that Kona is over, I'm hoping to get back to that in the coming weeks. Rebasing will be... fun.
I want to make sure we are all okay making that a DR following WG21 guidance, given that not making it a DR would have potentially large impact on the PR.
Thanks!

To me, it depends on how much existing code the DR breaks in practice. If we don't expect it to break a significant body of code (or if the code that breaks will be materially improved as a result of the required changes), then we should implement it as a DR as far back as we can go. If it breaks too much code in practice, then we might not want to implement it as a DR in older language modes. If that situation comes up and it's very hard for us to carry both implementations, then we should go back to WG21 with the further information before the DIS ballot goes out (if possible).

I'm adding clang-vendors to the review group because of the potential for disruption.

Now that Kona is over, I'm hoping to get back to that in the coming weeks. Rebasing will be... fun.
I want to make sure we are all okay making that a DR following WG21 guidance, given that not making it a DR would have potentially large impact on the PR.
Thanks!

To me, it depends on how much existing code the DR breaks in practice. If we don't expect it to break a significant body of code (or if the code that breaks will be materially improved as a result of the required changes), then we should implement it as a DR as far back as we can go.

We expect very little, if any, breakage

If it breaks too much code in practice, then we might not want to implement it as a DR in older language modes. If that situation comes up and it's very hard for us to carry both implementations, then we should go back to WG21 with the further information before the DIS ballot goes out (if possible).

Agreed, but time is short

I'm adding clang-vendors to the review group because of the potential for disruption.

Now that Kona is over, I'm hoping to get back to that in the coming weeks. Rebasing will be... fun.
I want to make sure we are all okay making that a DR following WG21 guidance, given that not making it a DR would have potentially large impact on the PR.
Thanks!

To me, it depends on how much existing code the DR breaks in practice. If we don't expect it to break a significant body of code (or if the code that breaks will be materially improved as a result of the required changes), then we should implement it as a DR as far back as we can go.

We expect very little, if any, breakage

Then I think we should approach it as a DR and we can react if we get further feedback.

cor3ntin updated this revision to Diff 477938.EditedNov 25 2022, 6:13 AM

Rebase. This is still a bit rough, I have a few tests to fix
and some dead code to remove.


@erichkeane I wouldn't mind picking your brain on this.
Consider:

template <typename T>
void dependent_init_capture(T x = 0) {
  [y = x] () requires (decltype(y){1}) {
    return y;
  }();
}

void test_dependent() {
  dependent_init_capture(0);
}

This used to ""work"" in the previous iteration of the patch,
but with the late concept checking it seems more involved.

My understanding so far is that in CheckFunctionConstraints
we can't find an instanciated declaration of y (so it asserts in LocalInstantiationScope::findInstantiationOf).

I'm not 100% sure of what's the correct approach here would be.
My guess is that we might need to retransform the captures in the instanciation scope used
for constraint checking?
I'm not actually sure whether an instanciation of the lambda does exist at that point in some other scope,
though. Ideally we would not instannciate lambdas more times than needed.

Another issue is that there is no way to walk back from a call expression to a lambda expression,
so we might need to add that.

@erichkeane Let me know if you have any insights on that constraint checking issue.
I might add a way to get from the CXXRecordDecl to the LambdaExpr and add the captures to the instantiation scope during constraint checking that way, but I'd love your opinion before going that direction. Thanks!

Rebase. This is still a bit rough, I have a few tests to fix
and some dead code to remove.


@erichkeane I wouldn't mind picking your brain on this.
Consider:

template <typename T>
void dependent_init_capture(T x = 0) {
  [y = x] () requires (decltype(y){1}) {
    return y;
  }();
}

void test_dependent() {
  dependent_init_capture(0);
}

This used to ""work"" in the previous iteration of the patch,
but with the late concept checking it seems more involved.

My understanding so far is that in CheckFunctionConstraints
we can't find an instanciated declaration of y (so it asserts in LocalInstantiationScope::findInstantiationOf).

I'm not 100% sure of what's the correct approach here would be.
My guess is that we might need to retransform the captures in the instanciation scope used
for constraint checking?
I'm not actually sure whether an instanciation of the lambda does exist at that point in some other scope,
though. Ideally we would not instannciate lambdas more times than needed.

Another issue is that there is no way to walk back from a call expression to a lambda expression,
so we might need to add that.

We shouldnn't -re-transform' the capture, but we need to make sure we get it into the current scope. I haven't looked at lambda instantiation much, so I don't know where that 'y' usually ends up? Can you do a run of it when it WORKS and see where it finds that when calling FindInstantiatedDecl. I'm guessing we are currently counting on that coming from a PARENT of the CurrentInstantiationScope (one of the Outers), which we can no longer count on.

FindInstantiatedDecl needs some unfortunate amount of work around lambdas, I found similar issues when working on https://reviews.llvm.org/D138148 which I need to get back to when I re-start work next year.

I came across a strange error when capturing arguments in a lambda inside another lambda. I filed an issue here - https://github.com/llvm/llvm-project/issues/59549

Short reproducer:

void foo () {
  constexpr int i = 2;

  [&]() {
    [=]() [[clang::annotate_type("test", i)]]{};
  };
}

<source>:5:42: error: variable 'i' cannot be implicitly captured in a lambda with no capture-default specified
    [=]() [[clang::annotate_type("test", i)]]{};
                                         ^
<source>:2:17: note: 'i' declared here
  constexpr int i = 2;

I noticed the error is not thrown if 'i' is captured in lambda body instead. IIUC all changes pertaining to P2036R3 has been reverted from clang right? So this bug is an existing issue in Clang with how attributes on lambdas are handled? Will this case be fixed in this PR?

I came across a strange error when capturing arguments in a lambda inside another lambda. I filed an issue here - https://github.com/llvm/llvm-project/issues/59549

Short reproducer:

void foo () {
  constexpr int i = 2;

  [&]() {
    [=]() [[clang::annotate_type("test", i)]]{};
  };
}

<source>:5:42: error: variable 'i' cannot be implicitly captured in a lambda with no capture-default specified
    [=]() [[clang::annotate_type("test", i)]]{};
                                         ^
<source>:2:17: note: 'i' declared here
  constexpr int i = 2;

I noticed the error is not thrown if 'i' is captured in lambda body instead. IIUC all changes pertaining to P2036R3 has been reverted from clang right? So this bug is an existing issue in Clang with how attributes on lambdas are handled? Will this case be fixed in this PR?

Yes, this does fix that, thanks for reporting.
Prior to this change, i would be looked for in the parent scope which explains why capturing it in the inner lambda solves the issue.

cor3ntin updated this revision to Diff 483757.Dec 17 2022, 9:32 AM

Fix fix-it tests

cor3ntin updated this revision to Diff 483812.Dec 18 2022, 5:54 AM

Handle late constraints checking and fully implement P2579.

cor3ntin updated this revision to Diff 483813.Dec 18 2022, 5:59 AM

Update release notes and commit message

cor3ntin retitled this revision from [Clang][WIP] Implement Change scope of lambda trailing-return-type - Take 2 to [Clang] Implement Change scope of lambda trailing-return-type.Dec 18 2022, 6:00 AM
cor3ntin edited the summary of this revision. (Show Details)
cor3ntin updated this revision to Diff 483816.Dec 18 2022, 6:18 AM

Remove unused variables

cor3ntin updated this revision to Diff 483817.Dec 18 2022, 6:25 AM

revert WS only change, removed unused DelayedCapture type

cor3ntin updated this revision to Diff 483820.Dec 18 2022, 6:54 AM

Remove the lookahead parsing code code which is no longer necessary

aaron.ballman added inline comments.Jan 9 2023, 11:00 AM
clang/include/clang/AST/DeclCXX.h
1802–1814

Minor simplification

clang/include/clang/Sema/Scope.h
144
clang/include/clang/Sema/ScopeInfo.h
868–870

Re-flow comment (I might have gotten that wrong) and add a full stop at the end of the comment.

clang/include/clang/Sema/Sema.h
6876–6880

You can re-flow this comment to 80 col as well.

6999
clang/lib/Parse/ParseExprCXX.cpp
1316

Typically, we call an ActOn method after having parsed the construct; in this case, we're calling ActOnLambdaIntroducer() when it was parsed elsewhere (this is the parsing code for after the introducer). So perhaps this should be moved elsewhere or renamed?

1389

This isn't being used?

1395–1401

The comment doesn't seem to match the code -- this isn't parsing into the late parsed attribute list?

1542–1544
1548–1550
clang/lib/Sema/SemaConcept.cpp
503–505 ↗(On Diff #483820)
508–509 ↗(On Diff #483820)
584–588 ↗(On Diff #483820)
615–618 ↗(On Diff #483820)
clang/lib/Sema/SemaLambda.cpp
441–444
975–976
975–976
976–977
983
986
1007

Spell out the type

1291–1370
1333
1343
clang/lib/Sema/TreeTransform.h
13142–13143
cor3ntin updated this revision to Diff 487539.Jan 9 2023, 1:08 PM
cor3ntin marked 24 inline comments as done.
  • Address Aaron's comments (except renaming ActOnLambdaIntroducer as we pounder on it)
  • Delete an aditional outdated comment
cor3ntin added inline comments.Jan 13 2023, 5:04 AM
clang/lib/Parse/ParseExprCXX.cpp
1316

ActOnLambdaExpressionAfterIntroducer?
We are actually acting on the lambda introducer, but it needs to be done in these scopes so it's why its there.

1389

It used to be! Thanks for catching that

1395–1401

More dead code!

aaron.ballman added inline comments.Jan 30 2023, 7:14 AM
clang/lib/Parse/ParseExprCXX.cpp
1316

Yeah, that's about the best name I can come up with as well. It's a bit messy because there's not really a clear separation of the steps of building up a lambda, but I think it's defensible.

aaron.ballman accepted this revision.Jan 30 2023, 7:16 AM

Aside from these minor nits and the naming concerns, I think this LGTM.

clang/docs/ReleaseNotes.rst
286–292

This change looks like it needs to be made on the Clang 16 branch as it was a drive-by change here.

288–291

This will probably need rebasing.

This revision is now accepted and ready to land.Jan 30 2023, 7:16 AM
cor3ntin updated this revision to Diff 493328.Jan 30 2023, 8:50 AM
  • rebase
  • rename ActOnLambdaIntroducer
  • Fix a crash in transformedLocalDecl
This revision was landed with ongoing or failed builds.Jan 31 2023, 2:06 AM
This revision was automatically updated to reflect the committed changes.

@aaron.ballman Thanks for the review! I hope we won't find further issue with the paper this time :)

nikic added a subscriber: nikic.Jan 31 2023, 8:12 AM

FYI this causes a minor compile-time regression (around 0.35% on 7zip at O0): http://llvm-compile-time-tracker.com/compare.php?from=cd173cbd7cca69c29df42cd4b42e60433435c29b&to=d708a186b6a9b050d09558163dd353d9f738c82d&stat=instructions%3Au

Just wanted to check whether that's expected.

FYI this causes a minor compile-time regression (around 0.35% on 7zip at O0): http://llvm-compile-time-tracker.com/compare.php?from=cd173cbd7cca69c29df42cd4b42e60433435c29b&to=d708a186b6a9b050d09558163dd353d9f738c82d&stat=instructions%3Au

Just wanted to check whether that's expected.

It's certainly not great but not entirely unexpected as we are doing quite a bit more work, although it should only affect code that constructs a lot of distinct lambda. Maybe an issue to track that would be useful, there may be some improvements opportunities
Thanks a lot for letting me know

FYI this causes a minor compile-time regression (around 0.35% on 7zip at O0): http://llvm-compile-time-tracker.com/compare.php?from=cd173cbd7cca69c29df42cd4b42e60433435c29b&to=d708a186b6a9b050d09558163dd353d9f738c82d&stat=instructions%3Au

Just wanted to check whether that's expected.

It's certainly not great but not entirely unexpected as we are doing quite a bit more work, although it should only affect code that constructs a lot of distinct lambda.

Does 7zip meet that criteria? As a gut-check that this only costs in that case, would be nice to know if the 7zip regression is that case, or is this the cost even when that isn't true - so we could expect higher costs when it is true?

FYI this causes a minor compile-time regression (around 0.35% on 7zip at O0): http://llvm-compile-time-tracker.com/compare.php?from=cd173cbd7cca69c29df42cd4b42e60433435c29b&to=d708a186b6a9b050d09558163dd353d9f738c82d&stat=instructions%3Au

Just wanted to check whether that's expected.

It's certainly not great but not entirely unexpected as we are doing quite a bit more work, although it should only affect code that constructs a lot of distinct lambda.

Does 7zip meet that criteria? As a gut-check that this only costs in that case, would be nice to know if the 7zip regression is that case, or is this the cost even when that isn't true - so we could expect higher costs when it is true?

I just downloaded their latest sources and didn't spot any lambdas in the C++ code with a simple grep. They also don't use auto so I suspect lambda usage is minimal. (It's possible I missed something, I didn't look exhaustively.)

FYI this causes a minor compile-time regression (around 0.35% on 7zip at O0): http://llvm-compile-time-tracker.com/compare.php?from=cd173cbd7cca69c29df42cd4b42e60433435c29b&to=d708a186b6a9b050d09558163dd353d9f738c82d&stat=instructions%3Au

Just wanted to check whether that's expected.

It's certainly not great but not entirely unexpected as we are doing quite a bit more work, although it should only affect code that constructs a lot of distinct lambda.

Does 7zip meet that criteria? As a gut-check that this only costs in that case, would be nice to know if the 7zip regression is that case, or is this the cost even when that isn't true - so we could expect higher costs when it is true?

I just downloaded their latest sources and didn't spot any lambdas in the C++ code with a simple grep. They also don't use auto so I suspect lambda usage is minimal. (It's possible I missed something, I didn't look exhaustively.)

Thanks. Then it's lookup related... I'll have to investigate. Coming up with a meaningful benchmark will be a challenge though.

Hi, me again :)

I ran into an interesting build breakage from this, I can't tell if it's a legitimate breakage based on reading P2036R3 and P2579R0 (mostly I'm not a language lawyer).

struct StringLiteral {
  template <int N>
  StringLiteral(const char (&array)[N])
      __attribute__((enable_if(N > 0 && N == __builtin_strlen(array) + 1,
                               "invalid string literal")));
};

struct Message {
  Message(StringLiteral);
};

void Func1() {
  auto x = Message("x");  // Note: this is fine

  // Note: "xx\0" to force a different type, StringLiteral<3>, otherwise this
  // successfully builds.
  auto y = [&](decltype(Message("xx"))) {};

  // ^ fails with: repro.cc:18:13: error: reference to local variable 'array'
  // declared in enclosing function 'StringLiteral::StringLiteral<3>'

  (void)x;
  (void)y;
}

https://godbolt.org/z/M4E6jKxxn

Does this look like an intended breakage from this patch?

cor3ntin added a comment.EditedFeb 2 2023, 11:32 PM

Hi, me again :)

I ran into an interesting build breakage from this, I can't tell if it's a legitimate breakage based on reading P2036R3 and P2579R0 (mostly I'm not a language lawyer).

struct StringLiteral {
  template <int N>
  StringLiteral(const char (&array)[N])
      __attribute__((enable_if(N > 0 && N == __builtin_strlen(array) + 1,
                               "invalid string literal")));
};

struct Message {
  Message(StringLiteral);
};

void Func1() {
  auto x = Message("x");  // Note: this is fine

  // Note: "xx\0" to force a different type, StringLiteral<3>, otherwise this
  // successfully builds.
  auto y = [&](decltype(Message("xx"))) {};

  // ^ fails with: repro.cc:18:13: error: reference to local variable 'array'
  // declared in enclosing function 'StringLiteral::StringLiteral<3>'

  (void)x;
  (void)y;
}

https://godbolt.org/z/M4E6jKxxn

Does this look like an intended breakage from this patch?

Looks like we fail to enter the appropriate context somewhere (my guess is that it might be specific to the attribute but it's hard to say without picking around), definitely a bug

I'll be away the next 2 weeks, I'll look at it when I get back. Maybe we should track it in an issue though.

Looks like we fail to enter the appropriate context somewhere (my guess is that it might be specific to the attribute but it's hard to say without picking around), definitely a bug

I'll be away the next 2 weeks, I'll look at it when I get back. Maybe we should track it in an issue though.

Should we revert this then?

Looks like we fail to enter the appropriate context somewhere (my guess is that it might be specific to the attribute but it's hard to say without picking around), definitely a bug

I'll be away the next 2 weeks, I'll look at it when I get back. Maybe we should track it in an issue though.

Should we revert this then?

Yes, I think we should. We should also file an issue so @cor3ntin doesn't lose track of the issue. Anyone want to volunteer to do that? (I can do it early next week otherwise.)

Looks like we fail to enter the appropriate context somewhere (my guess is that it might be specific to the attribute but it's hard to say without picking around), definitely a bug

I'll be away the next 2 weeks, I'll look at it when I get back. Maybe we should track it in an issue though.

Should we revert this then?

Yes, I think we should. We should also file an issue so @cor3ntin doesn't lose track of the issue. Anyone want to volunteer to do that? (I can do it early next week otherwise.)

Done in 74ce297045bac4bc475b8e762d2a1ea19bb16d3c and filed https://github.com/llvm/llvm-project/issues/60518 to track re-landing this.

Looks like we fail to enter the appropriate context somewhere (my guess is that it might be specific to the attribute but it's hard to say without picking around), definitely a bug

I'll be away the next 2 weeks, I'll look at it when I get back. Maybe we should track it in an issue though.

Should we revert this then?

Yes, I think we should. We should also file an issue so @cor3ntin doesn't lose track of the issue. Anyone want to volunteer to do that? (I can do it early next week otherwise.)

Done in 74ce297045bac4bc475b8e762d2a1ea19bb16d3c and filed https://github.com/llvm/llvm-project/issues/60518 to track re-landing this.

Thank you (and thank you for letting us know about the issue)!

cor3ntin reopened this revision.Feb 16 2023, 2:25 AM
This revision is now accepted and ready to land.Feb 16 2023, 2:25 AM
cor3ntin updated this revision to Diff 497937.Feb 16 2023, 2:26 AM

Reopen/Rebase

cor3ntin updated this revision to Diff 497994.Feb 16 2023, 6:14 AM

Fix GH60518 by not trying to capture parameters declared in the current
context.

aaron.ballman added inline comments.Feb 22 2023, 10:25 AM
clang/lib/Sema/SemaExpr.cpp
18653

Formatting nit.

I have a hazy recollection that we sometimes have a weird declaration context for parameters where the parameter doesn't think it's associated with a function DC but instead with the translation unit DC. I seem to recall this having something to do with PCH/AST importing, but I can't seem to find any open issues about it. So this might be fine, but it may have some weird edge cases still.

clang/test/SemaCXX/lambda-expressions.cpp
675–676 ↗(On Diff #497994)
  1. This code already compiles today without issue (https://godbolt.org/z/q6hPxoWq9), so are you sure this is testing what needs to be tested?
  1. What about for non-GNU-style attributes ([[]] attributes that appertain to the function type)? e.g.,
template <int N>
void foo(const char (&array)[N]) [[clang::annotate_type("test", array)]];
cor3ntin added inline comments.Feb 22 2023, 11:18 AM
clang/lib/Sema/SemaExpr.cpp
18653

Yep, parameters are all created in the top level dc before being attached to the function. but by the time they get to be captured by a lambda they are attached to the function
(default parameters initializers can't capture)

clang/test/SemaCXX/lambda-expressions.cpp
675–676 ↗(On Diff #497994)

Yes, this is something that was broken by this patch.
enable_if did create odr uses (incorrectly). I can add a test for annotate_type but I don't think it would run into the same issue at all.

aaron.ballman added inline comments.Feb 22 2023, 12:10 PM
clang/lib/Sema/SemaExpr.cpp
18653

Phew, thanks for checking!

clang/test/SemaCXX/lambda-expressions.cpp
675–676 ↗(On Diff #497994)

So I'm not certain what's being tested then -- how are you validating that array is no longer ODR used?

In terms of annotate_type, that attribute accepts a string literal followed by a variable number of expression arguments, so I think it should have the same issues as a GNU-style attribute accepting an expression argument. I was mostly trying to wrap my head around whether [[]] and __attribute__ are expected to have different behaviors when written in that position.

cor3ntin added inline comments.Feb 22 2023, 12:28 PM
clang/test/SemaCXX/lambda-expressions.cpp
675–676 ↗(On Diff #497994)
cor3ntin updated this revision to Diff 499754.Feb 23 2023, 12:23 AM

Address Aaron's comments

aaron.ballman accepted this revision.Feb 23 2023, 6:22 AM

LGTM

clang/test/SemaCXX/lambda-expressions.cpp
675–676 ↗(On Diff #497994)

I talked to @cor3ntin off-list to get a better understanding of the test situation, and I realized the issue was that this code used to give a diagnostic that it shouldn't have, so this is demonstrating we accept the code rather than reject it. That makes a lot more sense now!

rupprecht accepted this revision as: rupprecht.Feb 23 2023, 5:02 PM

All the things that were broken before are no longer broken, so LGTM

shafik added a subscriber: shafik.Feb 23 2023, 6:11 PM
shafik added inline comments.
clang/lib/Parse/ParseExprCXX.cpp
1321–1322

Minor edit

1321–1322
1389

If only we had a diagnostic for that....

1526–1533

This if feels pretty messy maybe refactor into something like CheckHasSpecificers, maybe?

shafik added inline comments.Feb 23 2023, 6:23 PM
clang/lib/Sema/SemaConcept.cpp
537 ↗(On Diff #499754)

This is just duplicated across the if and else, please refactor it out. If this ever gets modified folks have to fix it in both places and that is easy to mess up.

shafik added inline comments.Feb 23 2023, 6:57 PM
clang/lib/Sema/SemaExpr.cpp
18581

We have a bunch of bugs that crash in the method and it would be good to rescreen those bugs after landing this to see if it fixes any of those.

clang/lib/Sema/SemaLambda.cpp
977

This has changed a bit so might be worth updating the wording here to reflect C++20.

989–991

minor fix

1019
1022

Same as comment above about updating wording to reflect C++20.

1244–1245
clang/lib/Sema/TreeTransform.h
12985
cor3ntin updated this revision to Diff 500117.Feb 24 2023, 2:02 AM
cor3ntin marked 18 inline comments as done.

Address Shafik's comments

@rupprecht Great to hear! Sorry again for the disruption
@shafik Thanks for the review!

clang/lib/Sema/SemaLambda.cpp
1022

I don't even think we should repeat that comment here.

@shafik Can you let me know if you are happy with the changes i made to address your feedback? Thanks!

shafik accepted this revision.Mar 1 2023, 11:02 AM

LGTM

We're seeing new clang crashes that bisect to this commit, with modules only.

I have it mostly-reduced and will post shortly, trying to see if I can simplify any further (since modules reproducers are a pain).

Meanwhile, the assert/stack in case it's already useful:

assertion failed at third_party/llvm/llvm-project/clang/lib/Sema/SemaTemplateInstantiate.cpp:4065 in llvm::PointerUnion<Decl *, LocalInstantiationScope::DeclArgumentPack *> *clang::LocalInstantiationScope::findInstantiationOf(const Decl *): isa<LabelDecl>(D) && "declaration not instantiated in this scope"
*** Check failure stack trace: ***
    @     0x5609e4f16f44  __assert_fail
    @     0x5609e1894234  clang::LocalInstantiationScope::findInstantiationOf()
    @     0x5609e18d073c  clang::Sema::FindInstantiatedDecl()
    @     0x5609e18a99d0  clang::TreeTransform<>::TransformLambdaExpr()
    @     0x5609e189dbee  (anonymous namespace)::TemplateInstantiator::TransformLambdaExpr()
    @     0x5609e1892442  clang::TreeTransform<>::TransformExprs()
    @     0x5609e189a71a  clang::TreeTransform<>::TransformCallExpr()
    @     0x5609e189097a  clang::TreeTransform<>::TransformStmt()
    @     0x5609e18afa54  clang::TreeTransform<>::TransformCompoundStmt()
    @     0x5609e1890902  clang::Sema::SubstStmt()
    @     0x5609e18e31df  clang::Sema::InstantiateFunctionDefinition()
    @     0x5609e18e5ed2  clang::Sema::PerformPendingInstantiations()
    @     0x5609e0fcad44  clang::Sema::ActOnEndOfTranslationUnitFragment()
    @     0x5609e0fcbb66  clang::Sema::ActOnEndOfTranslationUnit()
    @     0x5609e0d298e6  clang::Parser::ParseTopLevelDecl()
    @     0x5609e0d2388e  clang::ParseAST()
    @     0x5609e0a647c3  clang::FrontendAction::Execute()
    @     0x5609e09d81ad  clang::CompilerInstance::ExecuteAction()
    @     0x5609dfa05b08  clang::ExecuteCompilerInvocation()
    @     0x5609df9f99f1  cc1_main()
    @     0x5609df9f5d28  ExecuteCC1Tool()
    @     0x5609e0b868be  llvm::function_ref<>::callback_fn<>()
    @     0x5609e4d9ec35  llvm::CrashRecoveryContext::RunSafely()
    @     0x5609e0b86103  clang::driver::CC1Command::Execute()
    @     0x5609e0b44166  clang::driver::Compilation::ExecuteCommand()
    @     0x5609e0b4448f  clang::driver::Compilation::ExecuteJobs()
    @     0x5609e0b63e70  clang::driver::Driver::ExecuteCompilation()
    @     0x5609df9f4ee7  clang_main()
    @     0x5609df9f1bc4  main
    @     0x7fad2cda4633  __libc_start_main
    @     0x5609df9f1b2a  _start

We're seeing new clang crashes that bisect to this commit, with modules only.

I have it mostly-reduced and will post shortly, trying to see if I can simplify any further (since modules reproducers are a pain).

Meanwhile, the assert/stack in case it's already useful:

assertion failed at third_party/llvm/llvm-project/clang/lib/Sema/SemaTemplateInstantiate.cpp:4065 in llvm::PointerUnion<Decl *, LocalInstantiationScope::DeclArgumentPack *> *clang::LocalInstantiationScope::findInstantiationOf(const Decl *): isa<LabelDecl>(D) && "declaration not instantiated in this scope"
*** Check failure stack trace: ***
    @     0x5609e4f16f44  __assert_fail
    @     0x5609e1894234  clang::LocalInstantiationScope::findInstantiationOf()
    @     0x5609e18d073c  clang::Sema::FindInstantiatedDecl()
    @     0x5609e18a99d0  clang::TreeTransform<>::TransformLambdaExpr()
    @     0x5609e189dbee  (anonymous namespace)::TemplateInstantiator::TransformLambdaExpr()
    @     0x5609e1892442  clang::TreeTransform<>::TransformExprs()
    @     0x5609e189a71a  clang::TreeTransform<>::TransformCallExpr()
    @     0x5609e189097a  clang::TreeTransform<>::TransformStmt()
    @     0x5609e18afa54  clang::TreeTransform<>::TransformCompoundStmt()
    @     0x5609e1890902  clang::Sema::SubstStmt()
    @     0x5609e18e31df  clang::Sema::InstantiateFunctionDefinition()
    @     0x5609e18e5ed2  clang::Sema::PerformPendingInstantiations()
    @     0x5609e0fcad44  clang::Sema::ActOnEndOfTranslationUnitFragment()
    @     0x5609e0fcbb66  clang::Sema::ActOnEndOfTranslationUnit()
    @     0x5609e0d298e6  clang::Parser::ParseTopLevelDecl()
    @     0x5609e0d2388e  clang::ParseAST()
    @     0x5609e0a647c3  clang::FrontendAction::Execute()
    @     0x5609e09d81ad  clang::CompilerInstance::ExecuteAction()
    @     0x5609dfa05b08  clang::ExecuteCompilerInvocation()
    @     0x5609df9f99f1  cc1_main()
    @     0x5609df9f5d28  ExecuteCC1Tool()
    @     0x5609e0b868be  llvm::function_ref<>::callback_fn<>()
    @     0x5609e4d9ec35  llvm::CrashRecoveryContext::RunSafely()
    @     0x5609e0b86103  clang::driver::CC1Command::Execute()
    @     0x5609e0b44166  clang::driver::Compilation::ExecuteCommand()
    @     0x5609e0b4448f  clang::driver::Compilation::ExecuteJobs()
    @     0x5609e0b63e70  clang::driver::Driver::ExecuteCompilation()
    @     0x5609df9f4ee7  clang_main()
    @     0x5609df9f1bc4  main
    @     0x7fad2cda4633  __libc_start_main
    @     0x5609df9f1b2a  _start

Thanks for letting me know.
A reproduction will be very useful to track that down indeed, I'll look into it as soon as i hear back from you. The stack trace doesn't tell me much unfortunately, although it definitely appears related.

bgraur added a subscriber: bgraur.Mar 8 2023, 5:02 AM

Sorry for the delay, extracting the repro from the build system seems about as much work as minimizing it :-)

- running repro.sh hits an assert for me on the main.cpp compile.

(I suspect something much smaller is breaking the expected invariant, I just can't get a smaller crasher)

Sorry for the delay, extracting the repro from the build system seems about as much work as minimizing it :-)

- running repro.sh hits an assert for me on the main.cpp compile.

(I suspect something much smaller is breaking the expected invariant, I just can't get a smaller crasher)

@ChuanqiXu @erichkeane I started to look at this repro but I might need help, it seems extremely weird (I failed to reduce it much further), and I don't have a good grasp on how modules affect things

Sorry for the delay, extracting the repro from the build system seems about as much work as minimizing it :-)

- running repro.sh hits an assert for me on the main.cpp compile.

(I suspect something much smaller is breaking the expected invariant, I just can't get a smaller crasher)

@ChuanqiXu @erichkeane I started to look at this repro but I might need help, it seems extremely weird (I failed to reduce it much further), and I don't have a good grasp on how modules affect things

I'm trying to help out here as well, but in addition to @ChuanqiXu and @erichkeane, I wonder if @iains or @urnathan have ideas as well (they've both done modules work before).

aaron.ballman added a comment.EditedMar 10 2023, 8:26 AM

Please keep in mind that I don't have a lot of expertise in modules, but this sure seems like a modules bug from what I can tell, and it might even relate to work happening in D145737 (CC @rsmith). That patch does not seem to address this issue, but what I'm seeing when I try to debug this is a bit confusing to me:

In the reproducer from Sam, there are three lambdas:
create::Create<R>([] {}); in main.cpp, but this does not participate in AST merging because it's within the main source file
BuildWidget<T>([dummy](Widget<T>*) {}); in create.h
create::Create<float>([] {}); in instantiate_create.h

There are two modules involved (wrap_create and instantiate_create). I would expect the wrap_create module to contain one lambda (from including create.h) and the instantiate_create module to contain two lambdas (one from InstantiateCreate() and one from including create.h).

So I would expect ASTDeclReader::ReadCXXDefinitionData() to find three lambdas, but instead it finds *four*. The first lambda it reads is the one in InstantiateCreate(). The next three all have captures of dummy; the first two come from the instantiate_create module and the last one comes from the wrap_create module.

In the reproducer from Sam, there are three lambdas:
create::Create<R>([] {}); in main.cpp, but this does not participate in AST merging because it's within the main source file
BuildWidget<T>([dummy](Widget<T>*) {}); in create.h

There should be three of these in total: one from wrap_create.h's textual inclusion of create.h, one from instantiate_create.h's textual inclusion of create.h, and one from instantiate_create.h's instantiation of Create<float>. The first two should get merged together if they're both loaded. This runs into some issues, because the local dummy variable in the two different bodies of the Create template don't get merged. In rG4a1ccfe8a3cfd4569bc962a38b6117a9a2b8ad21 I attempted to fix that by tracking all the lists of captures from different merged copies of the lambdas, so we can find the right way to name (eg) dummy in the enclosing context. Possibly some more work is needed here, to make template instantiation look through that list when instantiating a reference to a declaration, in the case where we pick a lambda body from one module but pick the enclosing function's body from a different module.

The first lambda it reads is the one in InstantiateCreate(). The next three all have captures of dummy; the first two come from the instantiate_create module and the last one comes from the wrap_create module.

That sounds right to me. One of the ones from instantiate_create (the one taking Widget<T>*, not the one taking Widget<float>*) should be merged with the one from wrap_create.h.

clang/lib/Sema/SemaCXXScopeSpec.cpp
295–298

Will this also reject __super in a local class inside a lambda? That seems like something we should still permit.

cor3ntin added inline comments.Mar 11 2023, 8:16 AM
clang/lib/Sema/SemaCXXScopeSpec.cpp
295–298

https://godbolt.org/z/dEj3vG8nc
Seems to work fine -

In both cases we create a non capturing function scope so getCurLambda returns nullptr

It looks like that the Create template functions get merged expectedly. And the dummy variable shouldn't get merged since it is not in a mergeable primary context (function context is not a mergeable primary context) (see https://github.com/llvm/llvm-project/blob/23bd0e037b744d1f93bdfad59b7575017725a96c/clang/lib/Serialization/ASTReaderDecl.cpp#L3107-L3151).

When I remove the use of dummy in the captures list, https://github.com/llvm/llvm-project/blob/23bd0e037b744d1f93bdfad59b7575017725a96c/clang/lib/Sema/SemaTemplateInstantiateDecl.cpp#L6079 is not executed. No matter if the dummy variable is captured or not, https://github.com/llvm/llvm-project/blob/23bd0e037b744d1f93bdfad59b7575017725a96c/clang/lib/Sema/SemaTemplateInstantiate.cpp#L4069-L4088 will execute twice for the dummy variable.

This patch causes an assertion when the attribute argument is an integer constant expression - https://godbolt.org/z/osKx5ejMb and has resulted in test fails downstream since any attribute which uses VerifyIntegerConstantExpression now hits this assert if used with lambdas.

The assert hit is in getCurLambda() :

auto *CurLSI = dyn_cast<LambdaScopeInfo>(*I);
 if (CurLSI && CurLSI->Lambda && CurLSI->CallOperator &&
     !CurLSI->Lambda->Encloses(CurContext) && CurLSI->AfterParameterList) {
   // We have switched contexts due to template instantiation.
   assert(!CodeSynthesisContexts.empty()); <----- Hits this
   return nullptr;
 }

Prior to this patch, Lambda Scope Information was not populated till after semantic analysis of attributes. This meant CurLSI->Lambda returned nullptr and we never entered the if. This patch however populates LSI during semantic analysis of lambda expression after introducer, which means we now enter the if during semantic analysis of the attribute` (AfterParameterList will be true at this point since this assert is hit way past parsing the parameters. CurContext is foo. I don't know if CurContext is right/wrong without further debugging.)

For the godbolt test pasted above, neither before nor after this patch do we hit the code where CodeSynthesisContexts is populated. I wanted to see what code actually enters that block and so I tried deleting the code to see what tests fails. What is interesting is that nothing does :/ So I don't know if that means our tests are incomplete or if this code is not required.

This patch causes an assertion when the attribute argument is an integer constant expression - https://godbolt.org/z/osKx5ejMb and has resulted in test fails downstream since any attribute which uses VerifyIntegerConstantExpression now hits this assert if used with lambdas.

The assert hit is in getCurLambda() :

auto *CurLSI = dyn_cast<LambdaScopeInfo>(*I);
 if (CurLSI && CurLSI->Lambda && CurLSI->CallOperator &&
     !CurLSI->Lambda->Encloses(CurContext) && CurLSI->AfterParameterList) {
   // We have switched contexts due to template instantiation.
   assert(!CodeSynthesisContexts.empty()); <----- Hits this
   return nullptr;
 }

Prior to this patch, Lambda Scope Information was not populated till after semantic analysis of attributes. This meant CurLSI->Lambda returned nullptr and we never entered the if. This patch however populates LSI during semantic analysis of lambda expression after introducer, which means we now enter the if during semantic analysis of the attribute` (AfterParameterList will be true at this point since this assert is hit way past parsing the parameters. CurContext is foo. I don't know if CurContext is right/wrong without further debugging.)

For the godbolt test pasted above, neither before nor after this patch do we hit the code where CodeSynthesisContexts is populated. I wanted to see what code actually enters that block and so I tried deleting the code to see what tests fails. What is interesting is that nothing does :/ So I don't know if that means our tests are incomplete or if this code is not required.

Thanks :)
As far i could tell, this check is only there to catch developer mistakes. We could probably hide it under #ifndef NDEBUG - seems preferable than to remove it entirely - and then make sure AfterParameterList is appropriately set to false when parsing GNU attributes. I might have time to look into it later today if you don't.

This patch causes an assertion when the attribute argument is an integer constant expression - https://godbolt.org/z/osKx5ejMb and has resulted in test fails downstream since any attribute which uses VerifyIntegerConstantExpression now hits this assert if used with lambdas.

The assert hit is in getCurLambda() :

auto *CurLSI = dyn_cast<LambdaScopeInfo>(*I);
 if (CurLSI && CurLSI->Lambda && CurLSI->CallOperator &&
     !CurLSI->Lambda->Encloses(CurContext) && CurLSI->AfterParameterList) {
   // We have switched contexts due to template instantiation.
   assert(!CodeSynthesisContexts.empty()); <----- Hits this
   return nullptr;
 }

Prior to this patch, Lambda Scope Information was not populated till after semantic analysis of attributes. This meant CurLSI->Lambda returned nullptr and we never entered the if. This patch however populates LSI during semantic analysis of lambda expression after introducer, which means we now enter the if during semantic analysis of the attribute` (AfterParameterList will be true at this point since this assert is hit way past parsing the parameters. CurContext is foo. I don't know if CurContext is right/wrong without further debugging.)

For the godbolt test pasted above, neither before nor after this patch do we hit the code where CodeSynthesisContexts is populated. I wanted to see what code actually enters that block and so I tried deleting the code to see what tests fails. What is interesting is that nothing does :/ So I don't know if that means our tests are incomplete or if this code is not required.

Thanks :)
then make sure AfterParameterList is appropriately set to false when parsing GNU attributes. I might have time to look into it later today if you don't.

I did look into AfterParameterList a bit yesterday. I don't fully understand this PR yet and so I could be wrong, but I think it is handled correctly at the moment. This assert is hit during semantic analysis of attributes (i.e. ProcessDeclAttributes in ActOnLambdaDefinition). Shouldn't AfterParameterList be true at this point (like it is now)?

By the way we see this crash with some of our clang attributes downstream as well.

As far i could tell, this check is only there to catch developer mistakes. We could probably hide it under #ifndef NDEBUG - seems preferable than to remove it entirely

Any attribute which takes the VerifyIntegerConstantExpression path fails. Since this assert is hit for valid code, I do think it needs to be fixed because valid code will fail in debug build even if we hide it under NDEBUG. If you can look into it today, that would be great!