This is an archive of the discontinued LLVM Phabricator instance.

[clang] Implement P2266 Simpler implicit move
ClosedPublic

Authored by mizvekov on Mar 19 2021, 8:24 PM.

Diff Detail

Event Timeline

mizvekov created this revision.Mar 19 2021, 8:24 PM
mizvekov requested review of this revision.Mar 19 2021, 8:24 PM
Herald added a project: Restricted Project. · View Herald TranscriptMar 19 2021, 8:24 PM
Herald added a subscriber: cfe-commits. · View Herald Transcript

For coroutines I have D68845. The current code is wrong, and not trivial to fix. We have never properly implemented P1825 and will probably just go with P2266 unconditionally, because that two-phase lookup is very cumbersome to imlement.

For coroutines I have D68845. The current code is wrong, and not trivial to fix. We have never properly implemented P1825 and will probably just go with P2266 unconditionally, because that two-phase lookup is very cumbersome to imlement.

Okay, once this is properly tested, we can spin a separate patch from this, just taking the coroutine bits unconditionally, and propose a new DR.

mizvekov updated this revision to Diff 332140.Mar 20 2021, 7:15 PM
  • Removed special flag, now is enabled on -std=c++2b
  • Always enabled for coroutines
  • Ran test suite
  • Some relevant test files updated to also run c++2b
  • Some new tests added
mizvekov edited the summary of this revision. (Show Details)Mar 20 2021, 7:19 PM

Conspicuously missing: tests for decltype(auto) return types, tests for co_return.

Personally I'd rather see these new p2266-related tests go in a separate test file, not appended to the existing file, so as to keep each test file relatively simpler and shorter. (But I don't know that those-in-charge would agree.)

clang/test/CXX/class/class.init/class.copy.elision/p3.cpp
4

These labels seem to be getting unwieldy. I propose that you rename cxx11_14_17 to cxx17 and cxx11_14_17_20 to cxx17_20.

(There are no cases where C++11, '14, '17's behavior differ.)

So then we'll have cxx17, cxx17_20, cxx20, cxx20_2b, and cxx2b.
IMHO it might even be nice to eliminate the overlapping labels; just, when a message is expected in two out of three modes, write two expectations.

B1(B1 &&) = delete;
  // cxx20-note@-1 {{'B1' has been explicitly marked deleted here}}
  // cxx2b-note@-2 {{'B1' has been explicitly marked deleted here}}

What do you think? (This could also be severed into its own earlier commit.)

332

FWIW here I would prefer

template MoveOnly& test_3a<MoveOnly&>(MoveOnly&);
template MoveOnly&& test_3a<MoveOnly>(MoveOnly&&);

with an expected error on the second line. The parameter-ness of w seems like a red herring in your version.

342

I believe this is incorrect; throw x here shouldn't implicit-move because x's scope exceeds the nearest enclosing try-block.

clang/test/CXX/special/class.copy/p33-0x.cpp
3 ↗(On Diff #332140)

Could you please ensure that this test is updated to be tested in all modes? My interpretation of the old line 1 is it's been locked to run only in '11 mode (not '14 or '17 or '20), which is just bizarre. There's nothing '11-specific about this file that I can see. (And again that change can be severed from the rest of this PR and landed earlier, I hope.)

41 ↗(On Diff #332140)

I believe this is incorrect; throw x here shouldn't implicit-move (not even in C++2b) because x's scope exceeds the nearest enclosing try-block.

With my previous comment I meant that it's better if you leave out the co_return bits for now because it's wrong anyway. We can't use PerformMoveOrCopyInitialization. It would just generate merge conflicts.

I'll probably just update my change once the committee made up its mind. I can rebase it on top of this if it needs to be dependent on C++20 vs C++2b.

Conspicuously missing: tests for decltype(auto) return types, tests for co_return.

Though the first comment warns you that this is a work in progress and I just didn't get around to doing it yet ;)

Though decltype I had tested manually, co_return I have just begun testing at all.

I believe the decltype tests should go into the other file relevant to the pertinent section. One problem with leaving them here is that it would be extra noise for the tests for the older standards which don't even support the keyword at all.

Personally I'd rather see these new p2266-related tests go in a separate test file, not appended to the existing file, so as to keep each test file relatively simpler and shorter. (But I don't know that those-in-charge would agree.)

They go into files named after section / paragraph, and I haven't come across examples where they were split yet, but I haven't looked specifically for this.

With my previous comment I meant that it's better if you leave out the co_return bits for now because it's wrong anyway. We can't use PerformMoveOrCopyInitialization. It would just generate merge conflicts.

I am using PerformMoveOrCopyInitialization with AllowNRVO always false for co_return, which effectively makes it a PerformCopyInitialization (just replacing them instead was one of the WIP cleanups I mentioned).
I am new with this part of the code here, so maybe this is obvious, but why is it broken? Do you have any examples of test cases where this would fail?
Would this patch here cause any regressions?
Right now this is passing the test suite, and seems to be reporting the correct results for the tests you created for D68845.
Remember, here we just want to evaluate the solution proposed for P2266 by testing this at large, in order to build support for it to be accepted into the standard.
Just waiting for it to be accepted first before implementing it would be too late :)

clang/test/CXX/class/class.init/class.copy.elision/p3.cpp
4

I like the idea of shortening the labels. Duplicating the warnings if it can be avoided easily, not so much. DRY is my mantra :D

Though what you said about severing this into an earlier commit, I think something may need to be done about it anyway.
So right now the tests are all over the place about which standard they are running, and none are testing c++2b at all...
I think we might need to gather input on this. Just changing every test to run every standard might be too much, so we could need a less obvious criteria.

332

Sure, sounds good.

342

Err, you are right, my bad, this is an easy fix...

clang/test/CXX/special/class.copy/p33-0x.cpp
3 ↗(On Diff #332140)

See my earlier comment about doing this systematically.

With my previous comment I meant that it's better if you leave out the co_return bits for now because it's wrong anyway. We can't use PerformMoveOrCopyInitialization. It would just generate merge conflicts.

Okay, I think I see now what you mean.
For example this:

struct task {                                                             
  struct promise_type {
    ...
    void return_value(T &&value) {}
  };                                                                      
};
task<NoCopyNoMove> local2val() {
  NoCopyNoMove value;
  co_return value;
 }

We should expect the test above to work, by binding value to the rvalue reference in task's promise, right?
Hmm, my natural course of action here would have been to figure out what is wrong and fix it.
I thought this would be good to have implemented before the committee decides on it.
@Quuxplusone thoughts? Is coroutine support optional here?

We should expect the test above to work, by binding value to the rvalue reference in task's promise, right?

Precisely, we don't want to do any initialization at all. (There will be an initialization for the parameters in BuildCallExpr.)

Implicit move just means that we're converting the lvalue to an xvalue.

On second thought, since @Quuxplusone's proposal makes things so simple you can just do this right yourself: instead of the initialization just do the cast, as D68845 does it. That's it.

Then let's see what we can get for C++20. If the proposal is accepted and gets DR status for co_return we might not need my change at all.

mizvekov updated this revision to Diff 332184.EditedMar 21 2021, 5:56 PM

Changed:

  • Implemented co_return as per Aaron's suggestion.
  • Imported coroutine tests from D68845.
  • Test tags shortened: cxxYY_cxxZZ means every version from YY to ZZ, instead of cxxAA_BB_CC_DD.
  • Existing test on decltype(auto) updated to also run on c++2b.
  • Change one of the test cases to just instantiate the relevant template, as per Arthur.

Still missing:

  • Making this run on all tests:
    • A lot of tests are running in only one old standard. Changing everything here would generate too much noise.
  • Some simplifications:
    • Aggregate common code for adding the implicit cast.
    • PerformMoveOrCopyInitialization can be changed to PerformCopyInitialization in the cases where AllowNRVO is false.
    • Can easily skip adding the implicit cast if expr was already a xvalue.
    • getCopyElisionCandidate needs refactoring IMHO. It is doing too much in one function, and this patch makes it worse. There are a lot of calls to it sprinkled around in the same sequence. Some problems:
      • On the current code, we want two things out of it: "Can we perform copy elision" AND "Can we perform implicit move". We end up calling it twice for that, with two parameters, instead of once returning both answers.
      • With this patch, now we need to call it even before we have the function return type available, then decide this type based on it, and then call it again to figure out if we can do copy elision. I will try to propose some way to improve that.
mizvekov updated this revision to Diff 332186.Mar 21 2021, 7:05 PM

small cleanup.

mizvekov updated this revision to Diff 332187.Mar 21 2021, 7:08 PM

small cleanup.

mizvekov edited the summary of this revision. (Show Details)Mar 22 2021, 10:28 AM

Splitting off the test changes into: https://reviews.llvm.org/D99225

mizvekov updated this revision to Diff 333158.Mar 24 2021, 3:43 PM

Rebased on top of D99225, running on a bunch of extra tests.

mizvekov edited the summary of this revision. (Show Details)Mar 24 2021, 3:45 PM
mizvekov marked 3 inline comments as done.

Data points: I've run this new Clang in -std=c++2b mode over my employer's (medium-sized, not too crazy) C++17 codebase, and also built Clang with Clang. Neither one required any source-code changes to deal with p2266. (The C++17 codebase did need about 10 identical changes to fix up ambiguous operator== overloads for C++20.)

Could we get someone to check out the Bloomberg codebase?

mizvekov updated this revision to Diff 333707.Mar 27 2021, 8:58 PM

Cleaned up implementation, not WIP anymore, ready for review.

mizvekov edited the summary of this revision. (Show Details)Mar 27 2021, 9:06 PM
mizvekov retitled this revision from [clang] WIP: Implement P2266 to [clang] Implement P2266 Simpler implicit move.
mizvekov updated this revision to Diff 333745.Mar 28 2021, 4:19 PM

So it turns out there was a pre-existing bug where ObjC++ blocks
with dependent return type were not getting copy elision.

My latest refactoring accidentally fixed this in the return statement
sema action, but unfortunately the template instantiator still expects
only FunctionDecl and crashes otherwise.

I am not sure there is an easy way to fix this, as BlockDecl
doesn't even carry a type unlike FunctionDecl, so there is no easy way
to get the return type.

This might require some surgery to fix, so for now I am just
skipping it and putting a FIXME.

mizvekov added inline comments.Mar 28 2021, 4:41 PM
clang/lib/Sema/SemaTemplateInstantiateDecl.cpp
1059

@rjmccall Hello! Do you have any tips for fixing this one?

mizvekov updated this revision to Diff 333749.Mar 28 2021, 4:57 PM

small fix

mizvekov updated this revision to Diff 333756.Mar 28 2021, 7:44 PM

some further simplifications

rjmccall added inline comments.Mar 28 2021, 11:37 PM
clang/lib/Sema/SemaTemplateInstantiateDecl.cpp
1059

I've always found it annoying that we only store the block type on the BlockExpr and not the BlockDecl, so if you've got a good reason to change that, I'm not opposed.

That said, while I'm not an expert in how we do NRVO variable tracking, the code must have some ability to handle an unknown return type when just processing declarations by delaying some of the bookkeeping/analysis until the end of the function. It's possible but quite uncommon to declare a return type explicitly in a block, so that's the dominant case here. I'm actually surprised we do any of this tracking when we're just instantiating a variable.

mizvekov updated this revision to Diff 333925.Mar 29 2021, 10:32 AM

Small update making some invariants more clear, with some simplifications.

mizvekov added inline comments.Mar 29 2021, 3:57 PM
clang/lib/Sema/SemaTemplateInstantiateDecl.cpp
1059

Thanks, your observations have been extremely helpful.

So this whole NRVO tracking thing looks to be in bad shape. This tracker here fails to handle 'auto' types. I don't think this gets fixed up anywhere else.

This is also badly covered by the test suite. The test included with the merge request that introduced this tracking (CodeGenCXX/nrvo.cpp) still passes if you either delete this whole block here, or if you just do Var->setNRVOVariable(D->isNRVOVariable()).

In fact, almost the whole test suite passes with those changes, the only tests which fail are CodeGenCXX/matrix-type-builtins.cpp and CodeGenCXX/matrix-type-operators.cpp respectively.

rjmccall added a subscriber: fhahn.Mar 29 2021, 6:54 PM
rjmccall added inline comments.
clang/lib/Sema/SemaTemplateInstantiateDecl.cpp
1059

I'm surprised that this has any impact on matrix types; pinging @fhahn in case he understands what the interaction might be.

It looks like we're assuming that the parse-time analysis of NRVO candidates is correct except for possibly needing adjustment for dependent expressions / return types. The parse-time analysis seems to work by keeping track of NRVO candidates on Scope; since we don't push meaningful Scopes during template instantiation, that won't work, so we have to trust the parse-time analysis. But the parse-time analysis can also be overly conservative, mostly because of if constexpr; that might be enough of a corner case to not merit something better, though. @rsmith is the expert here, though.

At any rate, it should be *correct* to handle non-function cases as if the return type were dependent, the same way it presumably works for functions with deduced return types.

mizvekov added inline comments.Mar 30 2021, 3:14 PM
clang/lib/Sema/SemaTemplateInstantiateDecl.cpp
1059

The main thing here is that the NRVO criteria for function return type is ignored if the type is dependent.
But it is not ignored for a non-dependent auto type, "because auto is not a record type", so this evaluates as not a NRVO candidate.

It is strange that we make this distinction between dependent and non-dependent auto here.

This is actually the only case where this function is called where the FnRetType is possibly auto, so it's an understandable oversight if that is the case.

I see we don't have the type deduced here even for functions with a single return statement and no if constexpr or other indirection, where it seems it would have been possible to have just deduced it to some dependent type during parsing.

That said, my (pretty uninformed) guess here is that it should be safe to treat auto the same as dependent, and it is probably the most reasonable choice.
That we are (again my guess) just trying here to weed out some candidates to get better results in some simple greedy algorithm later.

mizvekov updated this revision to Diff 335542.Apr 6 2021, 9:10 AM
mizvekov updated this revision to Diff 335604.Apr 6 2021, 11:12 AM
  • test/SemaCXX/coroutine-rvo: Make MoveOnly trivial, since the temporary is not created anymore and there is no special reason to test a non-trivial type here.
mizvekov updated this revision to Diff 351274.Jun 10 2021, 2:50 PM

rerun pipeline.

Quuxplusone accepted this revision.Jun 10 2021, 2:57 PM

IMHO we should just land this already. (It's been sitting without major update since mid-April.) It is intended to affect only -std=c++2b mode, and implements a paper (full disclosure: my paper) targeting C++2b that EWG wants implementation experience with. So I have proposed to @mizvekov that he should land this on Monday, unless someone yells "stop" before then.
If this breaks anyone compiling in C++2b mode, then (1) it can always be reverted, and (2) that would be exactly the successful gathering of field experience that we're hoping for.

This revision is now accepted and ready to land.Jun 10 2021, 2:57 PM

Actually we can land it now. @rsmith gave me the go ahead in private. The windows pipeline passes. The latest debian build failure is some random fortran thing, so we are good.

This revision was landed with ongoing or failed builds.Jun 10 2021, 3:56 PM
This revision was automatically updated to reflect the committed changes.
mizvekov reopened this revision.Jun 11 2021, 8:50 AM
This revision is now accepted and ready to land.Jun 11 2021, 8:50 AM
This revision was automatically updated to reflect the committed changes.
mizvekov reopened this revision.Jun 14 2021, 2:03 PM
This revision is now accepted and ready to land.Jun 14 2021, 2:03 PM
sberg added a subscriber: sberg.Jun 14 2021, 11:20 PM

(In a build prior to https://reviews.llvm.org/rGc60dd3b2626a4d9eefd9f82f9a406b0d28d3fd72 "Revert '[clang] NRVO: Improvements and handling of more cases.'") I see the following (reduced from https://git.libreoffice.org/core/+/649313625b94e6b879848fc19b607b74375100bf/o3tl/qa/compile-temporary.cxx) started to fail under -std=c++2b with this change (and continues to compile fine with -std=c++20):

$ cat test.cc
template <typename T> constexpr T& temporary(T&& x) { return x; }
template <typename T> constexpr T& temporary(T&) = delete;
void f(int*);
int g();
void h()
{
    f(&temporary(int()));
    f(&temporary(g()));
}

$ clang++ -std=c++2b -fsyntax-only test.cc
test.cc:1:62: error: non-const lvalue reference to type 'int' cannot bind to a temporary of type 'int'
template <typename T> constexpr T& temporary(T&& x) { return x; }
                                                             ^
test.cc:7:8: note: in instantiation of function template specialization 'temporary<int>' requested here
    f(&temporary(int()));
       ^
test.cc:8:8: error: no matching function for call to 'temporary'
    f(&temporary(g()));
       ^~~~~~~~~
test.cc:2:36: note: candidate function [with T = int] not viable: expects an lvalue for 1st argument
template <typename T> constexpr T& temporary(T&) = delete;
                                   ^
test.cc:1:36: note: candidate template ignored: substitution failure [with T = int]
template <typename T> constexpr T& temporary(T&& x) { return x; }
                                   ^
2 errors generated.

It is not clear to me whether that is an intended change in behavior according to http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p2266r1.html "Simpler implicit move", or whether it is a bug in this implementation.

(In a build prior to https://reviews.llvm.org/rGc60dd3b2626a4d9eefd9f82f9a406b0d28d3fd72 "Revert '[clang] NRVO: Improvements and handling of more cases.'") I see the following (reduced from https://git.libreoffice.org/core/+/649313625b94e6b879848fc19b607b74375100bf/o3tl/qa/compile-temporary.cxx) started to fail under -std=c++2b with this change (and continues to compile fine with -std=c++20):
It is not clear to me whether that is an intended change in behavior according to http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p2266r1.html "Simpler implicit move", or whether it is a bug in this implementation.

Yeah that one is intentional, under P2266 as far as implicit move is concerned, that code behaves similarly to this code under C++20 (which is diagnosed for the same reason):

template <typename T> constexpr T& temporary(T&& x) { return static_cast<T&&>(x); }

If you want this semantics with P2266, you could write:

template <typename T> constexpr T& temporary(T&& x) { return static_cast<T&>(x); }

And the above also works on C++20.

Thank you for reporting this, it's the kind of feedback we needed about real world usage of code that would break under these new rules.

@sberg: Thanks for the example! @mizvekov's comments are correct, but I wanted to put some more comments here for posterity. (I'll also try to find a place for this real-world example in the next revision of p2266.)

The code in question is o3tl::temporary, documented as "Cast an rvalue to an lvalue." and implemented as template<class T> T& temporary(T&& x) { return x; }
https://docs.libreoffice.org/o3tl/html/temporary_8hxx_source.html
p2266 proposes to break this code in C++23, by making the expression x an xvalue. EWG was well aware that p2266 would break this exact kind of "laundering" function, which takes an rvalue and launders it into an lvalue without a cast. This breakage was generally (though I'm sure not universally) seen as a good thing. We weren't aware of any specific real-world examples of such "laundering" functions, though, so this is very useful info.
As Matheus says, the best way to fix the code is to add an explicit cast: return static_cast<T&>(x);.

sberg added a comment.Jun 17 2021, 1:28 AM

For the record, the one other breakage of a LibreOffice build with this is the handful of places that needed https://git.libreoffice.org/core/+/433ab39b2175bdadb4916373cd2dc8e1aabc08a5%5E%21 "Adapt implicit OString return value construction to C++23 P2266R1": In a nutshell, LO has an OString class with (non-explicit) constructors from any kind of char pointer or array, where the "array of N const char" variant (no. 3 below) is special, as it determines the string length as N - 1 (whereas all the other variants search for the terminating NUL to determine the length). That requires another constructor overload (no. 2 below) for non-const arrays, taking a non-const lvalue reference, which now causes issues when that constructor shall implicitly be used in return statements:

$ cat test.cc
#include <cstddef>

template<typename> struct CharPtrDetector {};
template<> struct CharPtrDetector<char *>
{ using type = int; };
template<> struct CharPtrDetector<char const *>
{ using type = int; };

template<typename> struct NonConstCharArrayDetector {};
template<std::size_t N> struct NonConstCharArrayDetector<char [N]>
{ using type = int; };

template<typename> struct ConstCharArrayDetector {};
template<std::size_t N> struct ConstCharArrayDetector<char const [N]>
{ using type = int; };

struct OString {
    template<typename T>
    OString(T const &, typename CharPtrDetector<T>::type = 0); // #1
    template<typename T>
    OString(T &, typename NonConstCharArrayDetector<T>::type = 0); // #2
    template<typename T>
    OString(T &, typename ConstCharArrayDetector<T>::type = 0); // #3
};

OString good1a() {
    char * p;
    //...
    return p; // use #1
};

OString good1b() {
    char const * p;
    //...
    return p; // use #1
};

OString bad2() {
    char buf[10];
    //...
    return buf; // use #2
}

OString good3() {
    return "foo"; // use #3
}

$ clang++ -std=c++20 -fsyntax-only test.cc

$ clang++ -std=c++2b -fsyntax-only test.cc
test.cc:41:12: error: no viable conversion from returned value of type 'char [10]' to function return type 'OString'
    return buf; // use #2
           ^~~
test.cc:17:8: note: candidate constructor (the implicit copy constructor) not viable: no known conversion from 'char [10]' to 'const OString &' for 1st argument
struct OString {
       ^
test.cc:17:8: note: candidate constructor (the implicit move constructor) not viable: no known conversion from 'char [10]' to 'OString &&' for 1st argument
struct OString {
       ^
test.cc:21:5: note: candidate constructor [with T = char [10]] not viable: expects an lvalue for 1st argument
    OString(T &, typename NonConstCharArrayDetector<T>::type = 0); // #2
    ^
test.cc:19:5: note: candidate template ignored: substitution failure [with T = char [10]]: no type named 'type' in 'CharPtrDetector<char [10]>'
    OString(T const &, typename CharPtrDetector<T>::type = 0); // #1
    ^                                               ~~~~
test.cc:23:5: note: candidate template ignored: substitution failure [with T = char [10]]: no type named 'type' in 'ConstCharArrayDetector<char [10]>'
    OString(T &, typename ConstCharArrayDetector<T>::type = 0); // #3
    ^                                                ~~~~
1 error generated.
mizvekov updated this revision to Diff 352818.Jun 17 2021, 12:12 PM
  • Change block capture NRVO semantics to match P2266 in C++2b mode.
  • Update cxx_status.
mizvekov updated this revision to Diff 352985.Jun 18 2021, 5:49 AM

fix ctidy warning.

This revision was landed with ongoing or failed builds.Jun 18 2021, 8:09 AM
This revision was automatically updated to reflect the committed changes.
sberg added a comment.Jun 25 2021, 2:12 AM

And, again for the record, with a build of LibreOffice with clang-cl (and -Xclang -std=c++2b) on Windows, at least against the C++ standard library from Visual Studio 2019 version 16.20.2, I ran into two issues in the standard library itself, when using std::getline and std::istream::operator>>:

In file included from C:/lo-clang/core/setup_native/source/win32/customactions/reg_dlls/reg_dlls.cxx:12:
C:/PROGRA~2/MIB055~1/2019/COMMUN~1/VC/Tools/MSVC/1429~1.300/Include\string(66,12): error: non-const lvalue reference to type 'basic_istream<...>' cannot bind to a temporary of type 'basic_istream<...>'
    return _Istr;
           ^~~~~
C:/PROGRA~2/MIB055~1/2019/COMMUN~1/VC/Tools/MSVC/1429~1.300/Include\string(78,12): note: in instantiation of function template specialization 'std::getline<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t>>' requested here
    return getline(_STD move(_Istr), _Str, _Delim);
           ^
C:/lo-clang/core/setup_native/source/win32/customactions/reg_dlls/reg_dlls.cxx(197,17): note: in instantiation of function template specialization 'std::getline<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t>>' requested here
    while (std::getline(ss, sToken, L'|'))
                ^
1 error generated.
make[1]: *** [C:/lo-clang/core/solenv/gbuild/LinkTarget.mk:298: C:/lo-clang/core/workdir/CxxObject/setup_native/source/win32/customactions/reg_dlls/reg_dlls.o] Error 1

c:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.29.30037/include/string

and

[build CXX] l10ntools/source/merge.cxx
In file included from C:/lo-clang/core/l10ntools/source/merge.cxx:21:
In file included from C:/lo-clang/core/include\sal/log.hxx:20:
In file included from C:/PROGRA~2/MIB055~1/2019/COMMUN~1/VC/Tools/MSVC/1429~1.300/Include\sstream:11:
In file included from C:/PROGRA~2/MIB055~1/2019/COMMUN~1/VC/Tools/MSVC/1429~1.300/Include\istream:11:
In file included from C:/PROGRA~2/MIB055~1/2019/COMMUN~1/VC/Tools/MSVC/1429~1.300/Include\ostream:11:
In file included from C:/PROGRA~2/MIB055~1/2019/COMMUN~1/VC/Tools/MSVC/1429~1.300/Include\ios:11:
In file included from C:/PROGRA~2/MIB055~1/2019/COMMUN~1/VC/Tools/MSVC/1429~1.300/Include\xlocnum:16:
In file included from C:/PROGRA~2/MIB055~1/2019/COMMUN~1/VC/Tools/MSVC/1429~1.300/Include\streambuf:11:
In file included from C:/PROGRA~2/MIB055~1/2019/COMMUN~1/VC/Tools/MSVC/1429~1.300/Include\xiosbase:12:
In file included from C:/PROGRA~2/MIB055~1/2019/COMMUN~1/VC/Tools/MSVC/1429~1.300/Include\system_error:14:
In file included from C:/PROGRA~2/MIB055~1/2019/COMMUN~1/VC/Tools/MSVC/1429~1.300/Include\stdexcept:12:
C:/PROGRA~2/MIB055~1/2019/COMMUN~1/VC/Tools/MSVC/1429~1.300/Include\xstring(4971,12): error: non-const lvalue reference to type 'basic_istream<...>' cannot bind to a temporary of type 'basic_istream<...>'
    return _Istr;
           ^~~~~
C:/PROGRA~2/MIB055~1/2019/COMMUN~1/VC/Tools/MSVC/1429~1.300/Include\xstring(4977,29): note: in instantiation of function template specialization 'std::operator>><char, std::char_traits<char>, std::allocator<char>>' requested here
    return _STD move(_Istr) >> _Str;
                            ^
C:/lo-clang/core/l10ntools/source/merge.cxx(125,18): note: in instantiation of function template specialization 'std::operator>><char, std::char_traits<char>, std::allocator<char>>' requested here
    aInputStream >> sPoFile;
                 ^
1 error generated.
make[1]: *** [C:/lo-clang/core/solenv/gbuild/LinkTarget.mk:301: C:/lo-clang/core/workdir/CxxObject/l10ntools/source/merge.o] Error 1

In both cases, the reason is that library happens to implement the overload taking basic_istream& by forwarding to the overload taking basic_istream&& (and not the other way around, as libc++ and libstdc++ happen to do), and the fix is to replace

return _Istr;

in the overloads taking basic_istream&& _Istr with

return static_cast<basic_istream<_Elem, _Traits>&>(_Istr);

(No idea if any MSVC standard library developers are around here, who might want to be CC'ed on this.)

Thank you again @sberg.

I have talked to the MSVC STL maintainers. They would be open to a pull request for fixing this.
The repo can be found at https://github.com/microsoft/STL
Be my guest if you want to submit that fix yourself.
If not, I can probably do it in the near future.

I have talked to the MSVC STL maintainers. They would be open to a pull request for fixing this.
The repo can be found at https://github.com/microsoft/STL
Be my guest if you want to submit that fix yourself.
If not, I can probably do it in the near future.

Thanks a lot for the link. Filed https://github.com/microsoft/STL/pull/2025 "Adapt some return statements to P2266R1 'Simpler implicit move'" now.

This change has made this snippet fail.
https://godbolt.org/z/3ehK784hY Pass
https://godbolt.org/z/9q48WvsP7 fails.

Hello!
That is expected breakage from the changes proposed in P2266.
With simpler implicit move, we no longer have a fallback second overload resolution where the return expression would be an l-value and would thus bind to that non-const copy constructor.
You can work that around by introducing an explicit cast to lvalue reference, like this: https://godbolt.org/z/EhYbhYjch

But is this example a reduction from a real world code base?
The committee wants feedback and we are interested how hard you believe this change affects you.

zahiraam added a comment.EditedJun 28 2021, 8:26 AM

This change has made this snippet fail.
https://godbolt.org/z/3ehK784hY Pass
https://godbolt.org/z/9q48WvsP7 fails.

Hello!
That is expected breakage from the changes proposed in P2266.
With simpler implicit move, we no longer have a fallback second overload resolution where the return expression would be an l-value and would thus bind to that non-const copy constructor.
You can work that around by introducing an explicit cast to lvalue reference, like this: https://godbolt.org/z/EhYbhYjch

But is this example a reduction from a real world code base?
The committee wants feedback and we are interested how hard you believe this change affects you.

That's a reduced test from a test suite.

But is this example a reduction from a real world code base?
The committee wants feedback and we are interested how hard you believe this change affects you.

This is an example that also appears to be impacted by this change:

struct C {
    C();
    C(C &c1);
};

void foo()
{
    try {
        C c1;
        throw c1;
    }
    catch (C c2) {
        throw;
    }
}

https://godbolt.org/z/dvEbv7GKo

I'm not certain if this is as expected of an issue, though. In the original example, C carried state that was set up after initialization but was relying on the fallback to the non-idiomatic copy constructor when doing the throw. WDYT?

Thank you again @sberg.

I have talked to the MSVC STL maintainers. They would be open to a pull request for fixing this.
The repo can be found at https://github.com/microsoft/STL
Be my guest if you want to submit that fix yourself.
If not, I can probably do it in the near future.

I'm not certain it's reasonable to wait for MSVC STL to change as that leaves every existing user of older MSVC STLs out in the cold. This is causing some significant regressions at Intel, to the extent that I wonder if this should be temporarily reverted until the MSVC STL headers can be compiled again.

https://godbolt.org/z/dvEbv7GKo

I'm not certain if this is as expected of an issue, though. In the original example, C carried state that was set up after initialization but was relying on the fallback to the non-idiomatic copy constructor when doing the throw. WDYT?

Yeah that is the equivalent scenario for throw, we treat c1 as a temporary there. The same workaround applies, you can static cast to non-const lvalue reference.

As far as the implementation is concerned, it is following the proposal to the letter here.
And as I understand it, although I am not the author of the proposal (which is @Quuxplusone), the committee agrees that this breakage is a good thing.

I'm not certain it's reasonable to wait for MSVC STL to change as that leaves every existing user of older MSVC STLs out in the cold. This is causing some significant regressions at Intel, to the extent that I wonder if this should be temporarily reverted until the MSVC STL headers can be compiled again.

That does interfere with the wishes of the committee to get implementation experience with this.
I am not saying one way or another, but would leaving this effect on by default, but providing a command line flag to turn it off, be a reasonable option on the table?

https://godbolt.org/z/dvEbv7GKo

I'm not certain if this is as expected of an issue, though. In the original example, C carried state that was set up after initialization but was relying on the fallback to the non-idiomatic copy constructor when doing the throw. WDYT?

Yeah that is the equivalent scenario for throw, we treat c1 as a temporary there. The same workaround applies, you can static cast to non-const lvalue reference.

Thanks for verifying!

As far as the implementation is concerned, it is following the proposal to the letter here.
And as I understand it, although I am not the author of the proposal (which is @Quuxplusone), the committee agrees that this breakage is a good thing.

I'm less convinced than the committee; I put a high value on existing, working, conforming code continuing to work in newer language modes. However, this is what the standard requires for that language mode, so I can live with it unless the code pattern shows up in some common system header that users can't change themselves, and I haven't run into evidence of that for this particular case yet.

I'm not certain it's reasonable to wait for MSVC STL to change as that leaves every existing user of older MSVC STLs out in the cold. This is causing some significant regressions at Intel, to the extent that I wonder if this should be temporarily reverted until the MSVC STL headers can be compiled again.

That does interfere with the wishes of the committee to get implementation experience with this.

I would argue it gives the committee valuable implementation experience feedback: the change breaks at least one popular system header.

I am not saying one way or another, but would leaving this effect on by default, but providing a command line flag to turn it off, be a reasonable option on the table?

I don't think it's reasonable to leave this on by default for MSVC compatibility; it's not like std::ifstream is an obscure part of the STL or there's only one version of the STL that's impacted. The suggestions I have for how to move forward are:

  • Add a compatibility hack that recognizes using the STL as a system header in a limited way to disable the diagnostics and get the old behavior (we do this on occasion for libstdc++, no reason we couldn't do something similar for MSVC STL). Then old headers still work without needing a flag and new (fixed) headers never see the issue. Someday in the distant future, we can remove the compatibility hack.
  • Or, add a command line flag that defaults off for -fms-compatibility and defaults on otherwise. The default can be switched based on the compatibility version being used (perhaps) once the STL has been fixed.

I'd prefer to keep existing code working without requiring users to enable a flag, if at all possible (esp because I don't expect the flag to be needed forever). That said, I still think it makes sense to temporarily revert this while doing that work.

I would argue it gives the committee valuable implementation experience feedback: the change breaks at least one popular system header.

But we already knew that :-)
Reverting the change globally would stop us from finding further affected entities.
But it is a given that because of this STL bug, this is affecting our ability to get that feedback on user code in that ecosystem as well.

I don't think it's reasonable to leave this on by default for MSVC compatibility; it's not like std::ifstream is an obscure part of the STL or there's only one version of the STL that's impacted. The suggestions I have for how to move forward are:

  • Add a compatibility hack that recognizes using the STL as a system header in a limited way to disable the diagnostics and get the old behavior (we do this on occasion for libstdc++, no reason we couldn't do something similar for MSVC STL). Then old headers still work without needing a flag and new (fixed) headers never see the issue. Someday in the distant future, we can remove the compatibility hack.
  • Or, add a command line flag that defaults off for -fms-compatibility and defaults on otherwise. The default can be switched based on the compatibility version being used (perhaps) once the STL has been fixed.

I find interesting the idea of combining these two options. We can suppress the effect when compiling with fms-compatibility but only on the STL headers themselves.

I'd prefer to keep existing code working without requiring users to enable a flag, if at all possible (esp because I don't expect the flag to be needed forever). That said, I still think it makes sense to temporarily revert this while doing that work.

I'd like to give some time for other stakeholders to give their opinion if this is not too urgent, specially @Quuxplusone and @rsmith.
But thinking of another option, almost as easy as a global revert for this, we could suppress the effect under -fms-compatibility for now, and then implement a more targetted fix where we only suppress this when parsing the STL system headers themselves under fms-compatibility.

I'd like to give some time for other stakeholders to give their opinion if this is not too urgent, specially @Quuxplusone and @rsmith.

I have no strong/well-informed opinions here. I have an idea to offer, if there is precedent for it. My idea is, keep it as an error if the rvalue resolution finds nothing, but (1) fall back to an lvalue resolution for error-recovery purposes, and (2) suppress the error in system headers. Then, the usage in MSVC STL will keep working but only because it's in a system header. I seem to recall that there is precedent for diagnostics that are "non-downgradable error in user code, yet still suppressed in system headers." If I'm imagining things, then please do not invent such a facility purely for this feature.

My second choice is to do whatever minimalistic hack suffices to satisfy the customer (Aaron etc. in this case), but only with the understanding that we plan to eliminate that hack as soon as MSVC STL gets updated. My third choice is to revert the whole thing and then land a new version of it that always falls back to lvalue resolution, but with an on-by-default warning (not error) (so this would probably affect a ton of your test cases); then the user can downgrade that warning as a workaround.

I have no strong/well-informed opinions here. I have an idea to offer, if there is precedent for it. My idea is, keep it as an error if the rvalue resolution finds nothing, but (1) fall back to an lvalue resolution for error-recovery purposes, and (2) suppress the error in system headers. Then, the usage in MSVC STL will keep working but only because it's in a system header. I seem to recall that there is precedent for diagnostics that are "non-downgradable error in user code, yet still suppressed in system headers." If I'm imagining things, then please do not invent such a facility purely for this feature.

That would still leave the effect of changing the deduced type of a decltype(auto) returning function. But I am on the same boat here in that I am not sure if you are imagining things as well :)

My second choice is to do whatever minimalistic hack suffices to satisfy the customer (Aaron etc. in this case), but only with the understanding that we plan to eliminate that hack as soon as MSVC STL gets updated. My third choice is to revert the whole thing and then land a new version of it that always falls back to lvalue resolution, but with an on-by-default warning (not error) (so this would probably affect a ton of your test cases); then the user can downgrade that warning as a workaround.

Again, the change in flow where we make the expression an xvalue early and definitively before return type deduction occurs makes things more complicated.

My vote is the second choice, as I think making a workaround where we temporarily do the C++20 thing only in some contexts is far simpler, specially if we don't have to update a ton of test cases to cover it :)

@aaron.ballman I created a DR for the option of temporarily disabling the effects of this change under "-fms-compatibility".
https://reviews.llvm.org/D105518

I'd prefer to keep existing code working without requiring users to enable a flag, if at all possible (esp because I don't expect the flag to be needed forever). That said, I still think it makes sense to temporarily revert this while doing that work.

I'd like to give some time for other stakeholders to give their opinion if this is not too urgent, specially @Quuxplusone and @rsmith.

This makes clang-cl basically unusable for a significant portion of people (enough so that I'm asking for a temporary revert) and I don't think we should continue to leave trunk in a broken state while deciding what to do. Our usual community approach is to revert out changes that cause significant breakage because we always want people (such as downstreams) to be able to track against trunk. That said, based on below, hold off on reversion as I may have misunderstood the fix patch (see below) and that may be the quicker and less disruptive approach.

But thinking of another option, almost as easy as a global revert for this, we could suppress the effect under -fms-compatibility for now, and then implement a more targetted fix where we only suppress this when parsing the STL system headers themselves under fms-compatibility.

Ah! I missed that context in the other patch, thank you for this! I'll go comment on the other review so we keep the discussion in one place, but that makes me more comfortable with the approach from that review.