This is an archive of the discontinued LLVM Phabricator instance.

[libc++] Fix common_iterator for output_iterators
ClosedPublic

Authored by ldionne on Jan 16 2022, 7:24 PM.

Details

Summary

We were missing a constraint in common_iterator's iterator_traits and
we were eagerly instantiating iter_value_t even when invalid.

Thanks to Casey Carter for finding this bug.

Diff Detail

Event Timeline

CaseyCarter requested review of this revision.Jan 16 2022, 7:24 PM
CaseyCarter created this revision.
Herald added 1 blocking reviewer(s): Restricted Project. · View Herald TranscriptJan 16 2022, 7:24 PM
Quuxplusone added a subscriber: Quuxplusone.

Nice catch! That test is a mess, though. I don't object to landing this as-is, just to fix the bug; I would also be happy to merge it into D117400. Your call.

libcxx/test/std/iterators/predef.iterators/iterators.common/iterator_traits.compile.pass.cpp
67

Well this is weird. non_const_deref_iterator<int*>::iterator_category appears to be input_iterator_tag, but std::common_iterator<Iter, sentinel_type<int*>> is an output iterator instead?

Also, I'm seeing now that non_const_deref_iterator is a template, but it's used in only one place (with int*), so it can be massively simplified. Would you like me to just take this over and merge it into my existing D117400?

ldionne added inline comments.
libcxx/test/std/iterators/predef.iterators/iterators.common/iterator_traits.compile.pass.cpp
67

Yeah, I'd like to understand this part as well.

I don't object to landing this as-is, just to fix the bug; I would also be happy to merge it into D117400. Your call.

I have a mild preference to merge this small fix as-is.

libcxx/test/std/iterators/predef.iterators/iterators.common/iterator_traits.compile.pass.cpp
67

input_iterator requires that const iterators are dereferenceable. Hopefully it's obvious that is not the case for non_const_deref_iterator, so it does not trigger the iterator_traits<common_iterator<I, S>> partial specialization once that partial specialization is properly constrained to require input_iterator.

@ldionne rubberstamp here as-is, and revisit in D117400? :)

libcxx/test/std/iterators/predef.iterators/iterators.common/iterator_traits.compile.pass.cpp
67

I get everything you said, but I still don't get why "Iter doesn't satisfy input_iterator" should lead us to "iterator_traits<CommonIter>::iterator_category is defined and is equal to output_iterator_tag." Like, Iter isn't an output iterator either... is it? So the most logical outcomes (in increasing order of wild-west-ness) would have been for CommonIter to be ill-formed, or for CommonIter to be well-formed but not an iterator, or for iterator_traits<CommonIter>::iterator_category to be defined as equal to iterator_traits<Iter>::iterator_category. Having it just randomly equal to output_iterator_tag seems bizarre!

(I'm not disputing that it's probably correct according to C++20; I'm just confused about both the mental model and, literally, I don't see by what codepath we end up with output_iterator_tag. Not that I've looked hard at all.)

ldionne added inline comments.Jan 18 2022, 8:41 AM
libcxx/test/std/iterators/predef.iterators/iterators.common/iterator_traits.compile.pass.cpp
67

Thanks for explaining @CaseyCarter , but in that case I fail to see the point of this test at all. Instead, why don't we use common_iterator with an output_iterator, and then simply check that IterTraits::iterator_concept isn't defined (which would effectively be a regression test for your change). In other words:

using Iter = output_iterator<int*>;
using CommonIter = std::common_iterator<Iter, sentinel_type<int*>>;
using IterTraits = std::iterator_traits<CommonIter>;

static_assert(!HasIteratorConcept<IterTraits>);
static_assert(std::same_as<IterTraits::iterator_category, std::output_iterator_tag>);
// etc...

Do you agree that is what we want? Note that this won't work today because it appears that common_iterator<output_iterator<int*>> doesn't work at all today, but we can and should fix that separately (I have a WIP patch locally).

ldionne commandeered this revision.Jan 18 2022, 9:10 AM
ldionne edited reviewers, added: CaseyCarter; removed: ldionne.

Commandeering to fix common_iterator with output_iterator. I would have done it as part of a separate patch, but both are interdependent (I can't write tests for my part of the change without your changes).

ldionne updated this revision to Diff 400874.Jan 18 2022, 9:22 AM
ldionne retitled this revision from [libcxx] Constrain common_iterator's iterator_traits specialization to [libc++] Fix common_iterator for output_iterators.
ldionne edited the summary of this revision. (Show Details)

Fix common_iterator for output_iterator. Do you folks agree this is the right direction?

I continue to think all of these directions are better than the status quo. Consider me still "accept"ing.

libcxx/include/__iterator/common_iterator.h
34–36

Nit: Did you add line 34 to avoid prematurely instantiating iter_reference_t? Is it possible to regression-test for that? Should we? If removing line 34 would be conforming, I'd just do that.

libcxx/test/std/iterators/predef.iterators/iterators.common/plus_plus.pass.cpp
86–87
153–154

IIUC this comment is misleading. iter_value_t for an ordinary C++11 output iterator is just ThatType::value_type. It's only if the iterator doesn't define a value_type, or defines it as a non-object type, that iter_value_t might be ill-formed.
https://godbolt.org/z/1Ybjv7o9G
Our test iterator's output_iterator<int*>::value_type is void, so its iter_value_t is indeed ill-formed, but that's a special property of our output_iterator, not all output iterators in general.

I'd like to see test coverage for "output iterator with valid value_type" added to iterator_traits.compile.pass.cpp above, even if this means defining a little helper type a la the OutIt in my godbolt.

173

Consider checking iter != sent on at least one of the previous lines.

LGTM, with Arthur's comments. (Although I'm fine with the extra typename requirement in __can_use_postfix_proxy.)

libcxx/test/std/iterators/predef.iterators/iterators.common/iterator_traits.compile.pass.cpp
67

common_iterator assumes that non-input_iterators must be output_iterators. Testing with non_const_deref_iterator is equivalent to testing with output_iterator if we never dereference an iterator. common_iterator doesn't intend to be as general as possible: it exists only to adapt "new" iterator+sentinels into "old" iterator pairs. Consequently it has little patience with types that don't satisfy the concepts.

libcxx/test/std/iterators/predef.iterators/iterators.common/plus_plus.pass.cpp
153–154

"need not be valid for output iterators" would be fine here.

ldionne updated this revision to Diff 402951.Jan 25 2022, 9:54 AM
ldionne marked 8 inline comments as done.

Apply review comments.

libcxx/include/__iterator/common_iterator.h
34–36

No, the constraint was redundant with simply using iter_value_t in the expressions themselves. Removed.

libcxx/test/std/iterators/predef.iterators/iterators.common/plus_plus.pass.cpp
153–154

Agreed, the comment was wrong, I'll fix it. However:

I'd like to see test coverage for "output iterator with valid value_type" added to iterator_traits.compile.pass.cpp above, even if this means defining a little helper type a la the OutIt in my godbolt.

The problem is that even though iter_value_t<OutputIter> *can* be non-void, iter_value_t<common_iterator<OutputIter>> will never be, because we are now properly constraining the iterator_traits<common_iterator<...>> specialization to only work on input_iterators. So I can't add a test that checks that iter_value_t<common_iterator<OutputIter>> is going to be non-void.

Please LMK if you think I'm mistaken here.

I assume this is still OK, and still necessary, even after D117400 which I assume must have merge-conflicted with it a lot?

libcxx/test/std/iterators/predef.iterators/iterators.common/plus_plus.pass.cpp
153–154

I haven't investigated so I assume you're not mistaken. Then it just sounds like my request shifts from test coverage for common_iterator<OutputIterWithNonVoidValueType>::iterator_category, to test coverage that common_iterator<OutputIterWithNonVoidValueType> is SFINAE-friendly ill-formed. I believe we have some prior art, shaped roughly like

template<class T>
concept HasCommonIterator = requires { typename std::ranges::common_iterator<T>; };
static_assert(!HasCommonIterator<OutputIterWithNonVoidValueType>);
ldionne updated this revision to Diff 403379.Jan 26 2022, 12:49 PM
ldionne marked an inline comment as done.

Add test.

I assume this is still OK, and still necessary, even after D117400 which I assume must have merge-conflicted with it a lot?

Yes, still necessary. There were only small merge conflicts.

libcxx/test/std/iterators/predef.iterators/iterators.common/plus_plus.pass.cpp
153–154

I'm not following. common_iterator<OutputIterWithNonVoidValueType> does work, it's only that you can't query its iterator_traits.

I added a test in iterator_traits.compile.pass.cpp which isn't the most useful (because it ends up basically testing the iterator_traits base template), but should clarify what I mean.

ldionne accepted this revision as: Restricted Project.Jan 26 2022, 1:18 PM

Will ship once CI passes unless @Quuxplusone has other comments.

This revision is now accepted and ready to land.Jan 26 2022, 1:18 PM
Quuxplusone added inline comments.
libcxx/test/std/iterators/predef.iterators/iterators.common/plus_plus.pass.cpp
153–154

I'm not following. common_iterator<OutputIterWithNonVoidValueType> does work, it's only that you can't query its iterator_traits.

Well, now I'm confused. How can you have an iterator, that "works," but you can't query its iterator_traits? What does that even mean? I would think that by definition a "working iterator" must have iterator_traits that say it's an iterator. Otherwise it's not an iterator.

I pasted your NonVoidOutputIterator into a Godbolt; what happens after this patch? (Also I assume that spew on GCC indicates a libstdc++ bug @jwakely.) https://godbolt.org/z/KrxY6Wd97
I'm surprised that common_iterator<OI, ...>::value_type is void instead of OI::value_type, but I think that's what the standard literally says, yeah.

The CI only failed on Windows due to missing space. I'm going to ship this now, and we can make adjustments post-commit if the discussion ends up bringing up additional issues.

libcxx/test/std/iterators/predef.iterators/iterators.common/plus_plus.pass.cpp
153–154

Sorry, I should have been precise. I don't mean that iterator_traits<common_iterator<OutputIterWithNonVoidValueType>> doesn't work. I mean that iterator_traits<common_iterator<OutputIterWithNonVoidValueType>>::value_type is void, despite OutputIterWithNonVoidValueType::value_type not being void. And yes, I think it's a bit silly, but it's definitely what the Standard says.

The reason is simply that iterator_traits<common_iterator<It>> requires It to be an input_iterator. If it is, then the value_type is properly forwarded through. If it isn't, then we end up falling back on the base template of iterator_traits (https://en.cppreference.com/w/cpp/iterator/iterator_traits):

Otherwise, if Iter satisfies the exposition-only concept __LegacyIterator, the member types are declared as follows:

Member type         Definition
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
difference_type     std::incrementable_traits<Iter>::difference_type if valid, otherwise void
value_type          void
pointer             void
reference           void
iterator_category   std::output_iterator_tag

That's my understanding anyways.

This revision was landed with ongoing or failed builds.Jan 27 2022, 7:57 AM
This revision was automatically updated to reflect the committed changes.