Details
- Reviewers
ldionne cjdb • Quuxplusone - Group Reviewers
Restricted Project - Commits
- rG1a29403d2f8a: [libcxx][ranges] Add common_iterator.
Diff Detail
- Repository
- rG LLVM Github Monorepo
Event Timeline
libcxx/include/__iterator/common_iterator.h | ||
---|---|---|
100 | Note: all of these constexprs aren't required by the standard, but I'm filing an lwg issue for it. |
libcxx/include/__iterator/common_iterator.h | ||
---|---|---|
100 | Please link the issue in this PR before submitting (it doesn't need to be in the patch itself). |
libcxx/include/__iterator/common_iterator.h | ||
---|---|---|
42–44 | Default sentinel deserves its own (tiny) patch. |
libcxx/include/__iterator/common_iterator.h | ||
---|---|---|
53 | This isn't going to work if either of them has a non-trivial special member function. The exposition-only variant really solved a lot of problems in the specification that this will need to solve again. |
- I think I adressed all the review comments that apply from the other two views. (If not, sorry, let me know and I'll fix them here too.)
- I'm going to update common_iterator to use variant. Because it's not yet constexpr, I'm just going to comment out the static_assert in all the tests. This makes us support closer to the standard anyway. Note: I have not done this yet because it requries fixing a dependency cycle between iterator -> array -> variant -> iterator.
libcxx/test/std/iterators/predef.iterators/iterators.common/members.pass.cpp | ||
---|---|---|
1 ↗ | (On Diff #356841) | Oops, I can delete this now that it's tested elsewhere. |
libcxx/test/support/test_iterators.h | ||
165 | I think this will be a very helpful type to have now that we need to test the default constructibility of all our views/iterators. |
libcxx/include/__iterator/common_iterator.h | ||
---|---|---|
34–42 | Reminder that this should be using std::variant. | |
100 | I think we should not make them constexpr until the LWG issue has been accepted and resolved. I strongly suspect the reason why they are not constexpr is that the type is expected to use variant, which isn't constexpr yet. You can structure the tests so that constexpr is easy to test when we add it, but please don't add it in the implementation yet. | |
116 | http://eel.is/c++draft/iterators.common#common.iter.nav-5 says:
We're missing && move_constructible<...> here? | |
119 | This should be __postfix_proxy __p(**this); (double-dereference to get the value pointed-to by the iterator). It also means you have a hole in your coverage since your tests were passing, i.e. this branch of the if constexpr was never exercised. Please make sure all branches of the if constexpr are exercised (here and in operator-> above too). | |
libcxx/test/std/iterators/predef.iterators/iterators.common/arrow.pass.cpp | ||
27 | Here and for operator++(int), I'd like you to explicitly call out which part is testing each sub-bullet in the spec (which maps to the if constexpr in the implementation). | |
libcxx/test/std/iterators/predef.iterators/iterators.common/assign.pass.cpp | ||
31 | You have a typo in sentienl_type. | |
libcxx/test/support/test_iterators.h | ||
165 | I agree, but I would instead call it something like non_default_constructible_iterator. Or whatever you prefer, but I think it's important for the name to reflect the fact that it's an iterator. | |
171 | Shouldn't this be std::iterator_traits<It>::iterator_category? Actually, we could perhaps do this instead: template<class It> struct std::iterator_traits<no_default_constructor<It>> : std::iterator_traits<It> { }; Then, could we simply drop all the typedefs in no_default_constructor? |
libcxx/include/__iterator/common_iterator.h | ||
---|---|---|
116 | Good catch. It looks like this was recently changed. | |
119 | Yes.. you're right. I thought that value_iterator was testing this, but apparently not. I guess the type I need here is one where i is a __referenceable iterator and ++i is a __referenceable iterator, but i++ is void or something? Arg. OK. | |
libcxx/test/support/test_iterators.h | ||
171 | No, because we only provide members such that this is a forward iterator. For example, this is used mostly with pointer types. If we did what you're suggesting these would report that they were contiguous_iterators, but they are not (they're not even bidirectional). |
libcxx/test/support/test_iterators.h | ||
---|---|---|
171 | Turn this inside out. template<class T> class NonDefCtablePtr { T *p_; public: using iterator_category = std::random_access_iterator_tag; using value_type = T; using difference_type = std::ptrdiff_t; [...] constexpr NonDefCtablePtr(T *p) : p_(p) {} constexpr T& operator*() const { return *p_; } [...] friend constexpr T *base(NonDefCtablePtr p) { return p.p_; } }; and then in your test cases, use #include "test_iterator.h" test<bidirectional_iterator<int*>>(); // bidirectional, default-constructible test<bidirectional_iterator<NonDefCtablePtr<int>>>(); // bidirectional, non-default-constructible That is, we already have the iterator wrapper types in order to "downgrade" int* to forward/bidirectional/random-access level. So your NonDefCtable type should not "downgrade." It should just be contiguous, and then if someone wants a downgraded version, they can wrap it in one of the downgrade wrappers. |
libcxx/include/__iterator/common_iterator.h | ||
---|---|---|
116 | OK I added a test, but I'd like someone to verify it's correct. See my other comment. |
libcxx/include/__iterator/common_iterator.h | ||
---|---|---|
41 | Missing move here. | |
54 | Missing forward here. | |
71 | variant does not have heterogeneous conversions of this sort. | |
77 | Likewise. | |
80–83 | It's undefined if it is holding the wrong type, so the check-and-throw get performs is unnecessary (and hurts performance). There should be a way to avoid such checks. | |
124 | __y's sentinel type is _S2, not _Sent. Ditto below. There seems to be a lack of test coverage here. | |
libcxx/test/std/iterators/predef.iterators/iterators.common/plus_plus.pass.cpp | ||
88 | Yes, common_iterator does its best to return something from postfix ++ that meets the C++17 input iterator requirements. But hitting this case means that it doesn't know a way to produce anything meaningful, so it just gives up. | |
libcxx/test/support/test_iterators.h | ||
171 | Something that's not default constructible can't be anything more than input. |
libcxx/include/__iterator/common_iterator.h | ||
---|---|---|
54 | They are really both forwards; the difference is that operator->'s proxy only gets used when iter_reference_t<I> isn't a reference, so that forward ends up being a move. |
Comments per the over-the-shoulder review we just did.
libcxx/include/__iterator/common_iterator.h | ||
---|---|---|
32–39 | This should be defined in <variant>. Also, I would word the comment differently (because assertions are disabled in most builds):
| |
93 | I'm pretty sure this shouldn't even compile since we're passing the runtime __other.__hold_.index() as a non-type template argument, which expects a constant expression. This means that this isn't tested. I can think of two ways of implementing this. First (pseudo-code): __hold_([]() -> variant<_Iter, _Sent> { if (__other.__hold_.index() == 0) return {in_place_index<0>, __other.__hold_}; else return {in_place_index<1>, __other.__hold_}; _LIBCPP_UNREACHABLE(); }()) The second way would be to use variant<monostate, _Iter, _Sent> and to "default-construct" the monostate when entering this constructor, and then do the if-else dance above in the constructor body. Otherwise, we'll have to default-construct one of the variant alternatives, which we don't want to do (BTW it would be awesome if you could add a test for that). Pinging @mpark in case he has better ideas. | |
98 | Could we add _LIBCPP_ASSERT(!__other.__hold_.valueless_by_exception()) here since it's a precondition in the spec? Also in the other places where it applies. | |
100 | This should be std::get<index>(__hold_) = std::get<index>(__other.__hold_);. | |
102 | This is going to have the same problem as above, it won't compile. That means there's also a hole in our test coverage. |
libcxx/include/__iterator/common_iterator.h | ||
---|---|---|
106 | Maybe add _LIBCPP_ASSERT(_VSTD::holds_alternative<_Iter>(__hold_))? | |
115 | Could you break that { onto a different line? It's kind of hard to see where the requires end and the function body begins. | |
146 | Can we hoist the value of xxxx.index() into local variables? (here and below) | |
157 | Please make sure you test all the branches below (here and in the other functions below too). | |
203–205 | This would fit nicely on a single line IMO. | |
libcxx/test/std/iterators/predef.iterators/iterators.common/iterator_traits.compile.pass.cpp | ||
57 | This doesn't look right. According to the spec (http://eel.is/c++draft/iterators.common#common.iter.types-1.3), this should either be the type of operator->() or void, but never Iter const&. This means you most likely have a bug in the implementation. Also, please add a test where pointer ends up being void. |
libcxx/include/__iterator/common_iterator.h | ||
---|---|---|
157 | Added a few tests. Have verified all paths are taken. | |
libcxx/test/std/iterators/predef.iterators/iterators.common/iterator_traits.compile.pass.cpp | ||
57 |
Turns out this is actually correct. This is supposed to be the return type of common_iterator's operator->(). And if _Iter has an operator->() then it's supposed to simply return get<Iter>(v). (However, I'd argue that should maybe return get<Iter>(v).operator->(), if you agree, maybe I'll file an LWG issue.) Sound good now? This threw me off too :P | |
57 |
Done. |
libcxx/test/std/iterators/predef.iterators/iterators.common/iterator_traits.compile.pass.cpp | ||
---|---|---|
57 |
Producing decltype(a.operator->()) matches http://eel.is/c++draft/iterator.traits#1.sentence-3 but not the actual wording in http://eel.is/c++draft/iterators.common#common.iter.types-1.3 which says that it is the type of a.operator->(), aka remove_reference_t<decltype(a.operator->())>. What's the intent here @CaseyCarter? |
libcxx/include/__iterator/common_iterator.h | ||
---|---|---|
79 | This should be in the lambda above for it to trigger before we try to access __other.__hold_ unchecked. | |
93 | I think you want _VSTD::__unchecked_get here and elsewhere. | |
98 | Can you add a test where you assign to a common_iterator with a valueless_by_exception() variant? | |
108 | As discussed live: we can use a better message here. | |
170 | How is this behavior really what we want? IIUC, that means that if you compare two common_iterators that both contain different iterators (say It1 and It2) that are not comparable to each other, the common_iterators will compare equal to each other even if It1 and It2 are "pointing" to entirely different elements. Am I misunderstanding something here, or that's an incredibly subtle (and broken) behavior to have at runtime? @tcanens Can you shed some light on this? In all cases, we need a test where we exercise that. |
libcxx/include/__iterator/common_iterator.h | ||
---|---|---|
41 | Add a comment: "iter_reference_t" is never a reference here because if it is we catch it in the if constexpr branch above where this one is called. | |
108 | This should say something like "common_iterator not holding an iterator (holding a sentinel) must hold iterator." | |
170 | We should assert these aren't both zero, even though it's non-conforming, people will thank us. |
libcxx/include/__iterator/common_iterator.h | ||
---|---|---|
170 | This is meant for C++20 input iterators that aren't comparable with each other. Because incrementing an input iterator invalidates all copies, you can't have two valid iterators pointing to different elements in the same range.
Please don't. i == i had better work. |
libcxx/test/std/iterators/predef.iterators/iterators.common/iterator_traits.compile.pass.cpp | ||
---|---|---|
57 | http://eel.is/c++draft/iterator.traits#1.sentence-3 is authoritative; http://eel.is/c++draft/iterators.common#common.iter.types-1.3 should be corrected to agree. Presumably the writer of the latter paragraph was under the impression that a.operator->() is always a prvalue. |
libcxx/include/__iterator/common_iterator.h | ||
---|---|---|
170 |
Shouldn't this assert (or require) that these are input iterators then?
But this is not i == i, it's x == y. What you're saying makes sense for input iterators, but what about forward iterators, or even contiguous iterators? (Thanks for all the help with interpreting the standard's wording here, by the way.) |
libcxx/include/__iterator/common_iterator.h | ||
---|---|---|
170 | common_iterator is a C++17 compatibility shim. I don't think this is worse than returning true when both are sentinels - two sentinels can mean completely different things, even if they have the same type. |
libcxx/include/__iterator/common_iterator.h | ||
---|---|---|
170 |
The difference is sentinels must always be the end of the range. I can get behind saying "this must always be true because the sentinels must always be the end of the same range which must be the same element." The part I'm having trouble with is it's OK to have two different forward iterators that point to the same range in different places. Those should return false when compared (or somehow generate an error or UB). |
Apply remaining few review comments :)
libcxx/include/__iterator/common_iterator.h | ||
---|---|---|
170 |
Tested in both assign.pass.cpp (line 35, 50, 68, 74) and eq.pass.cpp (line 78, 88, 89). |
libcxx/include/__iterator/common_iterator.h | ||
---|---|---|
170 | I'd prefer to see an explicit static_assert here: static_assert(!equality_comparable_with<_Iter, _I2>); return true; I think this assertion would sufficiently explain why we aren't checking i1 == i2 here — it's because we physically can't. Also, the assertion serves as an important backstop against subsumption bugs. The absolute worst thing that could happen here is that the constraint on line 179 bit-rots under maintenance and we end up executing this code for iterators that are comparable. | |
174 | _VSTD::__unchecked_get<0>(__x.__hold_) == _VSTD::__unchecked_get<1>(__y.__hold_) I don't trust __unchecked_get<_Iter> to do the right thing when _Sent is cvref-qualified _Iter or vice versa (or they're in some other subtle corner-case way related to each other). Integer indices, on the other hand, are nice and solid and trustworthy. Also, cppreference documents that the code should use get<i> and get<j>, not get<I> and get<S2>; it's just barely possible that there might be a technical reason for that. |
libcxx/include/__iterator/common_iterator.h | ||
---|---|---|
170 | forward iterators are required to be comparable, so it would go to the other overload. |
libcxx/include/__iterator/common_iterator.h | ||
---|---|---|
170 |
This is the overload that's chosen when we evaluate i == i, so Tim's got a point. I'm also not sure what the assertion would achieve? Are you trying to prevent i == i? |
Thanks for working on such an ugly type. In addition to my inline comments, please add
- a module map entry
- iterator conformance tests
libcxx/include/__iterator/common_iterator.h | ||
---|---|---|
36 | Can we define __proxy and __postfix_proxy out-of-line please? | |
64 | This should be private. Also, hold is not a particularly descriptive name to me. | |
108 | How about "Dereferencing a common_iterator is only possible if it is holding an iterator, but this one is holding a sentinel."? | |
123–125 | Can we get this into a named object please? Perhaps __common_iterator_has_arrow? | |
154–156 | Please name this. | |
273–275 | Seconded. Please use _If, if you can. | |
libcxx/test/std/iterators/predef.iterators/iterators.common/plus_plus.pass.cpp | ||
87 | Please prefix "postfix" to this sentence. | |
libcxx/test/support/test_iterators.h | ||
165 | I think this functionality should instead be reflected in cpp20_input_iterator (and in a separate patch, if it breaks any code). |
libcxx/include/__iterator/common_iterator.h | ||
---|---|---|
170 | I'm talking about two completely unrelated iterators, for example, this: auto iter1 = random_access_iterator<int*>(buffer + 1); auto commonIter1 = std::common_iterator<decltype(iter1), sentinel_type<int*>>(iter1); auto iter2 = forward_iterator<int*>(buffer + 4); auto commonIter2 = std::common_iterator<decltype(iter2), sentinel_type<int*>>(iter2); assert(commonIter1 == commonIter2); (An actual test case ^) If above is UB, it's not clear to me where that is stated. Even so, if it is UB for some reason, I think it would be great to add an assertion. And if it's well defined, well that seems terrible (and my preference would be to create an LWG issue and still add an assertion or something, because that's definitely a but, but maybe others disagree). |
libcxx/include/__iterator/common_iterator.h | ||
---|---|---|
170 | So pathological mix-up scenarios where you compare completely unrelated things? It's hard to think of a non-contrived example where that matters. I don't really care about this case either way; I'm just not convinced that it's worth the trouble (coming up with something that doesn't damage valid uses can be tricky). @CaseyCarter thoughts? |
libcxx/include/__iterator/common_iterator.h | ||
---|---|---|
170 | It's hard to care about this. The domain of equality for forward_iterators is iterators obtained from the same range, so generic code can't perform this comparison. To do so in concrete code - when you know the result is useless - would be silly. In other words: if you think this is weird, then don't do it. Conversely, if you want comparison of common_iterator<I1, S> and common_iterator<I2, S> to be sensible you can always make I1 and I2 model equality_comparable. |
libcxx/include/__iterator/common_iterator.h | ||
---|---|---|
36 | Why? | |
64 | Because of the friend issues we discussed with transform_view on #libcxx we unfortunately can't have this be private. | |
123 | @tcanens Another potential LWG issue: shouldn't this be is_pointer_v<_Iter> || requires(const _Iter& __i) { __i.operator->(); }? If not, do we need to have the is_pointer condition below? Can it ever be hit? | |
123–125 | I don't want to add another "has arrow" concept that means something different from __has_arrow. Depending on my comment above, we might be able to use that here, though. | |
170 | I guess it's pathological, but it would also be pathological to even compare any iterators (read: not sentinels) that use this overload. Basically, you're saying "you should only compare iterators here that are equal." In which case, why does this overload even need to work with iterators (read: not sentinels)? We have two cases: 1) someone is using this correctly. They have two iterators that are equal. This returns true.
You're saying the second case isn't worth worrying about because it would be pathological. This implies that no one should ever call this function without knowing if their iterators are equal beforehand. This begs the question: why do they even need to call this function in the first place? If they know their iterators are equal, it's pointless to compare them. I think we should add a precondition or something that says i or j is 1. And I'd like to add that as an assertion here. | |
174 | I'm not opposed to using indexes, but a concrete example would be nice, to show that there's actually a problem. I find using types much more readable, so I'd want a solid reason not to do it. | |
libcxx/test/support/test_iterators.h | ||
165 | Common iterator requires that we're copyable, so we really need this type. But I'm fully supportive of making cpp20_input_iterator non-default-constructible in a separate patch. |
libcxx/include/__iterator/common_iterator.h | ||
---|---|---|
123 | is_reference_v<iter_reference_t<_I2>> should be enough to catch pointers. | |
170 |
Because perfectly valid generic code written to C++17 iterator requirements may want to do this, perhaps because it makes the code simpler. Breaking valid code makes common_iterator unfit for its only purpose. This is like "why does array<T, 0> has a front() when it's always UB to call it"? Well, because generic code may want to do so from a dynamically unreachable context.
That's a non-starter as far as I'm concerned. |
libcxx/include/__iterator/common_iterator.h | ||
---|---|---|
36 | I find class-in-class definitions to clutter the outer-class definition. For the record, it's still common_iterator<_Iter, _Sent>::__proxy. | |
170 |
That it's UB means an implementation can assert that N != 0.
Could you please elaborate on why you think this is a non-starter? Having said that, I keep bouncing on my opinion for this matter whenever new evidence is put forward. |
libcxx/include/__iterator/common_iterator.h | ||
---|---|---|
170 | @CaseyCarter said:
Ok. So basically, we never expect this overload to be called with It1 and It2 being forward_iterators, and this should only be called for input_iterators? In that case, I don't understand what's preventing us from adding a constraint !forward_iterator<It1> && !forward_iterator<It2> -- it's really cheap to do that, and we'd catch this pathological case at compile-time instead of having really strange behavior at runtime. And this will still work for all input_iterators, which is the only intended use case for this comparison operator if I followed correctly. Do you agree? @zoecarver Whatever we end up doing here, I agree with Arthur's suggestion to add an assertion like static_assert(!equality_comparable_with<_Iter, _I2>); before returning true. |
libcxx/include/__iterator/common_iterator.h | ||
---|---|---|
170 |
At runtime, not at compile time. "It's always UB when called" doesn't justify = delete; or removal or static_assert in the body, because reasonable code can do if (not a.empty()) { do_something(a.front()); }. Similarly here, just because the only meaningful comparison for iterators must return true doesn't justify making the comparison undefined.
Comparing a non-singular common_iterator with itself, or a copy of itself, must be valid. Making perfectly valid uses undefined at runtime just to catch pathological cases that are unlikely to ever arise in practice is totally backwards. I'm open to something like what Louis suggested. |
At runtime, not at compile time.
[...]
I'm open to something like what Louis suggested.
To clarify, I'm suggesting that we add a runtime assertion that does some combination of i == 1 || j == 1 || (!forward_iterator<It1> && !forward_iterator<It2>).
I'm not seeing any blocking issues anymore, and it looks reasonable to me. I think we should file the LWG issue for operator== and ship it. Other reviewers, please chime in if you have feedback you really want to see addressed before this is shipped, otherwise @zoecarver feel free to ship this after the weekend or so (to give time for people to see this).
(needs CI)
libcxx/include/__iterator/common_iterator.h | ||
---|---|---|
170 | @zoecarver Let's implement it per the spec for now and file a LWG issue with a proposed resolution like this: template<class _I2, sentinel_for<_Iter> _S2> requires sentinel_for<_Sent, _I2> && !equality_comparable_with<_Iter, _I2> && // I think this is required to avoid an ambiguity with the other overload !forward_iterator<_Iter> && !forward_iterator<_I2> friend bool operator==(const common_iterator& __x, const common_iterator<_I2, _S2>& __y); Either this is accepted by LWG and we fix it before we've had time to ship it anyways, or it's not and in that case we'll be conforming (and hopefully LWG explains why they think this isn't a good resolution). Are you willing to file it? |
libcxx/include/variant | ||
---|---|---|
209 | Please sort the list of includes. |
Can we define __proxy and __postfix_proxy out-of-line please?