This is an archive of the discontinued LLVM Phabricator instance.

[libcxx] deprecates `std::iterator` and removes it as a base class
AbandonedPublic

Authored by cjdb on May 2 2021, 1:37 PM.

Details

Reviewers
ldionne
EricWF
zoecarver
Mordante
Quuxplusone
curdeius
Group Reviewers
Restricted Project
Summary

C++17 deprecated std::iterator and removed it as a base class for all
iterator adaptors.

Affected subclauses:

  • [depr.iterator] (previously [iterator.iterator])
  • [reverse.iterator]
  • [back.insert.iterator]
  • [front.insert.iterator]
  • [insert.iterator]
  • [istream.iterator]
  • [ostream.iterator]
  • [istreambuf.iterator]
  • [ostreambuf.iterator]
  • [depr.storage.iterator]

Implements part of P0174R2 'Deprecating Vestigial Library Parts in C++17'.

Diff Detail

Event Timeline

cjdb requested review of this revision.May 2 2021, 1:37 PM
cjdb created this revision.
Herald added a project: Restricted Project. · View Herald TranscriptMay 2 2021, 1:37 PM
Herald added a reviewer: Restricted Project. · View Herald Transcript

Looks pretty good to me, mod comments.

libcxx/include/__memory/raw_storage_iterator.h
30–36

Interestingly, libc++ has never been conforming here.
https://timsong-cpp.github.io/cppwp/n3337/storage.iterator
requires inheriting from iterator<output_iterator_tag, void, void, void, void>.
But since this is removed in '20, do we even care about it? I guess not; so this #if seems like the minimal change, and therefore OK by me.

41–47

Here, we could drop the #if and just unconditionally define these members... except that that would visibly change the value of raw_storage_iterator::difference_type (from non-conforming to conforming), and maybe we don't want that? But again I'd think we don't really care about this type and so we might as well lose the #if.

50–54

I vote for not reformatting here. Keeps the diff smaller.

libcxx/include/iterator
85

Stray ) here. But please remove this diff anyway; [[deprecated]] doesn't appear in the Standard and even if it did I don't think it needs to appear in the synopsis comments.

629–631

Lose the #if here.
(Any idea why the old code bothered to redundantly redefine difference_type etc.?)

libcxx/test/libcxx/iterators/iterator.requirements/iterator.concepts/cpp20_iter_traits.pass.cpp
45–53

Keep this one inheriting from iterator_traits's primary template and the next one inheriting from iterator's primary template; otherwise you lose the point of the test.
I recommend

struct MyIter {
  using iterator_category = std::random_access_iterator_tag;
  using value_type = char;
  using difference_type = std::ptrdiff_t;
  using pointer = char*;
  using reference = char&;
};
struct MyIter2 : MyIter {};
struct MyIter3 : MyIter {};
template<> struct std::iterator_traits<MyIter> : std::iterator_traits<char*> {};
#if TEST_STD_VER <= 17
template<> struct std::iterator_traits<MyIter2> : std::iterator<char*> {};
#endif

However, if you just wanted to rip out MyIter2 for being excessively silly, that might be a better solution.

libcxx/test/std/iterators/predef.iterators/reverse.iterators/reverse.iterator/types.pass.cpp
56

This isn't right when T::iterator_category is contiguous_iterator_tag, although maybe the test doesn't care.
This also isn't right when T::difference_type isn't ptrdiff_t, etc.; but I'm sure the test doesn't care.

libcxx/test/std/iterators/stream.iterators/istream.iterator/types.pass.cpp
49

Might as well lose the double-parens while you're here. (Presumably a holdover from when static_assert was a macro.)

libcxx/test/std/iterators/stream.iterators/ostreambuf.iterator/types.pass.cpp
29–30

Throughout: It looks like buildkite isn't happy with some of these pragmas; and I don't really like them either... Aha!
git grep deprecated libcxx/include/
git grep const_fun_mem_t libcxx/test/
reveals that the correct incantation is

#define _LIBCPP_DISABLE_DEPRECATION_WARNINGS

at the top of the test. Please make that change throughout.

cjdb planned changes to this revision.May 4 2021, 1:39 PM

Haven't had time to respond to changes (and this is lowish in my p-queue).

libcxx/include/iterator
85

Let's take this one to Discord, I have some thoughts on this.

629–631

According to https://timsong-cpp.github.io/cppwp/n4140/reverse.iterator, it's standard to do this :S
I'm more curious to know why value_type is missing than why the others were redundantly added.

Does this change your opinion at all?

libcxx/test/std/iterators/predef.iterators/reverse.iterators/reverse.iterator/types.pass.cpp
56

This isn't right when T::iterator_category is contiguous_iterator_tag, although maybe the test doesn't care.
This also isn't right when T::difference_type isn't ptrdiff_t, etc.; but I'm sure the test doesn't care.

I'm inclined to say the test doesn't care today, but it will should care when we get around to applying P0896. I've got no qualms punting this to future me, but since I wager an even chance that @zoecarver will beat me to it, I'd like his input before making that decision.

Quuxplusone added inline comments.May 4 2021, 2:57 PM
libcxx/include/iterator
629–631

Nope, I'd still just drop the #if.

tcanens added a subscriber: tcanens.May 5 2021, 8:22 AM
tcanens added inline comments.
libcxx/include/iterator
629–631

It redeclares things that it needs to use in the rest of the class, because name lookup doesn't look into dependent bases.

ldionne requested changes to this revision.May 5 2021, 10:28 AM

Generally looks good, thanks for doing this cleanup. I agree we should unconditionally define the various typedefs everywhere to remove some #ifs.

libcxx/include/__memory/raw_storage_iterator.h
41–47

I think it's an acceptable change - I doubt it will break many users since raw_storage_iterator isn't widely used, and it's a compile-time breakage in all cases, so not some nasty runtime issue.

libcxx/include/iterator
85

Please use // deprecated in C++17 instead -- that is what we do elsewhere.

libcxx/test/libcxx/iterators/iterator.requirements/iterator.concepts/cpp20_iter_concepts.pass.cpp
11–12

Why do we need this? Aren't we removing the use of those deprecated decls?

Removing the std::iterator inheritance is potentially ABI breaking. In particular in the case of reverse_iterator.
If the iterator provided to reverse_iterator also inherits from std::iterator, then the first member of reverse_iterator goes from having an offset of 4 to an offset of 0 bytes
(because the compiler couldn't place the same empty base at the same address).

I'm not sure how much we care, but I see this being a case that could easily arise in real world code.
@ldionne thoughts?

Removing the std::iterator inheritance is potentially ABI breaking. In particular in the case of reverse_iterator.
If the iterator provided to reverse_iterator also inherits from std::iterator, then the first member of reverse_iterator goes from having an offset of 4 to an offset of 0 bytes
(because the compiler couldn't place the same empty base at the same address).

I'm not sure how much we care, but I see this being a case that could easily arise in real world code.
@ldionne thoughts?

Interesting catch Eric, thanks for chiming in. It looks like raw_storage_iterator has the same issue since the first member is a user-provided iterator, but I don't think we care for this one since it's deprecated in C++17 and removed in C++20 anyway (so there's likely no important ABI reliance on this type).

Thinking out loud: Before C++17, this is not an issue because std::iterator is there and we inherit from it. In C++17 and above, std::iterator is marked as [[deprecated]], so if a user inherits from it, they are getting the deprecation warning and also the potential ABI break in reverse_iterator. I see the following solutions:

  1. Don't care about it - it has a small chance to happen, and std::iterator is deprecated. If we do this, we should push to actually remove std::iterator ASAP, since we wouldn't have this problem if std::iterator were removed entirely.
  1. In >= C++17, add static_assert(!std::is_base_of<std::iterator, _Iter>::value, "Would be an ABI break"); inside reverse_iterator. That way, if anyone's using std::reverse_iterator with a type that would be potentially impacted by an ABI break, they'd be told at compile-time.
  1. Use:
class __one_byte { char __byte; };
class __empty { };

template <class _Iter>
class _LIBCPP_TEMPLATE_VIS reverse_iterator
#if _LIBCPP_STD_VER <= 14
    : public iterator<typename iterator_traits<_Iter>::iterator_category,
                      typename iterator_traits<_Iter>::value_type,
                      typename iterator_traits<_Iter>::difference_type,
                      typename iterator_traits<_Iter>::pointer,
                      typename iterator_traits<_Iter>::reference>
#else
    // We must keep the __t member below at the same offset in all Standards, which requires
    // some work if _Iter inherits from std::iterator.
    : private conditional_t<is_base_of_v<iterator<XXXX>, _Iter> && !is_final_v<_Iter>, __one_byte, __empty>
#endif
{
private:
    /*mutable*/ _Iter __t;  // no longer used as of LWG #2360, not removed due to ABI break

    static_assert(!__is_stashing_iterator<_Iter>::value,
      "The specified iterator type cannot be used with reverse_iterator; "
      "Using stashing iterators with reverse_iterator causes undefined behavior");

protected:
    _Iter current;

    // ...
};

(1) seems a bit reckless to me. (2) seems too constraining to me - I'm sure a lot of older code in the wild is still using std::iterator in ways that are not ABI sensitive, and it would be silly to break their code for that.

I think I'd go with (3) if we all agree it solves the ABI issue, since it's not such a big deal for us to do. In the future, I'd consider breaking the ABI of reverse_iterator and removing both this workaround and the __t workaround at once.

@ldionne: I recommend my blog post https://quuxplusone.github.io/blog/2021/05/07/std-iterator-as-a-base-class/
and the full discussion in D102657. I'll even volunteer to make the "remove all base classes for iterators and function-objects, and drive-by fix the reverse_iterator explosion, under an ABI flag" solution PR; it'll just take a week or two before I'm guaranteed to have time for it.

Also, concretely, @ldionne I believe your (3) still does not solve the ABI break for reverse_iterator<reverse_iterator<int*>>.
Also also, if we really cared about ABI breaks, we would put a test on them. Do you have any interest in adding some simple platform-specific tests (REQUIRES a certain platform) for things like static_assert(sizeof(std::reverse_iterator<...>) == ...)? Then ABI breakage would correspond to buildkite breakage, and people wouldn't be so likely to do it by accident.

cjdb added a comment.May 23 2021, 11:51 AM

@ldionne: I recommend my blog post https://quuxplusone.github.io/blog/2021/05/07/std-iterator-as-a-base-class/
and the full discussion in D102657. I'll even volunteer to make the "remove all base classes for iterators and function-objects, and drive-by fix the reverse_iterator explosion, under an ABI flag" solution PR; it'll just take a week or two before I'm guaranteed to have time for it.

That's what this patch does?

@ldionne: I recommend my blog post https://quuxplusone.github.io/blog/2021/05/07/std-iterator-as-a-base-class/
and the full discussion in D102657. I'll even volunteer to make the "remove all base classes for iterators and function-objects, and drive-by fix the reverse_iterator explosion, under an ABI flag" solution PR; it'll just take a week or two before I'm guaranteed to have time for it.

That's what this patch does?

This PR (D101729) currently doesn't do any of the three things I mentioned (remove base classes for iterators and function-objects, fix the reverse_iterator explosion, add an ABI flag). But I'd be happy to commandeer it; is that OK by you?

cjdb added a comment.May 23 2021, 7:27 PM

@ldionne: I recommend my blog post https://quuxplusone.github.io/blog/2021/05/07/std-iterator-as-a-base-class/
and the full discussion in D102657. I'll even volunteer to make the "remove all base classes for iterators and function-objects, and drive-by fix the reverse_iterator explosion, under an ABI flag" solution PR; it'll just take a week or two before I'm guaranteed to have time for it.

That's what this patch does?

This PR (D101729) currently doesn't do any of the three things I mentioned (remove base classes for iterators

This patch definitely removes std::iterator as a base class for all iterator adaptors. Is there some other base class I'm not seeing?

and function-objects,

Function objects are out of scope.

fix the reverse_iterator explosion,

I don't know what "explosion" you're referring to.

add an ABI flag). But I'd be happy to commandeer it; is that OK by you?

Your timeline is far out enough that it stalls other work. @Mordante is blocked by this patch.
It's also a simple enough thing to resolve (once we've agreed on a direction) that handing it over seems pointless.

Removing the std::iterator inheritance is potentially ABI breaking. In particular in the case of reverse_iterator.
If the iterator provided to reverse_iterator also inherits from std::iterator, then the first member of reverse_iterator goes from having an offset of 4 to an offset of 0 bytes
(because the compiler couldn't place the same empty base at the same address).

I'm not sure how much we care, but I see this being a case that could easily arise in real world code.
@ldionne thoughts?

Interesting catch Eric, thanks for chiming in. It looks like raw_storage_iterator has the same issue since the first member is a user-provided iterator, but I don't think we care for this one since it's deprecated in C++17 and removed in C++20 anyway (so there's likely no important ABI reliance on this type).

Thinking out loud: Before C++17, this is not an issue because std::iterator is there and we inherit from it. In C++17 and above, std::iterator is marked as [[deprecated]], so if a user inherits from it, they are getting the deprecation warning and also the potential ABI break in reverse_iterator. I see the following solutions:

  1. Don't care about it - it has a small chance to happen, and std::iterator is deprecated. If we do this, we should push to actually remove std::iterator ASAP, since we wouldn't have this problem if std::iterator were removed entirely.
  1. In >= C++17, add static_assert(!std::is_base_of<std::iterator, _Iter>::value, "Would be an ABI break"); inside reverse_iterator. That way, if anyone's using std::reverse_iterator with a type that would be potentially impacted by an ABI break, they'd be told at compile-time.
  1. Use:
class __one_byte { char __byte; };
class __empty { };

template <class _Iter>
class _LIBCPP_TEMPLATE_VIS reverse_iterator
#if _LIBCPP_STD_VER <= 14
    : public iterator<typename iterator_traits<_Iter>::iterator_category,
                      typename iterator_traits<_Iter>::value_type,
                      typename iterator_traits<_Iter>::difference_type,
                      typename iterator_traits<_Iter>::pointer,
                      typename iterator_traits<_Iter>::reference>
#else
    // We must keep the __t member below at the same offset in all Standards, which requires
    // some work if _Iter inherits from std::iterator.
    : private conditional_t<is_base_of_v<iterator<XXXX>, _Iter> && !is_final_v<_Iter>, __one_byte, __empty>
#endif
{
private:
    /*mutable*/ _Iter __t;  // no longer used as of LWG #2360, not removed due to ABI break

    static_assert(!__is_stashing_iterator<_Iter>::value,
      "The specified iterator type cannot be used with reverse_iterator; "
      "Using stashing iterators with reverse_iterator causes undefined behavior");

protected:
    _Iter current;

    // ...
};

(1) seems a bit reckless to me. (2) seems too constraining to me - I'm sure a lot of older code in the wild is still using std::iterator in ways that are not ABI sensitive, and it would be silly to break their code for that.

I think I'd go with (3) if we all agree it solves the ABI issue, since it's not such a big deal for us to do. In the future, I'd consider breaking the ABI of reverse_iterator and removing both this workaround and the __t workaround at once.

I'll grumblingly go along with (3), but if we're going to consider breaking iterator adaptor ABI at "some point" in the future, why not just rip the band-aid off here and now? It'll immediately simplify the implementation, and I can't see how it would be any less reckless to make this decision in a year's time as opposed to today.

Your timeline is far out enough that it stalls other work. @Mordante is blocked by this patch.
It's also a simple enough thing to resolve (once we've agreed on a direction) that handing it over seems pointless.

I'm not really blocked by this patch, I now have an ugly work-around in my format branch. So I could put that up for review and clean up later. And actually I'm not blocked by the base class but the fact the back_insert_iterator isn't default constructible and its difference_type is void.

(1) seems a bit reckless to me. (2) seems too constraining to me - I'm sure a lot of older code in the wild is still using std::iterator in ways that are not ABI sensitive, and it would be silly to break their code for that.

I think I'd go with (3) if we all agree it solves the ABI issue, since it's not such a big deal for us to do. In the future, I'd consider breaking the ABI of reverse_iterator and removing both this workaround and the __t workaround at once.

I'll grumblingly go along with (3), but if we're going to consider breaking iterator adaptor ABI at "some point" in the future, why not just rip the band-aid off here and now? It'll immediately simplify the implementation, and I can't see how it would be any less reckless to make this decision in a year's time as opposed to today.

I think it would be nice to consider making ABI v2 the default and move Unstable to v3. But I think this requires some planning and communicating to involved parties upfront. Maybe that becomes moot if https://wg21.link/p2123 "Extending the Type System to Provide API and ABI Flexibility" gets accepted before C++23.

cjdb abandoned this revision.May 26 2021, 8:49 AM