Details
- Reviewers
ldionne philnik var-const - Group Reviewers
Restricted Project - Commits
- rG0218ea4aaa54: [libc++] Implement ranges::ends_with
Diff Detail
- Repository
- rG LLVM Github Monorepo
Event Timeline
replace ranges::drop_view with ranges::subrange
libcxx/test/std/algorithms/alg.nonmodifying/alg.ends_with/ranges.ends_with.pass.cpp | ||
---|---|---|
219 | In line 252, forward_iterator_list is tested and include all input_iterators features. So input_iterators are tested? |
libcxx/include/__algorithm/ranges_ends_with.h | ||
---|---|---|
60 | Still seems not done? | |
63 | Wrapping them in std::ref will probably solve the issue. equal has a bunch of optimizations based on the iterator type and it's likely we're skipping some logic by bypassing both equal and __equal and going straight to __equal_impl. | |
82 | I would strongly advise to do so unless there's a compelling reason not to. It's better to avoid code duplication when possible in these cases. | |
libcxx/test/std/algorithms/alg.nonmodifying/alg.ends_with/ranges.ends_with.pass.cpp | ||
66 | I think these need to be reformatted (throughout the file). | |
77 | Seems not done? | |
155 | Looks like this test is duplicated (see line 139). | |
156 | Seems not done? | |
170 | How about: The suffix occurs several time within the string, but not at the end. ? | |
216 | Duplicated. | |
217 | How about: int a[] = {5, 1, 3, 2, 7}; int p[] = {-2, -7}; auto pred = [](int l, int r) { return l * -1 == r; }; I'm sorry for partially pushing my personal preference, but what I like about this is that a) it compares more than 1 element and b) the comparison is more specific (only one value would compare equal, whereas an infinite number of values would compare greater than). I also find that using the negative complement of a number is an easy transformation to read (no need to do any mental math). | |
219 | It's kind of the opposite -- for an algorithm that supports input iterators, we need to make sure we don't accidentally end up relying on some features that are only available on forward iterators (or only on bidirectional iterators, etc.). In other words, we're checking that the algorithm only uses the very minimalistic interface provided by an input_iterator and nothing else. If we only test with forward_iterators, for example, we wouldn't be able to catch if the algorithm presumes that calling i++ on an iterator i returns a value (an input iterator can return anything, including void). |
libcxx/test/std/algorithms/alg.nonmodifying/alg.ends_with/ranges.ends_with.pass.cpp | ||
---|---|---|
64 | We need to also add one test to check the return type, something like: { // Check the return type int a[] = {1, 2, 3, 4, 5, 6}; int p[] = {5, 6}; auto whole = std::ranges::subrange(Iter1(a), Sent1(Iter1(a + 6))); auto suffix = std::ranges::subrange(Iter2(p), Sent2(Iter2(p + 2))); { [[maybe_unused]] std::same_as<bool> decltype(auto) ret = std::ranges::ends_with(whole.begin(), whole.end(), suffix.begin(), suffix.end()); } { [[maybe_unused]] std::same_as<bool> decltype(auto) ret = std::ranges::ends_with(whole, suffix); } } |
Fix some nits:
- delete duplication
- call ranges::equal in the implementation
- use decltype
- add some tests
libcxx/test/std/algorithms/alg.nonmodifying/alg.ends_with/ranges.ends_with.pass.cpp | ||
---|---|---|
64 | okay I think the simple test is duplicated so I will make the simple test to check and use std::same_as<bool> |
- use ranges::advance() rather than std::advance()
- add two test cases: range consists of just one element and suffix consists of just one element
- remove the duplicated test case about predicate
libcxx/include/__algorithm/ranges_ends_with.h | ||
---|---|---|
82 | okay! I add a new helper function like ranges_mismatch.h to simplify code! |
libcxx/include/__algorithm/ranges_ends_with.h | ||
---|---|---|
44 | This helper function doesn't need default arguments (since it's never invoked outside this file) and can take the predicate and the projections by reference (so there's no need to wrap the arguments passed to it in std::ref -- std::ref is only needed when passing those arguments to ranges::equal). | |
60 | Friendly ping on this -- this sounds like a good optimization to implement. |
- Test input_iterators
- Update help function (will try to optimize later)
- some nits fixed
libcxx/include/__algorithm/ranges_ends_with.h | ||
---|---|---|
44 | Wrapped by std::ref only when passing args to ranges::equal will cause ranges_robust_against_copying_comparators.pass.cpp fail. |
libcxx/include/__algorithm/ranges_ends_with.h | ||
---|---|---|
40 | The helper function should take _Pred and _Proj1/_Proj2 by a reference, which would avoid the need for std::ref when delegating. Unlike the public functions which must have the same signature as in the standard, we can use references in the internal helper. | |
44 | I mean that the helper function should take the arguments by reference, then wrap them in std::ref before passing to other algorithms. Basically, I'm suggesting to move the wrapping from the public function into the helper function. | |
48 | Note: we only qualify the namespace on function calls (to avoid triggering ADL). There is no need to qualify class names, concept names, etc. | |
50 | How about something like if constexpr (bidirectional_iterator<_Iter1> && bidirectional_iterator<_Iter2>) { auto __rbegin1 = std::make_reverse_iterator(__first1); auto __rbegin2 = std::make_reverse_iterator(__first2); auto __rend1 = std::make_reverse_iterator(ranges::next(__first1, __last1)); auto __rend2 = std::make_reverse_iterator(ranges::next(__first2, __last2)); return ranges::starts_with(__rbegin1, __rend1, __rbegin2, __rend2, std::ref(__pred), std::ref(__proj1), std::ref(__proj2)); } ? | |
65 | Why do we have to unwrap here? Can't we delegate that to equal? | |
72 | std::ref should be here. | |
114 | I think we're missing the case where the ranges are sized (meaning we can get the size in constant time) but the sentinels aren't, and the iterators aren't random-access. In other words, we miss the case when we have a way to get the sizes in constant time while we still have the range objects, but once we reduce them to iterators, getting the size becomes linear. I took a stab at it and can suggest something like the following. There are many potential microoptimizations here (in particular if we consider all the cases where one of the ranges is sized and the other is not), but personally I think we need to strike the right balance between performance and code clarity. My current thinking is:
Pseudocode (I omitted the predicate and projections, as well as function attributes, for simplicity): template <class _Iter1, class _Iter2> bool __impl_bidirectional(_Iter1 __first1, _Iter1 __last1, _Iter2 __first2, _Iter2 __last2) { auto __rbegin1 = std::make_reverse_iterator(__first1); auto __rend1 = std::make_reverse_iterator(__last1); auto __rbegin2 = std::make_reverse_iterator(__first2); auto __rend2 = std::make_reverse_iterator(__last2); return ranges::starts_with(__rbegin1, __rend1, __rbegin2, __rend2); } template <class _Iter1, class _Sent1, class _Iter2, class _Sent2, class _Diff> bool __impl_with_offset(_Iter1 __first1, _Sent1 __last1, _Iter2 __first2, _Sent2 __last2, _Diff __offset) { ranges::advance(__first1, __offset); return ranges::equal(__first1, __last1, __first2, __last2); } template <class _Iter1, class _Sent1, class _Iter2, class _Sent2> bool __impl_with_sentinels(_Iter1 __first1, _Sent1 __last1, _Iter2 __first2, _Sent2 __last2) { if constexpr (bidirectional_iterator<_Sent1> && bidirectional_iterator<_Sent2>) { return __impl_bidirectional(__first1, __last1, __first2, __last2); } else { auto __n1 = ranges::distance(__first1, __last1); auto __n2 = ranges::distance(__first2, __last2); if (__n2 == 0) { return true; } if (__n2 > __n1) { // Also covers the case when `__n1` is `0`. return false; } return __impl_with_offset(__first1, __last1, __first2, __last2, __n1 - __n2); } // Public function that takes iterators { return __impl_with_sentinels(__first1, __last1, __first2, __last2); } // Public function that takes ranges { if constexpr (sized_range<_Range1> && sized_range<_Range2>) { auto __n1 = ranges::distance(__range1); auto __n2 = ranges::distance(__range2); if (__n2 == 0) { return true; } if (__n2 > __n1) { // Also covers the case when `__n1` is `0`. return false; } return __impl_with_offset(ranges::begin(__range1), ranges::end(__range1), ranges::begin(__range2), ranges::end(__range2), __n1 - __n2); } else { return __impl_with_sentinels(ranges::begin(__range1), ranges::end(__range1), ranges::begin(__range2), ranges::end(__range2)); } } | |
libcxx/test/std/algorithms/alg.nonmodifying/alg.ends_with/ranges.ends_with.pass.cpp | ||
256 | You might want to add a comment to this branch and the next one that this is to test the (forward_iterator<_Iter1> || sized_sentinel_for<_Sent1, _Iter1>) condition. |
- If we can get the size of input directly, get it, then compare the sizes and return the result in some cases. Otherwise, go to step2.
- If the input is bidirectional, go to __bidirectional_impl() directly. If not, go to step 3
- Call ranges::equal() to get the result
libcxx/include/__algorithm/ranges_ends_with.h | ||
---|---|---|
72 | yes, I tried to pass std::ref in ranges::equal() and ranges::start() only and copy tests failed. And tries to remove std::ref when passing to __ends_with_fn_impl() and still failed the tests. |
add a comment to make code more clear
libcxx/include/__algorithm/ranges_ends_with.h | ||
---|---|---|
48 | Do I need to add a comment here? |
Looks pretty good, but I'd like to get the ranges::equal optimization in again, since it's probably quite significant.
libcxx/include/__algorithm/ranges_ends_with.h | ||
---|---|---|
37 | It looks like we've dropped the ranges::equal optimization again? | |
65–83 | Just to make it a bit leaner to compile. | |
148–149 | To make it obvious that it's constant time and avoid a few instantiations. | |
165–173 |
libcxx/include/__algorithm/ranges_ends_with.h | ||
---|---|---|
159 | I don't think these std::moves are needed since ranges::begin() already return prvalue. |
- use ranges::size() instead of ranges::distance to get the size of an array at constant time
- remove std::move for all input which call ranges algorithms.
Get ranges::equal optimization back when the iterator is random access iterator, and add a benchmark test to verify the optimization works.
libcxx/test/std/algorithms/alg.nonmodifying/alg.ends_with/ranges.ends_with.pass.cpp | ||
---|---|---|
275 | Instead, we should add a benchmark under <monorepo>/libcxx/benchmarks and post the results of that benchmark here in this review. I know this isn't an automated test, but the test as-is risks being flaky. Let's leave the benchmarks in libcxx/benchmarks and we can figure out a way to track performance through time (and potentially compare performance of different operations?) more generally. |
Thanks a lot, this is getting very close to LGTM. The implementation looks good (just a few formatting nitpicks), just a few minor things in tests to address plus the benchmark needs to be reworked. My sincere apologies for the very slow review (mostly due to the upcoming LLVM 17 release).
libcxx/include/__algorithm/ranges_ends_with.h | ||
---|---|---|
40 | How about naming this function __ends_with_fn_impl_bidirectional, just for consistency with the other helpers? | |
68 | Nit: please add a blank line before this line. | |
74 | Nit: please add a blank line after this line. | |
105 | Nit: can you add spaces before and after the &&? Or just clang-format these lines. If it's clang-format that produces this formatting, then never mind (although I'd be surprised). | |
105 | Nit: the parentheses around !std::random_access_iterator<_Sent1> are unnecessary (same for the _Sent2 check). | |
106 | Nit: std::move the iterators here as well? | |
118 | Nit: please add a blank line after this line (to separate different functions from each other). | |
168 | Nit: please add a blank line before this line. | |
libcxx/test/std/algorithms/alg.nonmodifying/alg.ends_with/ranges.ends_with.pass.cpp | ||
43 | Nit: I think we need one successful test case for each number of template arguments, so here we need to add: static_assert(!HasEndsWithIt<int*, int*>); (this is to "test the test", so to speak) | |
52 | Nit: Range1 and Range2 should be forwarding references, and in the body of the concept, range1 and range2 should be forwarded. That way, it would be more similar to how the function is actually invoked. | |
59 | Here we also need to have a successful test case for two template arguments. | |
60 | Optional: IMO it's not necessary to test out every single not-forward-range we have, just a couple is enough (e.g. I would leave just ForwardRangeNotDerivedFrom and ForwardRangeNotSentinelSemiregular). | |
235 | Nit: formatting needs fixing. | |
256 | I think this should be random_access_iterator list, otherwise none of the iterator types in the list are a forward iterator so the if constexpr (std::forward_iterator...) below will never be taken. | |
275 | Nit: s/benmark/benchmark/ (here and below). | |
275 | +1, benchmarks should be under the benchmarks folder and use the infrastructure we're currently using (that is, Google Benchmark). Also, we shouldn't be benchmarking the implementation details like this, partially because it's easy for the benchmark and the actual implementation to get out of sync. Rather, you can use the same input vectors for different test cases but wrap their iterators into our helper iterators, then call ends_with using the wrappers: std::vector<int> a = /*some initializer*/; std::vector<int> p = /*some initializer*/; { std::random_access_iterator<int*> begin1(a.begin()); std::random_access_iterator<int*> end1(a.end()); std::random_access_iterator<int*> begin2(p.begin()); std::random_access_iterator<int*> end2(p.end()); std::ranges::ends_with(begin1, end1, begin2, end2); // ... } { std::bidirectional_iterator<int*> begin1(a.begin()); std::bidirectional_iterator <int*> end1(a.end()); std::bidirectional_iterator <int*> begin2(p.begin()); std::bidirectional_iterator <int*> end2(p.end()); std::ranges::ends_with(begin1, end1, begin2, end2); // ... } You can even test the sized optimization using sized_sentinel. Like Louis mentioned above, we unfortunately don't have a way to test for performance regressions currently, and we shouldn't try to solve that problem in this patch. Rather, it would be great if you could run the new benchmarks locally, make sure the numbers are as expected (i.e., random access iterators are faster than bidirectional iterators, etc.), then post the results in this review. | |
276 | This should be just std::vector. | |
278 | Nit: formatting (should be a space after for, i.e., for (). | |
284 | Names used in tests don't have to be uglified with leading __ underscores (in fact, they shouldn't be since these are technically reserved names for the library. Even though the test is testing the library, the test itself isn't a part of the library). | |
289 | Since these are the default values, I think they should be omitted (the predicate and the projections, that is). |
libcxx/test/std/algorithms/alg.nonmodifying/alg.ends_with/ranges.ends_with.pass.cpp | ||
---|---|---|
256 | I think cpp20_input_iterator_list already contains random_access_iterator? using random_access_iterator_list = type_list<Ptr, #if TEST_STD_VER >= 20 contiguous_iterator<Ptr>, #endif random_access_iterator<Ptr> >; template <class Ptr> using bidirectional_iterator_list = concatenate_t<random_access_iterator_list<Ptr>, type_list<bidirectional_iterator<Ptr> > >; template <class Ptr> using forward_iterator_list = concatenate_t<bidirectional_iterator_list<Ptr>, type_list<forward_iterator<Ptr> > >; template <class Ptr> using cpp17_input_iterator_list = concatenate_t<forward_iterator_list<Ptr>, type_list<cpp17_input_iterator<Ptr> > >; #if TEST_STD_VER >= 20 template <class Ptr> using cpp20_input_iterator_list = concatenate_t<forward_iterator_list<Ptr>, type_list<cpp20_input_iterator<Ptr>, cpp17_input_iterator<Ptr>>>; #endif |
libcxx/test/std/algorithms/alg.nonmodifying/alg.ends_with/ranges.ends_with.pass.cpp | ||
---|---|---|
53 | Nit: you would also need to call std::forward<Range1>(range1) here (same for the other argument). | |
275 | Sorry, rereading this comment, I mean random_access_iterator and bidirectional_iterator from our test_iterators.h header. Please ignore the std:: qualification, I copy-pasted it accidentally. So basically std::vector<int> a = /*some initializer*/; std::vector<int> p = /*some initializer*/; { random_access_iterator<int*> begin1(a.begin()); random_access_iterator<int*> end1(a.end()); random_access_iterator<int*> begin2(p.begin()); random_access_iterator<int*> end2(p.end()); std::ranges::ends_with(begin1, end1, begin2, end2); // ... } { bidirectional_iterator<int*> begin1(a.begin()); bidirectional_iterator<int*> end1(a.end()); bidirectional_iterator<int*> begin2(p.begin()); bidirectional_iterator<int*> end2(p.end()); std::ranges::ends_with(begin1, end1, begin2, end2); // ... } |
libcxx/benchmarks/algorithms/ranges_ends_with.bench.cpp | ||
---|---|---|
15 | I tried out the benchmarks locally. After applying the fixes outlined above, I noticed a couple of interesting things:
__is_trivial_equality_predicate<_Pred, _Tp, _Up>::value && __is_identity<_Proj1>::value && __is_identity<_Proj2>::value __is_trivial_equality_predicate recognizes ranges::equal_to as trivial, but not reference_wrapper<ranges::equal_to>, similarly for __is_identity. There are a few potential solutions:
I'm currently leaning towards option 1. @philnik Nikolas, what do you think? | |
22 | Sorry, I miswrote the suggestion above, this should be #include "test_iterators.h" // ... auto begin1 = random_access_iterator(a.begin()); // ... std::random_access_iterator is a concept and as such it evaluates to a boolean. In other words, begin1, end1, begin2 and end2 end up being boolean variables, then we create zero- or one-element containers from those booleans (because they implicitly convert to integers, unfortunately), which means we ignore the size of state.range() generated by benchmark and always measure the same (meaningless) thing. | |
26 | What we want to do is to test how the implementation of the algorithm treats different iterator categories, but at the same time we want to use the same underlying container type so that it's not a factor in our measurement (otherwise, if we compare e.g. a vector against a list, it's impossible to say how much of the performance difference is due to random-access iterator optimizations in ends_with and how much is due to the fact that vector is much more cache-friendly). Since we can make the iterator category more limited (e.g. turn a bidirectional iterator into a forward iterator) but not vice versa, that means we need to use a container with the most powerful iterator category -- so it would have to be a vector, an array or a built-in array. If we use a container, wrap its iterators but then create a new container from those iterators, it negates the idea because the new container would have the same iterator category as before. E.g. if we create a list, wrap its (bidirectional) iterators into forward iterators, then create a new list from those forward iterators, the new list once again would have bidirectional iterators, completely negating our wrapping. In other words, you should just do benchmark::DoNotOptimize(std::ranges::ends_with(begin1, end1, begin2, end2)); | |
40 | We would also want to test with contiguous_iterators and forward_iterators. |
libcxx/benchmarks/algorithms/ranges_ends_with.bench.cpp | ||
---|---|---|
15 | I think both (1) and (3) make sense. (1) Seems like a nice generalizations, but I doubt it will make much of a difference 99% of the time, and (3) seems like it simplifies some things quite a bit. I also have the suspicion that all the references in the algorithms might inhibit optimizations, but I didn't test that yet. I also don't expect most predicates and projections to be non-trivial, making this an optimization that only applies very rarely, and potentially a pessimization in some cases (although that's just conjecture on my part). |
Thank you for working on this and addressing all the feedback! LGTM with a green CI (let me know if you need any help with CI failures).
I don't know if you have commit permissions -- if not, I can help merge the patch (please provide the name and the email address to use for the author field in that case).
libcxx/benchmarks/algorithms/ranges_ends_with.bench.cpp | ||
---|---|---|
15 | @philnik Thanks! @ZijunZhao As discussed offline, let's implement (1) in a follow-up. | |
15 | Nit: I don't think this header is used. | |
43 | Nit: please add a blank line after this line. | |
68 | Nit: extraneous blank line (here and in some other functions in this file). | |
95 | Nit: please add a blank line after this line for consistency with other functions in this file. |
- add some headers in ranges_ends_with.h
- fix nits: add/remove some blank lines
- remove useless headers in ranges.ends_with.pass.cpp
I tried out the benchmarks locally. After applying the fixes outlined above, I noticed a couple of interesting things:
__is_trivial_equality_predicate recognizes ranges::equal_to as trivial, but not reference_wrapper<ranges::equal_to>, similarly for __is_identity.
There are a few potential solutions:
I'm currently leaning towards option 1. @philnik Nikolas, what do you think?