[libc++][spaceship] Additional tests for `operator<=>` `map` and `multimap`

[H-G-Hristov]

Summary

• Added additional tests
• Improved existing tests
• Moved misplaced test files to the correct location

[H-G-Hristov]

Try to fix CI

[H-G-Hristov]

Try to fix CI: Ignore expected-error message with {{}} syntax due to differing error message locally vs remotely (on CI).

[philnik]

In D151205#4364787, @H-G-Hristov wrote:

Try to fix CI: Ignore expected-error message with {{}} syntax due to differing error message locally vs remotely (on CI).

You probably want to update your local compiler then. We only support Clang 15, 16 and trunk.

libcxx/test/std/containers/associative/map/map.nonmember/compare.three_way.verify.cpp

22

This include doesn't seem to be used,

29

IMO it would be better to have two separate // expected-errors to make sure they are actually generated by the two lines below.

[H-G-Hristov]

Minor tweaks

[H-G-Hristov]

libcxx/test/std/containers/associative/map/map.nonmember/compare.three_way.verify.cpp

22

It is used for the last test.

29

Thanks! That makes sense. I'll update my tests.

[H-G-Hristov]

In D151205#4365640, @philnik wrote:

In D151205#4364787, @H-G-Hristov wrote:

Try to fix CI: Ignore expected-error message with {{}} syntax due to differing error message locally vs remotely (on CI).

You probably want to update your local compiler then. We only support Clang 15, 16 and trunk.

My local compiler is Apple Clang 14.0.3 which is based on Clang 15, which is the latest from Apple.

[philnik]

In D151205#4365999, @H-G-Hristov wrote:

In D151205#4365640, @philnik wrote:

In D151205#4364787, @H-G-Hristov wrote:

Try to fix CI: Ignore expected-error message with {{}} syntax due to differing error message locally vs remotely (on CI).

You probably want to update your local compiler then. We only support Clang 15, 16 and trunk.

My local compiler is Apple Clang 14.0.3 which is based on Clang 15, which is the latest from Apple.

What is the difference in error messages then? Can you check the actual message with a regex maybe? (// expected-error-re {{... {{regex}} ...}})

[H-G-Hristov]

In D151205#4366003, @philnik wrote:

In D151205#4365999, @H-G-Hristov wrote:

In D151205#4365640, @philnik wrote:

In D151205#4364787, @H-G-Hristov wrote:

Try to fix CI: Ignore expected-error message with {{}} syntax due to differing error message locally vs remotely (on CI).

You probably want to update your local compiler then. We only support Clang 15, 16 and trunk.

My local compiler is Apple Clang 14.0.3 which is based on Clang 15, which is the latest from Apple.

What is the difference in error messages then? Can you check the actual message with a regex maybe? (// expected-error-re {{... {{regex}} ...}})

I just updated this test with the `//expected-error-re {{... {{regex}} }} syntax now. Apple Clang outputs "static_assert" vs "static assertion" on the CI. I've seen a patch that you want to remove this syntax. Then I guess may need to revert this change and figure out how to setup a non-Apple compiler.

[philnik]

In D151205#4366016, @H-G-Hristov wrote:

I just updated this test with the `//expected-error-re {{... {{regex}} }} syntax now. Apple Clang outputs "static_assert" vs "static assertion" on the CI. I've seen a patch that you want to remove this syntax. Then I guess may need to revert this change and figure out how to setup a non-Apple compiler.

I thought AppleClang also outputs static assertion. If that's not the case yet, I guess I have to wait until we update it.

[H-G-Hristov]

Addressed review comments

[H-G-Hristov]

@philnik Thank you for the review.

libcxx/test/std/containers/associative/map/map.nonmember/compare.three_way.verify.cpp

22

It is used for the last test.

Moved the test with the allocator to the top.

[H-G-Hristov]

Rebased + fixed merge conflicts

[H-G-Hristov]

Rebased to fix CI

[H-G-Hristov]

@Mordante FYI this patch adds the additional tests to map you requested.

libcxx/test/std/containers/associative/map/map.nonmember/compare.three_way.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 // UNSUPPORTED: c++03, c++11, c++14, c++17
 
 // <map>
 
 // class map
 
 // template<class Key, class T, class Compare, class Allocator>
 //   synth-three-way-result<pair<const Key, T>>
 //     operator<=>(const map<Key, T, Compare, Allocator>& x,
 //                 const map<Key, T, Compare, Allocator>& y);
 
 #include <cassert>
 #include <map>
 
 #include "test_container_comparisons.h"
 
 int main(int, char**) {
   assert(test_ordered_map_container_spaceship<std::map>());
   // `std::map` is not constexpr, so no `static_assert` test here.
   return 0;
 }

libcxx/test/std/containers/associative/map/map.nonmember/compare.three_way.verify.cpp

+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+// UNSUPPORTED: c++03, c++11, c++14, c++17
+
+// <map>
+
+// class map
+
+// template<class Key, class T, class Compare, class Allocator>
+//   synth-three-way-result<pair<const Key, T>>
+//     operator<=>(const map<Key, T, Compare, Allocator>& x,
+//                 const map<Key, T, Compare, Allocator>& y);
+
+#include <map>
+
+#include "test_allocator.h"
+
+int main(int, char**) {
+  // Mismatching allocators
+  {
+    std::map<int, int, std::less<int>, std::allocator<int>> s1;
+    std::map<int, int, std::less<int>, test_allocator<int>> s2;
+    // expected-error-re@*:* {{{{(static_assert|static assertion)}} failed due to requirement 'is_same<int, std::pair<const int, int>>::value'{{.*}}Allocator::value_type must be same type as value_type}}
+    s1 <=> s2;
+    // expected-error-re@*:* {{{{(static_assert|static assertion)}} failed due to requirement 'is_same<int, std::pair<const int, int>>::value'{{.*}}Allocator::value_type must be same type as value_type}}
+    s2 <=> s1;
+  }
+  // Mismatching comparision functions
+  {
+    std::map<int, int, std::less<int>> s1;
+    std::map<int, int, std::greater<int>> s2;
+    // expected-error@+1 {{invalid operands to binary expression}}
+    s1 <=> s2;
+    // expected-error@+1 {{invalid operands to binary expression}}
+    s2 <=> s1;
+  }
+  {
+    std::map<int, int, std::less<int>> s1;
+    std::map<int, int, std::less<float>> s2;
+    // expected-error@+1 {{invalid operands to binary expression}}
+    s1 <=> s2;
+    // expected-error@+1 {{invalid operands to binary expression}}
+    s2 <=> s1;
+  }
+  // Mismatching types
+  {
+    std::map<int, int> s1;
+    std::map<int, float> s2;
+    // expected-error@+1 {{invalid operands to binary expression}}
+    s1 <=> s2;
+    // expected-error@+1 {{invalid operands to binary expression}}
+    s2 <=> s1;
+  }
+
+  return 0;
+}

libcxx/test/std/containers/associative/multimap/multimap.nonmember/compare.three_way.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 // UNSUPPORTED: c++03, c++11, c++14, c++17
 
 // <map>
 
 // class multimap
 
 // template<class Key, class T, class Compare, class Allocator>
 //   synth-three-way-result<pair<const Key, T>>
 //     operator<=>(const multimap<Key, T, Compare, Allocator>& x,
 //                 const multimap<Key, T, Compare, Allocator>& y);
 
 #include <cassert>
 #include <map>
 
 #include "test_container_comparisons.h"
 
 int main(int, char**) {
   assert(test_ordered_map_container_spaceship<std::multimap>());
   // `std::multimap` is not constexpr, so no `static_assert` test here.
   return 0;
 }

libcxx/test/std/containers/associative/multimap/multimap.nonmember/compare.three_way.verify.cpp

+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+// UNSUPPORTED: c++03, c++11, c++14, c++17
+
+// <map>
+
+// class multimap
+
+// template<class Key, class T, class Compare, class Allocator>
+//   synth-three-way-result<pair<const Key, T>>
+//     operator<=>(const multimap<Key, T, Compare, Allocator>& x,
+//                 const multimap<Key, T, Compare, Allocator>& y);
+
+#include <map>
+
+#include "test_allocator.h"
+
+int main(int, char**) {
+  // Mismatching allocators
+  {
+    std::multimap<int, int, std::less<int>, std::allocator<int>> s1;
+    std::multimap<int, int, std::less<int>, test_allocator<int>> s2;
+    // expected-error-re@*:* {{{{(static_assert|static assertion)}} failed due to requirement 'is_same<int, std::pair<const int, int>>::value'{{.*}}Allocator::value_type must be same type as value_type}}
+    s1 <=> s2;
+    // expected-error-re@*:* {{{{(static_assert|static assertion)}} failed due to requirement 'is_same<int, std::pair<const int, int>>::value'{{.*}}Allocator::value_type must be same type as value_type}}
+    s2 <=> s1;
+  }
+  // Mismatching comparision functions
+  {
+    std::multimap<int, int, std::less<int>> s1;
+    std::multimap<int, int, std::greater<int>> s2;
+    // expected-error@+1 {{invalid operands to binary expression}}
+    s1 <=> s2;
+    // expected-error@+1 {{invalid operands to binary expression}}
+    s2 <=> s1;
+  }
+  {
+    std::multimap<int, int, std::less<int>> s1;
+    std::multimap<int, int, std::less<float>> s2;
+    // expected-error@+1 {{invalid operands to binary expression}}
+    s1 <=> s2;
+    // expected-error@+1 {{invalid operands to binary expression}}
+    s2 <=> s1;
+  }
+  // Mismatching types
+  {
+    std::multimap<int, int> s1;
+    std::multimap<int, float> s2;
+    // expected-error@+1 {{invalid operands to binary expression}}
+    s1 <=> s2;
+    // expected-error@+1 {{invalid operands to binary expression}}
+    s2 <=> s1;
+  }
+
+  return 0;
+}

libcxx/test/support/test_container_comparisons.h

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //  Utility functions to test comparisons on containers.
 
 #ifndef TEST_CONTAINER_COMPARISONS
 #define TEST_CONTAINER_COMPARISONS
 
+#include <functional>
 #include <set>
 
 #include "test_comparisons.h"
 
 // Implementation detail of `test_sequence_container_spaceship`
 template <template <typename...> typename Container, typename Elem, typename Order>
 constexpr void test_sequence_container_spaceship_with_type() {
   // Empty containers
constexpr bool test_sequence_container_spaceship() {
   // Thanks to SFINAE, the following is not a compiler error but returns `false`
   struct NonComparable {};
   static_assert(!std::three_way_comparable<Container<NonComparable>>);
 
   return true;
 }
 
 // Implementation detail of `test_ordered_map_container_spaceship`
-template <template <typename...> typename Container, typename Key, typename Val, typename Order>
-constexpr void test_ordered_map_container_spaceship_with_type() {
+template <template <typename...> typename Container, typename Key, typename Val, typename Order, typename Compare>
+constexpr void test_ordered_map_container_spaceship_with_type(Compare comp) {
   // Empty containers
   {
-    Container<Key, Val> l1;
-    Container<Key, Val> l2;
+    Container<Key, Val, Compare> l1{{}, comp};
+    Container<Key, Val, Compare> l2{{}, comp};
     assert(testOrder(l1, l2, Order::equivalent));
   }
   // Identical contents
   {
-    Container<Key, Val> l1{{1, 1}, {2, 1}};
-    Container<Key, Val> l2{{1, 1}, {2, 1}};
+    Container<Key, Val, Compare> l1{{{1, 1}, {2, 1}}, comp};
+    Container<Key, Val, Compare> l2{{{1, 1}, {2, 1}}, comp};
     assert(testOrder(l1, l2, Order::equivalent));
   }
   // Less, due to contained values
   {
-    Container<Key, Val> l1{{1, 1}, {2, 1}};
-    Container<Key, Val> l2{{1, 1}, {2, 2}};
+    Container<Key, Val, Compare> l1{{{1, 1}, {2, 1}}, comp};
+    Container<Key, Val, Compare> l2{{{1, 1}, {2, 2}}, comp};
     assert(testOrder(l1, l2, Order::less));
   }
   // Greater, due to contained values
   {
-    Container<Key, Val> l1{{1, 1}, {2, 3}};
-    Container<Key, Val> l2{{1, 1}, {2, 2}};
+    Container<Key, Val, Compare> l1{{{1, 1}, {2, 3}}, comp};
+    Container<Key, Val, Compare> l2{{{1, 1}, {2, 2}}, comp};
     assert(testOrder(l1, l2, Order::greater));
   }
   // Shorter list
   {
-    Container<Key, Val> l1{{1, 1}};
-    Container<Key, Val> l2{{1, 1}, {2, 2}};
+    Container<Key, Val, Compare> l1{{{1, 1}}, comp};
+    Container<Key, Val, Compare> l2{{{1, 1}, {2, 2}}, comp};
     assert(testOrder(l1, l2, Order::less));
   }
   // Longer list
   {
-    Container<Key, Val> l1{{1, 2}, {2, 2}};
-    Container<Key, Val> l2{{1, 1}};
+    Container<Key, Val, Compare> l1{{{1, 2}, {2, 2}}, comp};
+    Container<Key, Val, Compare> l2{{{1, 1}}, comp};
     assert(testOrder(l1, l2, Order::greater));
   }
   // Unordered
   if constexpr (std::is_same_v<Val, PartialOrder>) {
-    Container<Key, Val> l1{{1, 1}, {2, std::numeric_limits<int>::min()}};
-    Container<Key, Val> l2{{1, 1}, {2, 2}};
+    Container<Key, Val, Compare> l1{{{1, 1}, {2, std::numeric_limits<int>::min()}}, comp};
+    Container<Key, Val, Compare> l2{{{1, 1}, {2, 2}}, comp};
     assert(testOrder(l1, l2, Order::unordered));
   }
 
   // Identical contents
   {
-    Container<Key, Val> l1{{1, 1}, {2, 1}, {2, 2}};
-    Container<Key, Val> l2{{1, 1}, {2, 1}, {2, 2}};
+    Container<Key, Val, Compare> l1{{{1, 1}, {2, 1}, {2, 2}}, comp};
+    Container<Key, Val, Compare> l2{{{1, 1}, {2, 1}, {2, 2}}, comp};
     assert(testOrder(l1, l2, Order::equivalent));
-    Container<Key, Val> l3{{1, 1}, {2, 1}, {2, 2}};
-    Container<Key, Val> l4{{2, 1}, {2, 2}, {1, 1}};
+
+    Container<Key, Val, Compare> l3{{{1, 1}, {2, 1}, {2, 2}}, comp};
+    Container<Key, Val, Compare> l4{{{2, 1}, {2, 2}, {1, 1}}, comp};
     assert(testOrder(l3, l4, Order::equivalent));
   }
   // Less, due to contained values
   {
-    Container<Key, Val> l1{{1, 1}, {2, 1}, {2, 1}};
-    Container<Key, Val> l2{{1, 1}, {2, 2}, {2, 2}};
+    Container<Key, Val, Compare> l1{{{1, 1}, {2, 1}, {2, 1}}, comp};
+    Container<Key, Val, Compare> l2{{{1, 1}, {2, 2}, {2, 2}}, comp};
     assert(testOrder(l1, l2, Order::less));
-    Container<Key, Val> l3{{1, 1}, {2, 1}, {2, 1}};
-    Container<Key, Val> l4{{2, 2}, {2, 2}, {1, 1}};
+
+    Container<Key, Val, Compare> l3{{{1, 1}, {2, 1}, {2, 1}}, comp};
+    Container<Key, Val, Compare> l4{{{2, 2}, {2, 2}, {1, 1}}, comp};
     assert(testOrder(l3, l4, Order::less));
   }
   // Greater, due to contained values
   {
-    Container<Key, Val> l1{{1, 1}, {2, 3}, {2, 3}};
-    Container<Key, Val> l2{{1, 1}, {2, 2}, {2, 2}};
+    Container<Key, Val, Compare> l1{{{1, 1}, {2, 3}, {2, 3}}, comp};
+    Container<Key, Val, Compare> l2{{{1, 1}, {2, 2}, {2, 2}}, comp};
     assert(testOrder(l1, l2, Order::greater));
-    Container<Key, Val> l3{{1, 1}, {2, 3}, {2, 3}};
-    Container<Key, Val> l4{{2, 2}, {2, 2}, {1, 1}};
+
+    Container<Key, Val, Compare> l3{{{1, 1}, {2, 3}, {2, 3}}, comp};
+    Container<Key, Val, Compare> l4{{{2, 2}, {2, 2}, {1, 1}}, comp};
     assert(testOrder(l3, l4, Order::greater));
   }
   // Shorter list
   {
-    Container<Key, Val> l1{{1, 1}, {2, 2}};
-    Container<Key, Val> l2{{1, 1}, {2, 2}, {2, 2}, {3, 1}};
+    Container<Key, Val, Compare> l1{{{1, 1}, {2, 2}}, comp};
+    Container<Key, Val, Compare> l2{{{1, 1}, {2, 2}, {2, 2}, {3, 1}}, comp};
     assert(testOrder(l1, l2, Order::less));
-    Container<Key, Val> l3{{1, 1}, {2, 2}};
-    Container<Key, Val> l4{{3, 1}, {2, 2}, {2, 2}, {1, 1}};
+
+    Container<Key, Val, Compare> l3{{{1, 1}, {2, 2}}, comp};
+    Container<Key, Val, Compare> l4{{{3, 1}, {2, 2}, {2, 2}, {1, 1}}, comp};
     assert(testOrder(l3, l4, Order::less));
   }
   // Longer list
   {
-    Container<Key, Val> l1{{1, 2}, {2, 2}, {2, 2}, {3, 1}};
-    Container<Key, Val> l2{{1, 1}, {2, 2}};
+    Container<Key, Val, Compare> l1{{{1, 2}, {2, 2}, {2, 2}, {3, 1}}, comp};
+    Container<Key, Val, Compare> l2{{{1, 1}, {2, 2}}, comp};
     assert(testOrder(l1, l2, Order::greater));
-    Container<Key, Val> l3{{1, 2}, {2, 2}, {2, 2}, {3, 1}};
-    Container<Key, Val> l4{{2, 2}, {1, 1}};
+
+    Container<Key, Val, Compare> l3{{{1, 2}, {2, 2}, {2, 2}, {3, 1}}, comp};
+    Container<Key, Val, Compare> l4{{{2, 2}, {1, 1}}, comp};
     assert(testOrder(l3, l4, Order::greater));
   }
   // Unordered
   if constexpr (std::is_same_v<Val, PartialOrder>) {
-    Container<Key, Val> l1{{1, 1}, {2, std::numeric_limits<int>::min()}, {2, 3}};
-    Container<Key, Val> l2{{1, 1}, {2, 2}, {2, 3}};
+    Container<Key, Val, Compare> l1{{{1, 1}, {2, std::numeric_limits<int>::min()}, {2, 3}}, comp};
+    Container<Key, Val, Compare> l2{{{1, 1}, {2, 2}, {2, 3}}, comp};
     assert(testOrder(l1, l2, Order::unordered));
-    Container<Key, Val> l3{{1, 1}, {2, std::numeric_limits<int>::min()}, {2, 3}};
-    Container<Key, Val> l4{{2, 3}, {2, 2}, {1, 1}};
+
+    Container<Key, Val, Compare> l3{{{1, 1}, {2, std::numeric_limits<int>::min()}, {2, 3}}, comp};
+    Container<Key, Val, Compare> l4{{{2, 3}, {2, 2}, {1, 1}}, comp};
     assert(testOrder(l3, l4, Order::unordered));
   }
 }
 
 // Tests the `operator<=>` on ordered map containers
 template <template <typename...> typename Container>
 constexpr bool test_ordered_map_container_spaceship() {
+  // Thanks to SFINAE, the following is not a compiler error but returns `false`
+  struct NonComparable {};
+  static_assert(!std::three_way_comparable<Container<int, NonComparable>>);
+
   // The container should fulfill `std::three_way_comparable`
   static_assert(std::three_way_comparable<Container<int, int>>);
 
   // Test different comparison categories
-  test_ordered_map_container_spaceship_with_type<Container, int, int, std::strong_ordering>();
-  test_ordered_map_container_spaceship_with_type<Container, int, StrongOrder, std::strong_ordering>();
-  test_ordered_map_container_spaceship_with_type<Container, int, WeakOrder, std::weak_ordering>();
-  test_ordered_map_container_spaceship_with_type<Container, int, PartialOrder, std::partial_ordering>();
+  test_ordered_map_container_spaceship_with_type<Container, int, int, std::strong_ordering>(std::less{});
+  test_ordered_map_container_spaceship_with_type<Container, int, int, std::strong_ordering>(std::greater{});
+  test_ordered_map_container_spaceship_with_type<Container, int, StrongOrder, std::strong_ordering>(std::less{});
+  test_ordered_map_container_spaceship_with_type<Container, int, StrongOrder, std::strong_ordering>(std::greater{});
+  test_ordered_map_container_spaceship_with_type<Container, int, WeakOrder, std::weak_ordering>(std::less{});
+  test_ordered_map_container_spaceship_with_type<Container, int, WeakOrder, std::weak_ordering>(std::greater{});
+  test_ordered_map_container_spaceship_with_type<Container, int, PartialOrder, std::partial_ordering>(std ::less{});
+  test_ordered_map_container_spaceship_with_type<Container, int, PartialOrder, std::partial_ordering>(std ::greater{});
 
   // `LessAndEqComp` does not have `operator<=>`. Ordering is synthesized based on `operator<`
-  test_ordered_map_container_spaceship_with_type<Container, int, LessAndEqComp, std::weak_ordering>();
-
-  // Thanks to SFINAE, the following is not a compiler error but returns `false`
-  struct NonComparable {};
-  static_assert(!std::three_way_comparable<Container<int, NonComparable>>);
+  test_ordered_map_container_spaceship_with_type<Container, int, LessAndEqComp, std::weak_ordering>(std::less{});
 
   return true;
 }
 
 // Implementation detail of `test_ordered_set_container_spaceship`
 template <template <typename...> typename Container, typename Elem, typename Order, typename Compare>
 constexpr void test_ordered_set_spaceship_with_type(Compare comp) {
   // Empty containers