[libc++][format] Adds new test macros.

[Mordante]

Summary

These macros make it easier to log additional information. This is
useful for formatting tests. It also properly disables additional
information when locales are disabled in libc++.

[ldionne]

I like where this is going -- we have a few places in the test suite where we want to have nice error messages but we don't have the machinery to. Centralizing this will also make it easier to have good error messages when the platform supports it, and no error message when it doesn't (e.g. when localization or IO are not supported).

libcxx/test/support/test_validation.h

1

I would call this assert_macros.h, I think. This would make a parallel with test_macros.h.

In fact it would be nice if those test macros could move to test_macros.h, but they can't because they have too many dependencies. I think this is something we should try to address separately.

34

41

51

If we dropped the free-form arguments from here and instead expected a string (or a C string), we could drop the dependency on iostream and make this much more widely useful in the test suite. It may also become useful for older tests. You'd probably want to define another helper function that will stringify your arguments and you could use that in your tests.

53

Here and elsewhere.

63

I would make those normal function templates instead. Otherwise one wonders whether there's a reason to define them this way.

84

88

All of these macros should fully qualify what they call.

91

I think I would name the macros this way:

TEST_FAIL(...)
TEST_ASSERT(...)
TEST_LIBCPP_ASSERT(...)

[Mordante]

libcxx/test/support/test_validation.h

51

As discussed I've done so. This still needs an additional include to "stringify" the messages.

[Mordante]

Rebased and addresses review comments.

[Mordante]

Forgot to save the last buffer which breaks the CI.

[Mordante]

Rebased to test CI.

[Mordante]

Fixes CI.

[ldionne]

libcxx/test/support/assert_macros.h

1 ↗

(On Diff #489038)

We have something different but similar in libcxx/test/support/rapid-cxx-test.h. Have you seen this before? I think it would be nice to unify both.

I am not a huge fan of having to use a macro for declaring test cases like we do in rapid-cxx-test.h, however we do already have macros with the same names. I guess my ask here would be to try and avoid introducing a third way of doing assertions in the test suite:

• standard <cassert>
• rapid-cxx-test.h
• now this header

I'd be OK with removing rapid-cxx-test if we can replace its uses by this, or I'd be fine with using rapid-cxx-test.h in the format tests instead. Sorry, I should have thought of rapid-cxx-test.h earlier.

[Mordante]

libcxx/test/support/assert_macros.h

1 ↗

(On Diff #489038)

As discussed I wasn't aware of rapid-cxx-test. I had a look and I don't see a way for adding custom messages, which is something I really like to use to make the format errors usable.

So my preference would be to remove rapid-cxx-test, if you agree I'll add it to my todo list and pick it up after landing the range patches for LLVM 16.

[ldionne]

libcxx/test/support/assert_macros.h

1 ↗

(On Diff #489038)

SGTM.

libcxx/test/std/utilities/format/format.functions/escaped_output.ascii.pass.cpp

 #include <cassert>
 #include <concepts>
 #include <list>
 #include <vector>
 
 #include "test_macros.h"
 #include "make_string.h"
 #include "test_format_string.h"
+#include "test_validation.h"
 
 #ifndef TEST_HAS_NO_LOCALIZATION
 #  include <iostream>
 #endif
 
 #define SV(S) MAKE_STRING_VIEW(CharT, S)
 
 auto test_format = []<class CharT, class... Args>(
                        std::basic_string_view<CharT> expected, test_format_string<CharT, Args...> fmt, Args&&... args) {
   {
     std::basic_string<CharT> out = std::format(fmt, std::forward<Args>(args)...);
-#ifndef TEST_HAS_NO_LOCALIZATION
-    if (out != expected) {
-      if constexpr (std::same_as<CharT, char>)
-        std::cerr << "\nFormat string   " << fmt.get() << "\nExpected output " << expected << "\nActual output   "
-                  << out << '\n';
-#  ifndef TEST_HAS_NO_WIDE_CHARACTERS
-      else
-        std::wcerr << L"\nFormat string   " << fmt.get() << L"\nExpected output " << expected << L"\nActual output   "
-                   << out << L'\n';
-#  endif // TEST_HAS_NO_WIDE_CHARACTERS
-    }
-#endif // TEST_HAS_NO_LOCALIZATION
-    assert(out == expected);
+    REQUIRE(out == expected,
+            "\nFormat string   ",
+            fmt.get(),
+            "\nExpected output ",
+            expected,
+            "\nActual output   ",
+            out,
+            '\n');
   }
 #ifndef TEST_HAS_NO_LOCALIZATION
   {
     std::basic_string<CharT> out = std::format(std::locale(), fmt, std::forward<Args>(args)...);
     assert(out == expected);
   }
 #endif // TEST_HAS_NO_LOCALIZATION
 };

libcxx/test/std/utilities/format/format.functions/escaped_output.unicode.pass.cpp

 #include <cassert>
 #include <concepts>
 #include <list>
 #include <vector>
 
 #include "test_macros.h"
 #include "make_string.h"
 #include "test_format_string.h"
+#include "test_validation.h"
 
 #ifndef TEST_HAS_NO_LOCALIZATION
 #  include <iostream>
 #endif
 
 #define SV(S) MAKE_STRING_VIEW(CharT, S)
 
 auto test_format = []<class CharT, class... Args>(
                        std::basic_string_view<CharT> expected, test_format_string<CharT, Args...> fmt, Args&&... args) {
   {
     std::basic_string<CharT> out = std::format(fmt, std::forward<Args>(args)...);
-#ifndef TEST_HAS_NO_LOCALIZATION
-    if (out != expected) {
-      if constexpr (std::same_as<CharT, char>)
-        std::cerr << "\nFormat string   " << fmt.get() << "\nExpected output " << expected << "\nActual output   "
-                  << out << '\n';
-#  ifndef TEST_HAS_NO_WIDE_CHARACTERS
-      else
-        std::wcerr << L"\nFormat string   " << fmt.get() << L"\nExpected output " << expected << L"\nActual output   "
-                   << out << L'\n';
-#  endif // TEST_HAS_NO_WIDE_CHARACTERS
-    }
-#endif // TEST_HAS_NO_LOCALIZATION
-    assert(out == expected);
+    REQUIRE(out == expected,
+            "\nFormat string   ",
+            fmt.get(),
+            "\nExpected output ",
+            expected,
+            "\nActual output   ",
+            out,
+            '\n');
   }
 #ifndef TEST_HAS_NO_LOCALIZATION
   {
     std::basic_string<CharT> out = std::format(std::locale(), fmt, std::forward<Args>(args)...);
     assert(out == expected);
   }
 #endif // TEST_HAS_NO_LOCALIZATION
 };

libcxx/test/std/utilities/format/format.functions/format.locale.pass.cpp

 #include <cassert>
 #include <iostream>
 #include <vector>
 
 #include "test_macros.h"
 #include "format_tests.h"
 #include "string_literal.h"
 #include "test_format_string.h"
+#include "test_validation.h"
 
 auto test =
     []<class CharT, class... Args>(
         std::basic_string_view<CharT> expected, test_format_string<CharT, Args...> fmt, Args&&... args) constexpr {
       std::basic_string<CharT> out = std::format(std::locale(), fmt, std::forward<Args>(args)...);
-      if constexpr (std::same_as<CharT, char>)
-        if (out != expected)
-          std::cerr << "\nFormat string   " << fmt.get() << "\nExpected output " << expected << "\nActual output   "
-                    << out << '\n';
-      assert(out == expected);
+      REQUIRE(out == expected,
+              "\nFormat string   ",
+              fmt.get(),
+              "\nExpected output ",
+              expected,
+              "\nActual output   ",
+              out,
+              '\n');
     };
 
 auto test_exception = []<class CharT, class... Args>(std::string_view, std::basic_string_view<CharT>, Args&&...) {
   // After P2216 most exceptions thrown by std::format become ill-formed.
   // Therefore this tests does nothing.
   // A basic ill-formed test is done in format.locale.verify.cpp
   // The exceptions are tested by other functions that don't use the basic-format-string as fmt argument.
 };

libcxx/test/std/utilities/format/format.functions/format.pass.cpp

 #include <format>
 #include <cassert>
 #include <vector>
 
 #include "test_macros.h"
 #include "format_tests.h"
 #include "string_literal.h"
 #include "test_format_string.h"
-
-#ifndef TEST_HAS_NO_LOCALIZATION
-#  include <iostream>
-#  include <type_traits>
-#endif
+#include "test_validation.h"
 
 auto test =
     []<class CharT, class... Args>(
         std::basic_string_view<CharT> expected, test_format_string<CharT, Args...> fmt, Args&&... args) constexpr {
       std::basic_string<CharT> out = std::format(fmt, std::forward<Args>(args)...);
-#ifndef TEST_HAS_NO_LOCALIZATION
-      if constexpr (std::same_as<CharT, char>)
-        if (out != expected)
-          std::cerr << "\nFormat string   " << fmt.get() << "\nExpected output " << expected << "\nActual output   "
-                    << out << '\n';
-#endif
-      assert(out == expected);
+      REQUIRE(out == expected,
+              "\nFormat string   ",
+              fmt.get(),
+              "\nExpected output ",
+              expected,
+              "\nActual output   ",
+              out,
+              '\n');
     };
 
 auto test_exception = []<class CharT, class... Args>(std::string_view, std::basic_string_view<CharT>, Args&&...) {
   // After P2216 most exceptions thrown by std::format become ill-formed.
   // Therefore this tests does nothing.
   // A basic ill-formed test is done in format.verify.cpp
   // The exceptions are tested by other functions that don't use the basic-format-string as fmt argument.
 };

libcxx/test/std/utilities/format/format.functions/locale-specific_form.pass.cpp

 #include <vector>
 
 #include "test_macros.h"
 #include "make_string.h"
 #include "platform_support.h" // locale name macros
 #include "format_tests.h"
 #include "string_literal.h"
 #include "test_format_string.h"
+#include "test_validation.h"
 
 #define STR(S) MAKE_STRING(CharT, S)
 #define SV(S) MAKE_STRING_VIEW(CharT, S)
 
 template <class CharT>
 struct numpunct;
 
 template <>

 };
 #endif
 
 template <class CharT, class... Args>
 void test(std::basic_string_view<CharT> expected, test_format_string<CharT, Args...> fmt, Args&&... args) {
   // *** format ***
   {
     std::basic_string<CharT> out = std::format(fmt, std::forward<Args>(args)...);
-    if constexpr (std::same_as<CharT, char>)
-      if (out != expected)
-        std::cerr << "\nFormat string   " << fmt.get() << "\nExpected output " << expected << "\nActual output   "
-                  << out << '\n';
-    assert(out == expected);
+    REQUIRE(out == expected,
+            "\nFormat string   ",
+            fmt.get(),
+            "\nExpected output ",
+            expected,
+            "\nActual output   ",
+            out,
+            '\n');
   }
   // *** vformat ***
   {
     std::basic_string<CharT> out = std::vformat(fmt.get(), std::make_format_args<context_t<CharT>>(args...));
     assert(out == expected);
   }
   // *** format_to ***
   {

libcxx/test/std/utilities/format/format.functions/vformat.locale.pass.cpp

 
 #include <format>
 #include <cassert>
 #include <vector>
 
 #include "test_macros.h"
 #include "format_tests.h"
 #include "string_literal.h"
+#include "test_validation.h"
 
 auto test = []<class CharT, class... Args>(
                 std::basic_string_view<CharT> expected, std::basic_string_view<CharT> fmt, Args&&... args) constexpr {
   std::basic_string<CharT> out = std::vformat(std::locale(), fmt, std::make_format_args<context_t<CharT>>(args...));
-  assert(out == expected);
+  REQUIRE(out == expected, "\nFormat string   ", fmt, "\nExpected output ", expected, "\nActual output   ", out, '\n');
 };
 
 auto test_exception =
     []<class CharT, class... Args>(
         [[maybe_unused]] std::string_view what,
         [[maybe_unused]] std::basic_string_view<CharT> fmt,
         [[maybe_unused]] Args&&... args) {
 #ifndef TEST_HAS_NO_EXCEPTIONS
       try {
-        (void)std::vformat(std::locale(), fmt, std::make_format_args<context_t<CharT>>(args...));
-        assert(false);
+        TEST_IGNORE_NODISCARD std::vformat(std::locale(), fmt, std::make_format_args<context_t<CharT>>(args...));
+        FAIL("\nFormat string   ", fmt, "\nDidn't throw an exception.\n");
       } catch ([[maybe_unused]] const std::format_error& e) {
-        LIBCPP_ASSERT(e.what() == what);
+        LIBCPP_REQUIRE(
+            e.what() == what,
+            "\nFormat string   ",
+            fmt,
+            "\nExpected exception ",
+            what,
+            "\nActual exception   ",
+            e.what(),
+            '\n');
         return;
       }
       assert(false);
 #endif
     };
 
 int main(int, char**) {
   format_tests<char, execution_modus::full>(test, test_exception);
 
 #ifndef TEST_HAS_NO_WIDE_CHARACTERS
   format_tests_char_to_wchar_t(test);
   format_tests<wchar_t, execution_modus::full>(test, test_exception);
 #endif
 
   return 0;
 }

libcxx/test/std/utilities/format/format.functions/vformat.pass.cpp

 
 #include <format>
 #include <cassert>
 #include <vector>
 
 #include "test_macros.h"
 #include "format_tests.h"
 #include "string_literal.h"
+#include "test_validation.h"
 
 auto test = []<class CharT, class... Args>(
                 std::basic_string_view<CharT> expected, std::basic_string_view<CharT> fmt, Args&&... args) constexpr {
   std::basic_string<CharT> out = std::vformat(fmt, std::make_format_args<context_t<CharT>>(args...));
-  assert(out == expected);
+  REQUIRE(out == expected, "\nFormat string   ", fmt, "\nExpected output ", expected, "\nActual output   ", out, '\n');
 };
 
 auto test_exception =
     []<class CharT, class... Args>(
         [[maybe_unused]] std::string_view what,
         [[maybe_unused]] std::basic_string_view<CharT> fmt,
         [[maybe_unused]] Args&&... args) {
 #ifndef TEST_HAS_NO_EXCEPTIONS
       try {
         TEST_IGNORE_NODISCARD std::vformat(fmt, std::make_format_args<context_t<CharT>>(args...));
-        assert(false);
+        FAIL("\nFormat string   ", fmt, "\nDidn't throw an exception.\n");
       } catch ([[maybe_unused]] const std::format_error& e) {
-        LIBCPP_ASSERT(e.what() == what);
+        LIBCPP_REQUIRE(
+            e.what() == what,
+            "\nFormat string   ",
+            fmt,
+            "\nExpected exception ",
+            what,
+            "\nActual exception   ",
+            e.what(),
+            '\n');
+
         return;
       }
       assert(false);
 #endif
     };
 
 int main(int, char**) {
   format_tests<char, execution_modus::full>(test, test_exception);
 
 #ifndef TEST_HAS_NO_WIDE_CHARACTERS
   format_tests_char_to_wchar_t(test);
   format_tests<wchar_t, execution_modus::full>(test, test_exception);
 #endif
 
   return 0;
 }

libcxx/test/std/utilities/format/format.tuple/format.functions.format.pass.cpp

 //   wstring format(wformat-string<Args...> fmt, const Args&... args);
 
 #include <format>
 #include <cassert>
 
 #include "format.functions.tests.h"
 #include "test_format_string.h"
 #include "test_macros.h"
-
-#ifndef TEST_HAS_NO_LOCALIZATION
-#  include <iostream>
-#  include <concepts>
-#endif
+#include "test_validation.h"
 
 auto test = []<class CharT, class... Args>(
                 std::basic_string_view<CharT> expected, test_format_string<CharT, Args...> fmt, Args&&... args) {
   std::basic_string<CharT> out = std::format(fmt, std::forward<Args>(args)...);
-#ifndef TEST_HAS_NO_LOCALIZATION
-  if constexpr (std::same_as<CharT, char>)
-    if (out != expected)
-      std::cerr << "\nFormat string   " << fmt.get() << "\nExpected output " << expected << "\nActual output   " << out
-                << '\n';
-#endif // TEST_HAS_NO_LOCALIZATION
-  assert(out == expected);
+  REQUIRE(out == expected, "\nFormat string   ", fmt, "\nExpected output ", expected, "\nActual output   ", out, '\n');
 };
 
 auto test_exception = []<class CharT, class... Args>(std::string_view, std::basic_string_view<CharT>, Args&&...) {
   // After P2216 most exceptions thrown by std::format become ill-formed.
   // Therefore this tests does nothing.
   // A basic ill-formed test is done in format.verify.cpp
   // The exceptions are tested by other functions that don't use the basic-format-string as fmt argument.
 };

libcxx/test/std/utilities/format/format.tuple/format.functions.vformat.pass.cpp

 // string vformat(string_view fmt, format_args args);
 // wstring vformat(wstring_view fmt, wformat_args args);
 
 #include <format>
 #include <cassert>
 
 #include "test_macros.h"
 #include "format.functions.tests.h"
-
-#ifndef TEST_HAS_NO_LOCALIZATION
-#  include <iostream>
-#  include <concepts>
-#endif
+#include "test_validation.h"
 
 auto test = []<class CharT, class... Args>(
                 std::basic_string_view<CharT> expected, std::basic_string_view<CharT> fmt, Args&&... args) {
   std::basic_string<CharT> out = std::vformat(fmt, std::make_format_args<context_t<CharT>>(args...));
-#ifndef TEST_HAS_NO_LOCALIZATION
-  if constexpr (std::same_as<CharT, char>)
-    if (out != expected)
-      std::cerr << "\nFormat string   " << fmt << "\nExpected output " << expected << "\nActual output   " << out
-                << '\n';
-#endif // TEST_HAS_NO_LOCALIZATION
-  assert(out == expected);
+  REQUIRE(out == expected, "\nFormat string   ", fmt, "\nExpected output ", expected, "\nActual output   ", out, '\n');
 };
 
 auto test_exception =
     []<class CharT, class... Args>(
         [[maybe_unused]] std::string_view what,
         [[maybe_unused]] std::basic_string_view<CharT> fmt,
         [[maybe_unused]] Args&&... args) {
 #ifndef TEST_HAS_NO_EXCEPTIONS
       try {
         TEST_IGNORE_NODISCARD std::vformat(fmt, std::make_format_args<context_t<CharT>>(args...));
-#  if !defined(TEST_HAS_NO_LOCALIZATION)
-        if constexpr (std::same_as<CharT, char>)
-          std::cerr << "\nFormat string   " << fmt << "\nDidn't throw an exception.\n";
-#  endif //  !defined(TEST_HAS_NO_LOCALIZATION
-        assert(false);
+        FAIL("\nFormat string   ", fmt, "\nDidn't throw an exception.\n");
       } catch ([[maybe_unused]] const std::format_error& e) {
-#  if defined(_LIBCPP_VERSION)
-#    if !defined(TEST_HAS_NO_LOCALIZATION)
-        if constexpr (std::same_as<CharT, char>) {
-          if (e.what() != what)
-            std::cerr << "\nFormat string   " << fmt << "\nExpected exception " << what << "\nActual exception   "
-                      << e.what() << '\n';
-        }
-#    endif // !defined(TEST_HAS_NO_LOCALIZATION
-        assert(e.what() == what);
-#  endif   // defined(_LIBCPP_VERSION)
+        LIBCPP_REQUIRE(
+            e.what() == what,
+            "\nFormat string   ",
+            fmt,
+            "\nExpected exception ",
+            what,
+            "\nActual exception   ",
+            e.what(),
+            '\n');
+
         return;
       }
       assert(false);
 #endif
     };
 
 int main(int, char**) {
   run_tests<char>(test, test_exception);
 
 #ifndef TEST_HAS_NO_WIDE_CHARACTERS
   run_tests<wchar_t>(test, test_exception);
 #endif
 
   return 0;
 }

libcxx/test/support/test_validation.h

+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef TEST_SUPPORT_TEST_VALIDATION_H
+#define TEST_SUPPORT_TEST_VALIDATION_H
+
+// Contains a set of validation macros.
+//
+// Note these test were added after C++20 was well supported by the compilers
+// used. To make the implementation simple the macros require C++20 or newer.
+// It's not expected that existing tests start to use these new macros.
+//
+// These macros are an alternative to using assert. The differences are:
+// - The assert message isn't localized.
+// - It's possible to log additional information. This is useful when the
+//   function asserting is a helper function. In these cases the assertion
+//   failure contains to little information to find the issue. For example, in
+//   the format functions, the really useful information is the actual output,
+//   the expected output, and the format string used. These macros allow
+//   logging additional arguments.
+
+#include "test_macros.h"
+
+#include <cstdio>
+#include <cstdlib>
+
+#ifndef TEST_HAS_NO_LOCALIZATION
+#  include <iostream>
+#endif // TEST_HAS_NO_LOCALIZATION
+
+#if TEST_STD_VER > 17
+
+#  ifndef TEST_HAS_NO_LOCALIZATION
+template <class T>
+concept test_char_streamable = requires(T&& value) { std::cerr << std::forward<T>(value); };
+#  endif // TEST_HAS_NO_LOCALIZATION
+
+// Logs the error and calls exit.
+//
+// It show a generic assert like message and if possible streams the args to
+// std::cerr else streams a default message.  Not being able to stream is not
+// considered and error. For example, streaming to std::wcerr doesn't work
+// properly in the CI. Therefore the formatting tests should only stream to
+// std::cerr.
+template <class... Args>
+[[noreturn]] void test_log_error(const char* condition, const char* file, int line, [[maybe_unused]] Args&&... args) {
+  const char* msg = condition ? "Assertion failure: " : "Unconditional failure:";
+  fprintf(stderr, "%s%s %s %d\n", msg, condition, file, line);
+#  ifndef TEST_HAS_NO_LOCALIZATION
+  if constexpr ((test_char_streamable<Args> && ...))
+    ((std::cerr << std::forward<Args>(args)), ...);
+  else
+    std::cerr << "Message discarded since it can't be streamed to std::cerr.\n";
+#  endif
+  exit(EXIT_FAILURE);
+}
+
+inline constexpr auto test_fail = []<class... Args>(const char* file, int line, Args&&... args) {
+  test_log_error("", file, line, std::forward<Args>(args)...);
+};
+
+inline constexpr auto test_require =
+    []<class... Args>(bool condition, const char* condition_str, const char* file, int line, Args&&... args) {
+      if (condition)
+        return;
+
+      test_log_error(condition_str, file, line, std::forward<Args>(args)...);
+    };
+
+inline constexpr auto test_libcpp_require =
+    []<class... Args>(
+        [[maybe_unused]] bool condition,
+        [[maybe_unused]] const char* condition_str,
+        [[maybe_unused]] const char* file,
+        [[maybe_unused]] int line,
+        [[maybe_unused]] Args&&... args) {
+#  if defined(_LIBCPP_VERSION)
+      test_require(condition, condition_str, file, line, std::forward<Args>(args)...);
+#  endif // _LIBCPP_VERSION
+    };
+
+// assert(false) replacement
+#  define FAIL(...) test_fail(__FILE__, __LINE__ __VA_OPT__(, ) __VA_ARGS__)
+
+// assert replacement.
+#  define REQUIRE(CONDITION, ...) test_require(CONDITION, #CONDITION, __FILE__, __LINE__ __VA_OPT__(, ) __VA_ARGS__)
+
+// LIBCPP_ASSERT replacement
+//
+// This requirement is only tested when the test suite is used for libc++.
+// This allows checking libc++ specific requirements, for example the error
+// messages of exceptions.
+#  define LIBCPP_REQUIRE(CONDITION, ...)                                                                               \
+    test_libcpp_require(CONDITION, #CONDITION, __FILE__, __LINE__ __VA_OPT__(, ) __VA_ARGS__)
+
+#endif // TEST_STD_VER > 17
+
+#endif // TEST_SUPPORT_TEST_VALIDATION_H