diff --git a/libcxx/docs/Contributing.rst b/libcxx/docs/Contributing.rst --- a/libcxx/docs/Contributing.rst +++ b/libcxx/docs/Contributing.rst @@ -31,7 +31,7 @@ Before committing or creating a review, please go through this check-list to make sure you don't forget anything: -- Do you have tests for every public class and/or function you're adding or modifying? +- Do you have :ref:`tests ` for every public class and/or function you're adding or modifying? - Did you update the synopsis of the relevant headers? - Did you update the relevant files to track implementation status (in ``docs/Status/``)? - Did you mark all functions and type declarations with the :ref:`proper visibility macro `? diff --git a/libcxx/docs/TestingLibcxx.rst b/libcxx/docs/TestingLibcxx.rst --- a/libcxx/docs/TestingLibcxx.rst +++ b/libcxx/docs/TestingLibcxx.rst @@ -5,6 +5,8 @@ .. contents:: :local: +.. _testing: + Getting Started =============== @@ -121,7 +123,7 @@ This is useful for editing source files as you're testing your code in the Docker container. Writing Tests -------------- +============= When writing tests for the libc++ test suite, you should follow a few guidelines. This will ensure that your tests can run on a wide variety of hardware and under @@ -143,6 +145,197 @@ necessarily available on all devices we may want to run the tests on (even though supporting Python is probably trivial for the build-host). +Structure of the testing related directories +-------------------------------------------- + +The tests of libc++ are stored in libc++'s testing related subdirectories: + +- ``test/support`` This directory contains several helper headers with generic + parts for the tests. The most important header is ``test_macros.h``. This file + contains configuration information regarding the platform used. This is + similar to the ``__config`` file in libc++'s ``include`` directory. Since + libc++'s tests are used by other Standard libraries the ``TEST_FOO`` macros + should be using instead of the ``_LIBCPP_FOO`` macros. +- ``test/std`` This directory contains the tests that validate the library under + test conforms to the C++ Standard. The paths and the names of the test match + the section names in the C++ Standard. Note that the C++ Standard sometimes + reorganises its structure, therefore some tests are at their historical + location instead of their current location. +- ``test/libcxx`` This directory contains the tests that validate libc++ + specific implementation details. For example libc++ has "wrapped + iterators", that allow doing bounds checks. These are tested in this + directory Tests that mainly test Standard conformance and a few libc++ + specific parts are in the ``test/std`` directory. The structure of this + directories follows the structure of ``test/std``. +- ``utils/libcxx/test`` This directory contains the testing infrastructure that + configures LIT testing. For example, when configuring libc++ without + locales in CMake it sets the appropriate LIT test feature. + +Structure of a test +------------------- + +Some platforms where libc++ is tested have requirement on the signature of +``main`` and require ``main`` to explicitly return a value. Therefore the +typical ``main`` function should look like: + +.. code-block:: cpp + + int main(int, char**) { + ... + return 0; + } + + +The C++ Standard has ``constexpr`` requirements. The typical way to test that, +is to create a helper ``test`` function that returns a ``bool`` and use the +following ``main`` function: + +.. code-block:: cpp + + constexpr bool test() { + ... + return true; + } + + int main(int, char**) { + test() + static_assert(test()); + + return 0; + } + +Tests in libc++ mainly use ``assert`` and ``static_assert`` for testing. There +are a few helper macros and function that can be used to make it easier to +write common tests. + +test/support/assert_macros.h +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The header contains several macros with user specified log messages. This is +useful when a normal assertion failure lacks the information to easily +understand why the test has failed. This usually happens when the test is in a +helper function. For example the ``std::format`` tests use a helper function +for its validation. When the test fails it will give the line in the helper +function with the condition ``out == expected`` failed. Without knowing what +the value of ``format string``, ``out`` and ``expected`` are it is not easy to +understand why the test has failed. By logging these three values the point of +failure can be found without resorting to a debugger. + +Several of these macros are documented to take an ``ARG``. This ``ARG``: + + - if it is a ``const char*`` or ``std::string`` its contents are written to + the ``stderr``, + - else it is invoked without any additional arguments. + +This makes it possible to write additional information when a test fails, +either by supplying a hard-coded string or generate it at runtime. For example + +TEST_FAIL(ARG) +^^^^^^^^^^^^^^ + +This macro is an unconditional failure with a log message ``ARG``. The main +use-case is to fail when code is reached that should be unreachable. + + +TEST_REQUIRE(CONDITION, ARG) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This macro requires its ``CONDITION`` to evaluate to ``true``. If that fails it +will fail the test with a log message ``ARG``. + + +TEST_LIBCPP_REQUIRE((CONDITION, ARG) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If the library under test is libc++ it behaves like ``TEST_REQUIRE``, else it +is a no-op. This makes it possible to test libc++ specific behaviour. For +example testing whether the ``what()`` of an exception thrown matches libc++'s +expectations. (Usually the Standard requires certain exceptions to be thrown, +but not the contents of its ``what()`` message.) + + +TEST_DOES_NOT_THROW(EXPR) +^^^^^^^^^^^^^^^^^^^^^^^^^ + +Validates execution of ``EXPR`` does not throw an exception. + +TEST_THROWS_TYPE(TYPE, EXPR) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Validates the execution of ``EXPR`` throws an exception of the type ``TYPE``. + + +TEST_VALIDATE_EXCEPTION(TYPE, PRED, EXPR) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Validates the execution of ``EXPR`` throws an exception of the type ``TYPE`` +which passes validation of ``PRED``. Using this macro makes it easier to write +tests using exceptions. The code to write a test manually would be: + + +.. code-block:: cpp + + void test_excption([[maybe_unused]] int arg) { + #ifndef TEST_HAS_NO_EXCEPTIONS // Do nothing when tests are disables + try { + foo(arg); + assert(false); // validates foo really throws. + } catch ([[maybe_unused]] const bar& e) { + LIBCPP_ASSERT(e.what() == what); + return; + } + assert(false); // validates bar was thrown + #endif + } + +The same test using a macro: + +.. code-block:: cpp + + void test_excption([[maybe_unused]] int arg) { + TEST_VALIDATE_EXCEPTION(bar, + [](const bar& e) { + LIBCPP_ASSERT(e.what() == what); + }, + foo(arg)); + } + +TEST_IS_UNSUPPORTED +^^^^^^^^^^^^^^^^^^^ + +An alias for ``return``. This can be used to document a part of the tests is +not supported on some platforms or with some configurations. For example +several filesystem permissions differ between Windows and POSIX platforms, +therefore some of parts of the permission tests are not done on Windows. + + +test/support/concat_macros.h +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This file contains a helper macro ``TEST_WRITE_CONCATENATED`` to lazily +concatenate its arguments to a ``std::string`` and write it to ``stderr``. When +the output can't be concatenated a default message will be written to +``stderr``. This is useful for tests where the arguments use different +character types like ``char`` and ``wchar_t``, the latter can't simply be +written to ``stderrr``. + +This macro is in a different header as ``assert_macros.h`` since it pulls in +additional headers. + + .. note: This macro can only be used in test using C++20 or newer. The macro + was added at a time where most of lib++'s C++17 support was complete. + Since it is not expected to add this to existing tests no effort was + taken to make it work in earlier language versions. + + +Additional reading +------------------ + +The function ``CxxStandardLibraryTest`` in the file +``libcxx/utils/libcxx/test/format.py`` has documentation about writing test. It +explains the difference between the test named ``foo.pass.cpp`` and named +``foo.verify.cpp`` are. + Benchmarks ========== diff --git a/libcxx/test/std/containers/container.adaptors/container.adaptors.format/format.functions.format.pass.cpp b/libcxx/test/std/containers/container.adaptors/container.adaptors.format/format.functions.format.pass.cpp --- a/libcxx/test/std/containers/container.adaptors/container.adaptors.format/format.functions.format.pass.cpp +++ b/libcxx/test/std/containers/container.adaptors/container.adaptors.format/format.functions.format.pass.cpp @@ -35,13 +35,14 @@ #include "test_format_string.h" #include "test_macros.h" #include "assert_macros.h" +#include "concat_macros.h" auto test = []( std::basic_string_view expected, test_format_string fmt, Args&&... args) { std::basic_string out = std::format(fmt, std::forward(args)...); - TEST_REQUIRE( - out == expected, - test_concat_message("\nFormat string ", fmt, "\nExpected output ", expected, "\nActual output ", out, '\n')); + TEST_REQUIRE(out == expected, + TEST_WRITE_CONCATENATED( + "\nFormat string ", fmt, "\nExpected output ", expected, "\nActual output ", out, '\n')); }; auto test_exception = [](std::string_view, std::basic_string_view, Args&&...) { diff --git a/libcxx/test/std/containers/container.adaptors/container.adaptors.format/format.functions.vformat.pass.cpp b/libcxx/test/std/containers/container.adaptors/container.adaptors.format/format.functions.vformat.pass.cpp --- a/libcxx/test/std/containers/container.adaptors/container.adaptors.format/format.functions.vformat.pass.cpp +++ b/libcxx/test/std/containers/container.adaptors/container.adaptors.format/format.functions.vformat.pass.cpp @@ -32,13 +32,14 @@ #include "format.functions.tests.h" #include "test_macros.h" #include "assert_macros.h" +#include "concat_macros.h" auto test = []( std::basic_string_view expected, std::basic_string_view fmt, Args&&... args) { std::basic_string out = std::vformat(fmt, std::make_format_args>(args...)); - TEST_REQUIRE( - out == expected, - test_concat_message("\nFormat string ", fmt, "\nExpected output ", expected, "\nActual output ", out, '\n')); + TEST_REQUIRE(out == expected, + TEST_WRITE_CONCATENATED( + "\nFormat string ", fmt, "\nExpected output ", expected, "\nActual output ", out, '\n')); }; auto test_exception = @@ -49,11 +50,11 @@ #ifndef TEST_HAS_NO_EXCEPTIONS try { TEST_IGNORE_NODISCARD std::vformat(fmt, std::make_format_args>(args...)); - TEST_FAIL(test_concat_message("\nFormat string ", fmt, "\nDidn't throw an exception.\n")); + TEST_FAIL(TEST_WRITE_CONCATENATED("\nFormat string ", fmt, "\nDidn't throw an exception.\n")); } catch (const std::format_error& e) { TEST_LIBCPP_REQUIRE( e.what() == what, - test_concat_message( + TEST_WRITE_CONCATENATED( "\nFormat string ", fmt, "\nExpected exception ", what, "\nActual exception ", e.what(), '\n')); return; diff --git a/libcxx/test/std/containers/sequences/vector.bool/vector.bool.fmt/format.functions.format.pass.cpp b/libcxx/test/std/containers/sequences/vector.bool/vector.bool.fmt/format.functions.format.pass.cpp --- a/libcxx/test/std/containers/sequences/vector.bool/vector.bool.fmt/format.functions.format.pass.cpp +++ b/libcxx/test/std/containers/sequences/vector.bool/vector.bool.fmt/format.functions.format.pass.cpp @@ -29,13 +29,14 @@ #include "test_format_string.h" #include "test_macros.h" #include "assert_macros.h" +#include "concat_macros.h" auto test = []( std::basic_string_view expected, test_format_string fmt, Args&&... args) { std::basic_string out = std::format(fmt, std::forward(args)...); - TEST_REQUIRE( - out == expected, - test_concat_message("\nFormat string ", fmt, "\nExpected output ", expected, "\nActual output ", out, '\n')); + TEST_REQUIRE(out == expected, + TEST_WRITE_CONCATENATED( + "\nFormat string ", fmt, "\nExpected output ", expected, "\nActual output ", out, '\n')); }; auto test_exception = [](std::string_view, std::basic_string_view, Args&&...) { diff --git a/libcxx/test/std/containers/sequences/vector.bool/vector.bool.fmt/format.functions.vformat.pass.cpp b/libcxx/test/std/containers/sequences/vector.bool/vector.bool.fmt/format.functions.vformat.pass.cpp --- a/libcxx/test/std/containers/sequences/vector.bool/vector.bool.fmt/format.functions.vformat.pass.cpp +++ b/libcxx/test/std/containers/sequences/vector.bool/vector.bool.fmt/format.functions.vformat.pass.cpp @@ -30,13 +30,14 @@ #include "format.functions.tests.h" #include "test_macros.h" #include "assert_macros.h" +#include "concat_macros.h" auto test = []( std::basic_string_view expected, std::basic_string_view fmt, Args&&... args) { std::basic_string out = std::vformat(fmt, std::make_format_args>(args...)); - TEST_REQUIRE( - out == expected, - test_concat_message("\nFormat string ", fmt, "\nExpected output ", expected, "\nActual output ", out, '\n')); + TEST_REQUIRE(out == expected, + TEST_WRITE_CONCATENATED( + "\nFormat string ", fmt, "\nExpected output ", expected, "\nActual output ", out, '\n')); }; auto test_exception = @@ -47,11 +48,11 @@ #ifndef TEST_HAS_NO_EXCEPTIONS try { TEST_IGNORE_NODISCARD std::vformat(fmt, std::make_format_args>(args...)); - TEST_FAIL(test_concat_message("\nFormat string ", fmt, "\nDidn't throw an exception.\n")); + TEST_FAIL(TEST_WRITE_CONCATENATED("\nFormat string ", fmt, "\nDidn't throw an exception.\n")); } catch (const std::format_error& e) { TEST_LIBCPP_REQUIRE( e.what() == what, - test_concat_message( + TEST_WRITE_CONCATENATED( "\nFormat string ", fmt, "\nExpected exception ", what, "\nActual exception ", e.what(), '\n')); return; diff --git a/libcxx/test/std/utilities/format/format.functions/escaped_output.ascii.pass.cpp b/libcxx/test/std/utilities/format/format.functions/escaped_output.ascii.pass.cpp --- a/libcxx/test/std/utilities/format/format.functions/escaped_output.ascii.pass.cpp +++ b/libcxx/test/std/utilities/format/format.functions/escaped_output.ascii.pass.cpp @@ -26,6 +26,7 @@ #include "make_string.h" #include "test_format_string.h" #include "assert_macros.h" +#include "concat_macros.h" #ifndef TEST_HAS_NO_LOCALIZATION # include @@ -38,7 +39,7 @@ { std::basic_string out = std::format(fmt, std::forward(args)...); TEST_REQUIRE(out == expected, - test_concat_message( + TEST_WRITE_CONCATENATED( "\nFormat string ", fmt.get(), "\nExpected output ", expected, "\nActual output ", out, '\n')); } #ifndef TEST_HAS_NO_LOCALIZATION diff --git a/libcxx/test/std/utilities/format/format.functions/escaped_output.unicode.pass.cpp b/libcxx/test/std/utilities/format/format.functions/escaped_output.unicode.pass.cpp --- a/libcxx/test/std/utilities/format/format.functions/escaped_output.unicode.pass.cpp +++ b/libcxx/test/std/utilities/format/format.functions/escaped_output.unicode.pass.cpp @@ -32,6 +32,7 @@ #include "make_string.h" #include "test_format_string.h" #include "assert_macros.h" +#include "concat_macros.h" #ifndef TEST_HAS_NO_LOCALIZATION # include @@ -44,7 +45,7 @@ { std::basic_string out = std::format(fmt, std::forward(args)...); TEST_REQUIRE(out == expected, - test_concat_message( + TEST_WRITE_CONCATENATED( "\nFormat string ", fmt.get(), "\nExpected output ", expected, "\nActual output ", out, '\n')); } #ifndef TEST_HAS_NO_LOCALIZATION diff --git a/libcxx/test/std/utilities/format/format.functions/format.locale.pass.cpp b/libcxx/test/std/utilities/format/format.functions/format.locale.pass.cpp --- a/libcxx/test/std/utilities/format/format.functions/format.locale.pass.cpp +++ b/libcxx/test/std/utilities/format/format.functions/format.locale.pass.cpp @@ -28,6 +28,7 @@ #include "string_literal.h" #include "test_format_string.h" #include "assert_macros.h" +#include "concat_macros.h" auto test = []( @@ -35,7 +36,7 @@ std::basic_string out = std::format(std::locale(), fmt, std::forward(args)...); TEST_REQUIRE( out == expected, - test_concat_message( + TEST_WRITE_CONCATENATED( "\nFormat string ", fmt.get(), "\nExpected output ", expected, "\nActual output ", out, '\n')); }; diff --git a/libcxx/test/std/utilities/format/format.functions/format.pass.cpp b/libcxx/test/std/utilities/format/format.functions/format.pass.cpp --- a/libcxx/test/std/utilities/format/format.functions/format.pass.cpp +++ b/libcxx/test/std/utilities/format/format.functions/format.pass.cpp @@ -30,6 +30,7 @@ #include "string_literal.h" #include "test_format_string.h" #include "assert_macros.h" +#include "concat_macros.h" auto test = []( @@ -37,7 +38,7 @@ std::basic_string out = std::format(fmt, std::forward(args)...); TEST_REQUIRE( out == expected, - test_concat_message( + TEST_WRITE_CONCATENATED( "\nFormat string ", fmt.get(), "\nExpected output ", expected, "\nActual output ", out, '\n')); }; diff --git a/libcxx/test/std/utilities/format/format.functions/format_tests.h b/libcxx/test/std/utilities/format/format.functions/format_tests.h --- a/libcxx/test/std/utilities/format/format.functions/format_tests.h +++ b/libcxx/test/std/utilities/format/format.functions/format_tests.h @@ -2669,7 +2669,6 @@ check_exception("The format string contains an invalid escape sequence", SV("{:}-}"), 42); check_exception("The format string contains an invalid escape sequence", SV("} ")); - check_exception("The arg-id of the format-spec starts with an invalid character", SV("{-"), 42); check_exception("Argument index out of bounds", SV("hello {}")); check_exception("Argument index out of bounds", SV("hello {0}")); diff --git a/libcxx/test/std/utilities/format/format.functions/locale-specific_form.pass.cpp b/libcxx/test/std/utilities/format/format.functions/locale-specific_form.pass.cpp --- a/libcxx/test/std/utilities/format/format.functions/locale-specific_form.pass.cpp +++ b/libcxx/test/std/utilities/format/format.functions/locale-specific_form.pass.cpp @@ -94,6 +94,7 @@ #include "string_literal.h" #include "test_format_string.h" #include "assert_macros.h" +#include "concat_macros.h" #define STR(S) MAKE_STRING(CharT, S) #define SV(S) MAKE_STRING_VIEW(CharT, S) @@ -129,7 +130,7 @@ { std::basic_string out = std::format(fmt, std::forward(args)...); TEST_REQUIRE(out == expected, - test_concat_message( + TEST_WRITE_CONCATENATED( "\nFormat string ", fmt.get(), "\nExpected output ", expected, "\nActual output ", out, '\n')); } // *** vformat *** diff --git a/libcxx/test/std/utilities/format/format.functions/vformat.locale.pass.cpp b/libcxx/test/std/utilities/format/format.functions/vformat.locale.pass.cpp --- a/libcxx/test/std/utilities/format/format.functions/vformat.locale.pass.cpp +++ b/libcxx/test/std/utilities/format/format.functions/vformat.locale.pass.cpp @@ -24,13 +24,14 @@ #include "format_tests.h" #include "string_literal.h" #include "assert_macros.h" +#include "concat_macros.h" auto test = []( std::basic_string_view expected, std::basic_string_view fmt, Args&&... args) constexpr { std::basic_string out = std::vformat(std::locale(), fmt, std::make_format_args>(args...)); - TEST_REQUIRE( - out == expected, - test_concat_message("\nFormat string ", fmt, "\nExpected output ", expected, "\nActual output ", out, '\n')); + TEST_REQUIRE(out == expected, + TEST_WRITE_CONCATENATED( + "\nFormat string ", fmt, "\nExpected output ", expected, "\nActual output ", out, '\n')); }; auto test_exception = @@ -41,11 +42,11 @@ #ifndef TEST_HAS_NO_EXCEPTIONS try { TEST_IGNORE_NODISCARD std::vformat(std::locale(), fmt, std::make_format_args>(args...)); - TEST_FAIL(test_concat_message("\nFormat string ", fmt, "\nDidn't throw an exception.\n")); + TEST_FAIL(TEST_WRITE_CONCATENATED("\nFormat string ", fmt, "\nDidn't throw an exception.\n")); } catch ([[maybe_unused]] const std::format_error& e) { TEST_LIBCPP_REQUIRE( e.what() == what, - test_concat_message( + TEST_WRITE_CONCATENATED( "\nFormat string ", fmt, "\nExpected exception ", what, "\nActual exception ", e.what(), '\n')); return; } diff --git a/libcxx/test/std/utilities/format/format.functions/vformat.pass.cpp b/libcxx/test/std/utilities/format/format.functions/vformat.pass.cpp --- a/libcxx/test/std/utilities/format/format.functions/vformat.pass.cpp +++ b/libcxx/test/std/utilities/format/format.functions/vformat.pass.cpp @@ -23,13 +23,14 @@ #include "format_tests.h" #include "string_literal.h" #include "assert_macros.h" +#include "concat_macros.h" auto test = []( std::basic_string_view expected, std::basic_string_view fmt, Args&&... args) constexpr { std::basic_string out = std::vformat(fmt, std::make_format_args>(args...)); - TEST_REQUIRE( - out == expected, - test_concat_message("\nFormat string ", fmt, "\nExpected output ", expected, "\nActual output ", out, '\n')); + TEST_REQUIRE(out == expected, + TEST_WRITE_CONCATENATED( + "\nFormat string ", fmt, "\nExpected output ", expected, "\nActual output ", out, '\n')); }; auto test_exception = @@ -40,11 +41,11 @@ #ifndef TEST_HAS_NO_EXCEPTIONS try { TEST_IGNORE_NODISCARD std::vformat(fmt, std::make_format_args>(args...)); - TEST_FAIL(test_concat_message("\nFormat string ", fmt, "\nDidn't throw an exception.\n")); + TEST_FAIL(TEST_WRITE_CONCATENATED("\nFormat string ", fmt, "\nDidn't throw an exception.\n")); } catch ([[maybe_unused]] const std::format_error& e) { TEST_LIBCPP_REQUIRE( e.what() == what, - test_concat_message( + TEST_WRITE_CONCATENATED( "\nFormat string ", fmt, "\nExpected exception ", what, "\nActual exception ", e.what(), '\n')); return; diff --git a/libcxx/test/std/utilities/format/format.range/format.range.fmtmap/format.functions.format.pass.cpp b/libcxx/test/std/utilities/format/format.range/format.range.fmtmap/format.functions.format.pass.cpp --- a/libcxx/test/std/utilities/format/format.range/format.range.fmtmap/format.functions.format.pass.cpp +++ b/libcxx/test/std/utilities/format/format.range/format.range.fmtmap/format.functions.format.pass.cpp @@ -32,13 +32,14 @@ #include "test_format_string.h" #include "test_macros.h" #include "assert_macros.h" +#include "concat_macros.h" auto test = []( std::basic_string_view expected, test_format_string fmt, Args&&... args) { std::basic_string out = std::format(fmt, std::forward(args)...); - TEST_REQUIRE( - out == expected, - test_concat_message("\nFormat string ", fmt, "\nExpected output ", expected, "\nActual output ", out, '\n')); + TEST_REQUIRE(out == expected, + TEST_WRITE_CONCATENATED( + "\nFormat string ", fmt, "\nExpected output ", expected, "\nActual output ", out, '\n')); }; auto test_exception = [](std::string_view, std::basic_string_view, Args&&...) { diff --git a/libcxx/test/std/utilities/format/format.range/format.range.fmtmap/format.functions.vformat.pass.cpp b/libcxx/test/std/utilities/format/format.range/format.range.fmtmap/format.functions.vformat.pass.cpp --- a/libcxx/test/std/utilities/format/format.range/format.range.fmtmap/format.functions.vformat.pass.cpp +++ b/libcxx/test/std/utilities/format/format.range/format.range.fmtmap/format.functions.vformat.pass.cpp @@ -29,13 +29,14 @@ #include "format.functions.tests.h" #include "test_macros.h" #include "assert_macros.h" +#include "concat_macros.h" auto test = []( std::basic_string_view expected, std::basic_string_view fmt, Args&&... args) { std::basic_string out = std::vformat(fmt, std::make_format_args>(args...)); - TEST_REQUIRE( - out == expected, - test_concat_message("\nFormat string ", fmt, "\nExpected output ", expected, "\nActual output ", out, '\n')); + TEST_REQUIRE(out == expected, + TEST_WRITE_CONCATENATED( + "\nFormat string ", fmt, "\nExpected output ", expected, "\nActual output ", out, '\n')); }; auto test_exception = @@ -46,11 +47,11 @@ #ifndef TEST_HAS_NO_EXCEPTIONS try { TEST_IGNORE_NODISCARD std::vformat(fmt, std::make_format_args>(args...)); - TEST_FAIL(test_concat_message("\nFormat string ", fmt, "\nDidn't throw an exception.\n")); + TEST_FAIL(TEST_WRITE_CONCATENATED("\nFormat string ", fmt, "\nDidn't throw an exception.\n")); } catch (const std::format_error& e) { TEST_LIBCPP_REQUIRE( e.what() == what, - test_concat_message( + TEST_WRITE_CONCATENATED( "\nFormat string ", fmt, "\nExpected exception ", what, "\nActual exception ", e.what(), '\n')); return; diff --git a/libcxx/test/std/utilities/format/format.range/format.range.fmtset/format.functions.format.pass.cpp b/libcxx/test/std/utilities/format/format.range/format.range.fmtset/format.functions.format.pass.cpp --- a/libcxx/test/std/utilities/format/format.range/format.range.fmtset/format.functions.format.pass.cpp +++ b/libcxx/test/std/utilities/format/format.range/format.range.fmtset/format.functions.format.pass.cpp @@ -32,13 +32,14 @@ #include "test_format_string.h" #include "test_macros.h" #include "assert_macros.h" +#include "concat_macros.h" auto test = []( std::basic_string_view expected, test_format_string fmt, Args&&... args) { std::basic_string out = std::format(fmt, std::forward(args)...); - TEST_REQUIRE( - out == expected, - test_concat_message("\nFormat string ", fmt, "\nExpected output ", expected, "\nActual output ", out, '\n')); + TEST_REQUIRE(out == expected, + TEST_WRITE_CONCATENATED( + "\nFormat string ", fmt, "\nExpected output ", expected, "\nActual output ", out, '\n')); }; auto test_exception = [](std::string_view, std::basic_string_view, Args&&...) { diff --git a/libcxx/test/std/utilities/format/format.range/format.range.fmtset/format.functions.vformat.pass.cpp b/libcxx/test/std/utilities/format/format.range/format.range.fmtset/format.functions.vformat.pass.cpp --- a/libcxx/test/std/utilities/format/format.range/format.range.fmtset/format.functions.vformat.pass.cpp +++ b/libcxx/test/std/utilities/format/format.range/format.range.fmtset/format.functions.vformat.pass.cpp @@ -29,13 +29,14 @@ #include "format.functions.tests.h" #include "test_macros.h" #include "assert_macros.h" +#include "concat_macros.h" auto test = []( std::basic_string_view expected, std::basic_string_view fmt, Args&&... args) { std::basic_string out = std::vformat(fmt, std::make_format_args>(args...)); - TEST_REQUIRE( - out == expected, - test_concat_message("\nFormat string ", fmt, "\nExpected output ", expected, "\nActual output ", out, '\n')); + TEST_REQUIRE(out == expected, + TEST_WRITE_CONCATENATED( + "\nFormat string ", fmt, "\nExpected output ", expected, "\nActual output ", out, '\n')); }; auto test_exception = @@ -46,11 +47,11 @@ #ifndef TEST_HAS_NO_EXCEPTIONS try { TEST_IGNORE_NODISCARD std::vformat(fmt, std::make_format_args>(args...)); - TEST_FAIL(test_concat_message("\nFormat string ", fmt, "\nDidn't throw an exception.\n")); + TEST_FAIL(TEST_WRITE_CONCATENATED("\nFormat string ", fmt, "\nDidn't throw an exception.\n")); } catch (const std::format_error& e) { TEST_LIBCPP_REQUIRE( e.what() == what, - test_concat_message( + TEST_WRITE_CONCATENATED( "\nFormat string ", fmt, "\nExpected exception ", what, "\nActual exception ", e.what(), '\n')); return; diff --git a/libcxx/test/std/utilities/format/format.range/format.range.formatter/format.functions.format.pass.cpp b/libcxx/test/std/utilities/format/format.range/format.range.formatter/format.functions.format.pass.cpp --- a/libcxx/test/std/utilities/format/format.range/format.range.formatter/format.functions.format.pass.cpp +++ b/libcxx/test/std/utilities/format/format.range/format.range.formatter/format.functions.format.pass.cpp @@ -33,13 +33,14 @@ #include "test_format_string.h" #include "test_macros.h" #include "assert_macros.h" +#include "concat_macros.h" auto test = []( std::basic_string_view expected, test_format_string fmt, Args&&... args) { std::basic_string out = std::format(fmt, std::forward(args)...); - TEST_REQUIRE( - out == expected, - test_concat_message("\nFormat string ", fmt, "\nExpected output ", expected, "\nActual output ", out, '\n')); + TEST_REQUIRE(out == expected, + TEST_WRITE_CONCATENATED( + "\nFormat string ", fmt, "\nExpected output ", expected, "\nActual output ", out, '\n')); }; auto test_exception = [](std::string_view, std::basic_string_view, Args&&...) { diff --git a/libcxx/test/std/utilities/format/format.range/format.range.formatter/format.functions.vformat.pass.cpp b/libcxx/test/std/utilities/format/format.range/format.range.formatter/format.functions.vformat.pass.cpp --- a/libcxx/test/std/utilities/format/format.range/format.range.formatter/format.functions.vformat.pass.cpp +++ b/libcxx/test/std/utilities/format/format.range/format.range.formatter/format.functions.vformat.pass.cpp @@ -30,13 +30,14 @@ #include "format.functions.tests.h" #include "test_macros.h" #include "assert_macros.h" +#include "concat_macros.h" auto test = []( std::basic_string_view expected, std::basic_string_view fmt, Args&&... args) { std::basic_string out = std::vformat(fmt, std::make_format_args>(args...)); - TEST_REQUIRE( - out == expected, - test_concat_message("\nFormat string ", fmt, "\nExpected output ", expected, "\nActual output ", out, '\n')); + TEST_REQUIRE(out == expected, + TEST_WRITE_CONCATENATED( + "\nFormat string ", fmt, "\nExpected output ", expected, "\nActual output ", out, '\n')); }; auto test_exception = @@ -47,11 +48,11 @@ #ifndef TEST_HAS_NO_EXCEPTIONS try { TEST_IGNORE_NODISCARD std::vformat(fmt, std::make_format_args>(args...)); - TEST_FAIL(test_concat_message("\nFormat string ", fmt, "\nDidn't throw an exception.\n")); + TEST_FAIL(TEST_WRITE_CONCATENATED("\nFormat string ", fmt, "\nDidn't throw an exception.\n")); } catch (const std::format_error& e) { TEST_LIBCPP_REQUIRE( e.what() == what, - test_concat_message( + TEST_WRITE_CONCATENATED( "\nFormat string ", fmt, "\nExpected exception ", what, "\nActual exception ", e.what(), '\n')); return; diff --git a/libcxx/test/std/utilities/format/format.tuple/format.functions.format.pass.cpp b/libcxx/test/std/utilities/format/format.tuple/format.functions.format.pass.cpp --- a/libcxx/test/std/utilities/format/format.tuple/format.functions.format.pass.cpp +++ b/libcxx/test/std/utilities/format/format.tuple/format.functions.format.pass.cpp @@ -34,13 +34,14 @@ #include "test_format_string.h" #include "test_macros.h" #include "assert_macros.h" +#include "concat_macros.h" auto test = []( std::basic_string_view expected, test_format_string fmt, Args&&... args) { std::basic_string out = std::format(fmt, std::forward(args)...); - TEST_REQUIRE( - out == expected, - test_concat_message("\nFormat string ", fmt, "\nExpected output ", expected, "\nActual output ", out, '\n')); + TEST_REQUIRE(out == expected, + TEST_WRITE_CONCATENATED( + "\nFormat string ", fmt, "\nExpected output ", expected, "\nActual output ", out, '\n')); }; auto test_exception = [](std::string_view, std::basic_string_view, Args&&...) { diff --git a/libcxx/test/std/utilities/format/format.tuple/format.functions.vformat.pass.cpp b/libcxx/test/std/utilities/format/format.tuple/format.functions.vformat.pass.cpp --- a/libcxx/test/std/utilities/format/format.tuple/format.functions.vformat.pass.cpp +++ b/libcxx/test/std/utilities/format/format.tuple/format.functions.vformat.pass.cpp @@ -30,13 +30,14 @@ #include "test_macros.h" #include "format.functions.tests.h" #include "assert_macros.h" +#include "concat_macros.h" auto test = []( std::basic_string_view expected, std::basic_string_view fmt, Args&&... args) { std::basic_string out = std::vformat(fmt, std::make_format_args>(args...)); - TEST_REQUIRE( - out == expected, - test_concat_message("\nFormat string ", fmt, "\nExpected output ", expected, "\nActual output ", out, '\n')); + TEST_REQUIRE(out == expected, + TEST_WRITE_CONCATENATED( + "\nFormat string ", fmt, "\nExpected output ", expected, "\nActual output ", out, '\n')); }; auto test_exception = @@ -47,11 +48,11 @@ #ifndef TEST_HAS_NO_EXCEPTIONS try { TEST_IGNORE_NODISCARD std::vformat(fmt, std::make_format_args>(args...)); - TEST_FAIL(test_concat_message("\nFormat string ", fmt, "\nDidn't throw an exception.\n")); + TEST_FAIL(TEST_WRITE_CONCATENATED("\nFormat string ", fmt, "\nDidn't throw an exception.\n")); } catch ([[maybe_unused]] const std::format_error& e) { TEST_LIBCPP_REQUIRE( e.what() == what, - test_concat_message( + TEST_WRITE_CONCATENATED( "\nFormat string ", fmt, "\nExpected exception ", what, "\nActual exception ", e.what(), '\n')); return; diff --git a/libcxx/test/support/assert_macros.h b/libcxx/test/support/assert_macros.h --- a/libcxx/test/support/assert_macros.h +++ b/libcxx/test/support/assert_macros.h @@ -28,79 +28,127 @@ #include #include +#include -#ifndef TEST_HAS_NO_LOCALIZATION -# include -#endif +void test_log(const char* condition, const char* file, int line, const char* message) { + const char* msg = condition ? "Assertion failure: " : "Unconditional failure:"; + std::fprintf(stderr, "%s%s %s %d\n%s", msg, condition, file, line, message); +} -#if TEST_STD_VER > 17 - -# ifndef TEST_HAS_NO_LOCALIZATION -template -concept test_char_streamable = requires(T&& value) { std::stringstream{} << std::forward(value); }; -# endif - -// If possible concatenates message for the assertion function, else returns a -// default message. Not being able to stream is not considered and error. For -// example, streaming to std::wcerr doesn't work properly in the CI. Therefore -// the formatting tests should only stream to std::string_string. -template -std::string test_concat_message([[maybe_unused]] Args&&... args) { -# ifndef TEST_HAS_NO_LOCALIZATION - if constexpr ((test_char_streamable && ...)) { - std::stringstream sstr; - ((sstr << std::forward(args)), ...); - return sstr.str(); - } else -# endif - return "Message discarded since it can't be streamed to std::cerr.\n"; +void test_log(const char* condition, const char* file, int line, std::string&& message) { + test_log(condition, file, line, message.c_str()); } -#endif // TEST_STD_VER > 17 +template +void test_log(const char* condition, const char* file, int line, const F& functor) { + std::fprintf(stderr, "Assertion failure: %s %s %d\n", condition, file, line); + functor(); +} // Logs the error and calls exit. // // It shows a generic assert like message including a custom message. This // message should end with a newline. -[[noreturn]] void test_log_error(const char* condition, const char* file, int line, std::string&& message) { - const char* msg = condition ? "Assertion failure: " : "Unconditional failure:"; - std::fprintf(stderr, "%s%s %s %d\n%s", msg, condition, file, line, message.c_str()); - exit(EXIT_FAILURE); -} - -inline void test_fail(const char* file, int line, std::string&& message) { - test_log_error("", file, line, std::move(message)); +template +[[noreturn]] void test_fail(const char* file, int line, Arg&& arg) { + test_log("", file, line, std::forward(arg)); + std::abort(); } -inline void test_require(bool condition, const char* condition_str, const char* file, int line, std::string&& message) { +template +void test_require(bool condition, const char* condition_str, const char* file, int line, Arg&& arg) { if (condition) return; - test_log_error(condition_str, file, line, std::move(message)); -} - -inline void test_libcpp_require( - [[maybe_unused]] bool condition, - [[maybe_unused]] const char* condition_str, - [[maybe_unused]] const char* file, - [[maybe_unused]] int line, - [[maybe_unused]] std::string&& message) { -#if defined(_LIBCPP_VERSION) - test_require(condition, condition_str, file, line, std::move(message)); -#endif + test_log(condition_str, file, line, std::forward(arg)); + std::abort(); } // assert(false) replacement -#define TEST_FAIL(MSG) ::test_fail(__FILE__, __LINE__, MSG) +// The ARG is either a +// - c-ctring or std::string, in which case the string is printed to stderr, +// - an invocable object, which will be invoked. +#define TEST_FAIL(ARG) ::test_fail(__FILE__, __LINE__, ARG) // assert replacement. -#define TEST_REQUIRE(CONDITION, MSG) ::test_require(CONDITION, #CONDITION, __FILE__, __LINE__, MSG) +// ARG is the same as for TEST_FAIL +#define TEST_REQUIRE(CONDITION, ARG) ::test_require(CONDITION, #CONDITION, __FILE__, __LINE__, ARG) // LIBCPP_ASSERT replacement // // This requirement is only tested when the test suite is used for libc++. // This allows checking libc++ specific requirements, for example the error // messages of exceptions. -#define TEST_LIBCPP_REQUIRE(CONDITION, MSG) ::test_libcpp_require(CONDITION, #CONDITION, __FILE__, __LINE__, MSG) +// ARG is the same as for TEST_FAIL +#if defined(_LIBCPP_VERSION) +# define TEST_LIBCPP_REQUIRE(CONDITION, ARG) ::test_require(CONDITION, #CONDITION, __FILE__, __LINE__, ARG) +#else +# define TEST_LIBCPP_REQUIRE(...) /* DO NOTHING */ +#endif + +// Helper macro to test an expression does not throw any exception. +#ifndef TEST_HAS_NO_EXCEPTIONS +# define TEST_DOES_NOT_THROW(EXPR) \ + do { \ + try { \ + static_cast(EXPR); \ + } catch (...) { \ + ::test_log(#EXPR, __FILE__, __LINE__, "no exception was expected\n"); \ + ::std::abort(); \ + } \ + } while (false) /* */ + +// Helper macro to test an expression throws an exception of the expected type. +# define TEST_THROWS_TYPE(TYPE, EXPR) \ + do { \ + try { \ + static_cast(EXPR); \ + ::test_log(nullptr, \ + __FILE__, \ + __LINE__, \ + "no exception is thrown while an exception of type " #TYPE " was expected\n"); \ + ::std::abort(); \ + } catch (const TYPE&) { \ + /* DO NOTHING */ \ + } catch (...) { \ + ::test_log(nullptr, \ + __FILE__, \ + __LINE__, \ + "the type of the exception caught differs from the expected type " #TYPE "\n"); \ + ::std::abort(); \ + } \ + } while (false) /* */ + +// Helper macro to test an expression throws an exception of the expected type and satisfies a predicate. +// +// In order to log additional information the predicate can use log macros. +// The exception caught is used as argument to the predicate. +# define TEST_VALIDATE_EXCEPTION(TYPE, PRED, EXPR) \ + do { \ + try { \ + static_cast(EXPR); \ + ::test_log(nullptr, \ + __FILE__, \ + __LINE__, \ + "no exception is thrown while an exception of type " #TYPE " was expected\n"); \ + ::std::abort(); \ + } catch (const TYPE& EXCEPTION) { \ + PRED(EXCEPTION); \ + } catch (...) { \ + ::test_log(nullptr, \ + __FILE__, \ + __LINE__, \ + "the type of the exception caught differs from the expected type " #TYPE "\n"); \ + ::std::abort(); \ + } \ + } while (false) /* */ + +#else // TEST_HAS_NO_EXCEPTIONS +# define TEST_DOES_NOT_THROW(EXPR) static_cast(EXPR); +# define TEST_THROWS_TYPE(...) /* DO NOTHING */ +# define TEST_VALIDATE_EXCEPTION(...) /* DO NOTHING */ +#endif // TEST_HAS_NO_EXCEPTIONS + +#define TEST_IS_UNSUPPORTED(...) return #endif // TEST_SUPPORT_ASSERT_MACROS_H diff --git a/libcxx/test/support/concat_macros.h b/libcxx/test/support/concat_macros.h new file mode 100644 --- /dev/null +++ b/libcxx/test/support/concat_macros.h @@ -0,0 +1,53 @@ +//===----------------------------------------------------------------------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// + +#ifndef TEST_SUPPORT_CONCAT_MACROS_H +#define TEST_SUPPORT_CONCAT_MACROS_H + +#include +#include + +#include "test_macros.h" + +#ifndef TEST_HAS_NO_LOCALIZATION +# include +#endif + +#if TEST_STD_VER > 17 + +# ifndef TEST_HAS_NO_LOCALIZATION +template +concept test_char_streamable = requires(T&& value) { std::stringstream{} << std::forward(value); }; +# endif + +// If possible concatenates message for the assertion function, else returns a +// default message. Not being able to stream is not considered and error. For +// example, streaming to std::wcerr doesn't work properly in the CI. Therefore +// the formatting tests should only stream to std::string. +// +// The macro TEST_WRITE_CONCATENATED can be used to evaluate the arguments +// lazily. This useful when using this function in combination with +// assert_macros.h. +template +std::string test_concat_message([[maybe_unused]] Args&&... args) { +# ifndef TEST_HAS_NO_LOCALIZATION + if constexpr ((test_char_streamable && ...)) { + std::stringstream sstr; + ((sstr << std::forward(args)), ...); + return sstr.str(); + } else +# endif + return "Message discarded since it can't be streamed to std::cerr.\n"; +} + +// Writes its arguments to stderr, using the test_concat_message helper. +# define TEST_WRITE_CONCATENATED(...) [&] { ::std::fprintf(stderr, "%s", ::test_concat_message(__VA_ARGS__).c_str()); } + +#endif // TEST_STD_VER > 17 + +#endif // TEST_SUPPORT_CONCAT_MACROS_H