[libc] Extends the testing framework to support typed test

[gchatelet]

Summary

This patch provides TYPED_TEST and TYPED_TEST_F (similar in functionnality to gtest).
This is needed to extensively test building blocks for memory functions.

Example for TYPED_TEST_F:

template <typename T> class LlvmLibcMyTestFixture : public testing::Test {};

using Types = testing::TypeList<char, int, long>;

TYPED_TEST_F(LlvmLibcMyTestFixture, Simple, Types) {
  EXPECT_LE(sizeof(ParamType), 8UL);
}

Example for TYPED_TEST:

using Types = testing::TypeList<char, int, long>;

TYPED_TEST(LlvmLibcMyTest, Simple, Types) {
  EXPECT_LE(sizeof(ParamType), 8UL);
}

ParamType is displayed as fully qualified canonical type which can be difficult to read, the user can provide a more readable name by using the REGISTER_TYPE_NAME macro.

I noticed that Fuchsia is using its own testing framework. We can either add the same functionality to zxtest or disable them on Fuchsia. WDYT?

[gchatelet]

• Fix typo and static assert message.

[gchatelet]

• Fixed base revision

[sivachandra]

While I am not opposed to this, I want to learn the usefulness of this feature. IIUC, this limits to a single ParamType - as in a uni-dimensional type variance. We have examples in math functions where there is two-dimensional type variance. I am not sure the feature added in this patch can help with it. For such cases, have used template test classes and helper macros to list the various combination of tests: https://github.com/llvm/llvm-project/blob/main/libc/test/src/math/RoundToIntegerTest.h.

[gchatelet]

In D100631#2695126, @sivachandra wrote:

While I am not opposed to this, I want to learn the usefulness of this feature. IIUC, this limits to a single ParamType - as in a uni-dimensional type variance. We have examples in math functions where there is two-dimensional type variance. I am not sure the feature added in this patch can help with it. For such cases, have used template test classes and helper macros to list the various combination of tests: https://github.com/llvm/llvm-project/blob/main/libc/test/src/math/RoundToIntegerTest.h.

Yes I've spotted these tests but I found that it scaled poorly as you define more tests since you have to define the tests into a macro which makes for poor editing.
You ended up moving the body of the test to an external function to help with this but I think it's not ideal (especially for failure location).

In the case of multi-dimensional types you can pack them into a tuple and unpack them in the test fixture. I'll try to implement RoundToIntegerTest with TYPED_TEST and let you know how it looks like. My particular use case is for 19 types and 4 tests (but more to come).

[sivachandra]

In D100631#2695208, @gchatelet wrote:

In D100631#2695126, @sivachandra wrote:

While I am not opposed to this, I want to learn the usefulness of this feature. IIUC, this limits to a single ParamType - as in a uni-dimensional type variance. We have examples in math functions where there is two-dimensional type variance. I am not sure the feature added in this patch can help with it. For such cases, have used template test classes and helper macros to list the various combination of tests: https://github.com/llvm/llvm-project/blob/main/libc/test/src/math/RoundToIntegerTest.h.

Yes I've spotted these tests but I found that it scaled poorly as you define more tests since you have to define the tests into a macro which makes for poor editing.
You ended up moving the body of the test to an external function to help with this but I think it's not ideal (especially for failure location).

There is no problem with failure location - you will get the correct file and line number. With math tests we do not have a problem with scaling also - we want tests for each type combination to be in different files as they correspond to different functions.
Also, I am not sure what you mean by "define tests in a macro". The tests are member functions of the template class. The listing part is done with a macro, which is minor.

In the case of multi-dimensional types you can pack them into a tuple and unpack them in the test fixture. I'll try to implement RoundToIntegerTest with TYPED_TEST and let you know how it looks like. My particular use case is for 19 types and 4 tests (but more to come).

If you find it useful, I am totally fine with it. Lets wait to hear from phosek about adding this to zxtest.

[phosek]

I'll check with zxtest maintainers.

[phosek]

In D100631#2695553, @phosek wrote:

I'll check with zxtest maintainers.

Looks like we already have a few use cases for TYPED_TEST in Fuchsia, this is tracked as fxbug.dev/62890.

[sivachandra]

Accepting this as it looks like zxtest can be extended to have this feature. Irrespective though, as I think more about it, it is definitely of value when listing the differently-typed tests that can all be listed in a single file/single test.

[gchatelet]

• Recover ParamType from PRETTY_FUNCTION instead of RTTI or user provided type name.

[gchatelet]

• Added documentation

[gchatelet]

In D100631#2695607, @phosek wrote:

In D100631#2695553, @phosek wrote:

I'll check with zxtest maintainers.

Looks like we already have a few use cases for TYPED_TEST in Fuchsia, this is tracked as fxbug.dev/62890.

Thx @phosek ! I'll ping on the bug to see how I can bring this to zxtest.

Thx @sivachandra for the LGTM.

[sivachandra]

Note that Fuchsia does not block us from using this feature in LLVM libc source tree as only ctype tests from LLVM libc run in Fuchsia.

[gchatelet]

In D100631#2695941, @sivachandra wrote:

Note that Fuchsia does not block us from using this feature in LLVM libc source tree as only ctype tests from LLVM libc run in Fuchsia.

Ha thx I thought it was a blocker.

libc/utils/UnitTest/LibcTest.h

 private:
   virtual void Run() = 0;
   virtual const char *getName() const = 0;
 
   static Test *Start;
   static Test *End;
 };
 
-} // namespace testing
-} // namespace __llvm_libc
-
-namespace __llvm_libc {
 namespace internal {
+
 constexpr bool same_prefix(char const *lhs, char const *rhs, int const len) {
   for (int i = 0; (*lhs || *rhs) && (i < len); ++lhs, ++rhs, ++i)
     if (*lhs != *rhs)
       return false;
   return true;
 }
+
+constexpr bool valid_prefix(char const *lhs) {
+  return same_prefix(lhs, "LlvmLibc", 8);
+}
+
+// 'str' is a null terminated string of the form
+// "const char *__llvm_libc::testing::internal::GetTypeName() [ParamType = XXX]"
+// We return the substring that start at character '[' or a default message.
+constexpr char const *GetPrettyFunctionParamType(char const *str) {
+  for (const char *ptr = str; *ptr != '\0'; ++ptr)
+    if (*ptr == '[')
+      return ptr;
+  return "UNSET : declare with REGISTER_TYPE_NAME";
+}
+
+// This function recovers ParamType at compile time by using __PRETTY_FUNCTION__
+// It can be customized by using the REGISTER_TYPE_NAME macro below.
+template <typename ParamType> static constexpr const char *GetTypeName() {
+  return GetPrettyFunctionParamType(__PRETTY_FUNCTION__);
+}
+
+template <typename T>
+static inline void GenerateName(char *buffer, int buffer_size,
+                                const char *prefix) {
+  if (buffer_size == 0)
+    return;
+
+  // Make sure string is null terminated.
+  --buffer_size;
+  buffer[buffer_size] = '\0';
+
+  const auto AppendChar = [&](char c) {
+    if (buffer_size > 0) {
+      *buffer = c;
+      ++buffer;
+      --buffer_size;
+    }
+  };
+  const auto AppendStr = [&](const char *str) {
+    for (; str && *str != '\0'; ++str)
+      AppendChar(*str);
+  };
+
+  AppendStr(prefix);
+  AppendChar(' ');
+  AppendStr(GetTypeName<T>());
+  AppendChar('\0');
+}
+
+// TestCreator implements a linear hierarchy of test instances, effectively
+// instanciating all tests with Types in a single object.
+template <template <typename> class TemplatedTestClass, typename... Types>
+struct TestCreator;
+
+template <template <typename> class TemplatedTestClass, typename Head,
+          typename... Tail>
+struct TestCreator<TemplatedTestClass, Head, Tail...>
+    : private TestCreator<TemplatedTestClass, Tail...> {
+  TemplatedTestClass<Head> instance;
+};
+
+template <template <typename> class TemplatedTestClass>
+struct TestCreator<TemplatedTestClass> {};
+
+// A type list to declare the set of types to instantiate the tests with.
+template <typename... Types> struct TypeList {
+  template <template <typename> class TemplatedTestClass> struct Tests {
+    using type = TestCreator<TemplatedTestClass, Types...>;
+  };
+};
+
 } // namespace internal
+
+// Make TypeList visible in __llvm_libc::testing.
+template <typename... Types> using TypeList = internal::TypeList<Types...>;
+
+} // namespace testing
 } // namespace __llvm_libc
 
+// For TYPED_TEST and TYPED_TEST_F below we need to display which type was used
+// to run the test. The default will return the fully qualified canonical type
+// but it can be difficult to read. We provide the following macro to allow the
+// client to register the type name as they see it in the code.
+#define REGISTER_TYPE_NAME(TYPE)                                               \
+  template <>                                                                  \
+  constexpr const char *__llvm_libc::testing::internal::GetTypeName<TYPE>() {  \
+    return "[ParamType = " #TYPE "]";                                          \
+  }
+
+#define TYPED_TEST(SuiteName, TestName, TypeList)                              \
+  static_assert(                                                               \
+      __llvm_libc::testing::internal::valid_prefix(#SuiteName),                \
+      "All LLVM-libc TYPED_TEST suite names must start with 'LlvmLibc'.");     \
+  template <typename T>                                                        \
+  class SuiteName##_##TestName : public __llvm_libc::testing::Test {           \
+  public:                                                                      \
+    using ParamType = T;                                                       \
+    char name[256];                                                            \
+    SuiteName##_##TestName() {                                                 \
+      addTest(this);                                                           \
+      __llvm_libc::testing::internal::GenerateName<T>(                         \
+          name, sizeof(name), #SuiteName "." #TestName);                       \
+    }                                                                          \
+    void Run() override;                                                       \
+    const char *getName() const override { return name; }                      \
+  };                                                                           \
+  TypeList::Tests<SuiteName##_##TestName>::type                                \
+      SuiteName##_##TestName##_Instance;                                       \
+  template <typename T> void SuiteName##_##TestName<T>::Run()
+
+#define TYPED_TEST_F(SuiteClass, TestName, TypeList)                           \
+  static_assert(__llvm_libc::testing::internal::valid_prefix(#SuiteClass),     \
+                "All LLVM-libc TYPED_TEST_F suite class names must start "     \
+                "with 'LlvmLibc'.");                                           \
+  template <typename T> class SuiteClass##_##TestName : public SuiteClass<T> { \
+  public:                                                                      \
+    using ParamType = T;                                                       \
+    char name[256];                                                            \
+    SuiteClass##_##TestName() {                                                \
+      SuiteClass<T>::addTest(this);                                            \
+      __llvm_libc::testing::internal::GenerateName<T>(                         \
+          name, sizeof(name), #SuiteClass "." #TestName);                      \
+    }                                                                          \
+    void Run() override;                                                       \
+    const char *getName() const override { return name; }                      \
+  };                                                                           \
+  TypeList::Tests<SuiteClass##_##TestName>::type                               \
+      SuiteClass##_##TestName##_Instance;                                      \
+  template <typename T> void SuiteClass##_##TestName<T>::Run()
+
 #define TEST(SuiteName, TestName)                                              \
-  static_assert(__llvm_libc::internal::same_prefix(#SuiteName, "LlvmLibc", 8), \
+  static_assert(__llvm_libc::testing::internal::valid_prefix(#SuiteName),      \
                 "All LLVM-libc TEST suite names must start with 'LlvmLibc'."); \
   class SuiteName##_##TestName : public __llvm_libc::testing::Test {           \
   public:                                                                      \
     SuiteName##_##TestName() { addTest(this); }                                \
     void Run() override;                                                       \
     const char *getName() const override { return #SuiteName "." #TestName; }  \
   };                                                                           \
   SuiteName##_##TestName SuiteName##_##TestName##_Instance;                    \
   void SuiteName##_##TestName::Run()
 
 #define TEST_F(SuiteClass, TestName)                                           \
   static_assert(                                                               \
-      __llvm_libc::internal::same_prefix(#SuiteClass, "LlvmLibc", 8),          \
+      __llvm_libc::testing::internal::valid_prefix(#SuiteClass),               \
       "All LLVM-libc TEST_F suite class names must start with 'LlvmLibc'.");   \
   class SuiteClass##_##TestName : public SuiteClass {                          \
   public:                                                                      \
     SuiteClass##_##TestName() { addTest(this); }                               \
     void Run() override;                                                       \
     const char *getName() const override { return #SuiteClass "." #TestName; } \
   };                                                                           \
   SuiteClass##_##TestName SuiteClass##_##TestName##_Instance;                  \