[libc] Add integration tests.

[PaulkaToast]

Summary

This patch aims to add integration tests to check the following:

1. Header files are generated as expected.
2. Libc functions have the correct public name.
3. Libc functions have the correct return type and parameter types.
4. Symbols are exposed in the public lib.a files.

[sivachandra]

Few high level comments and questions:

• Instead of creating a target for every entrypoint, have you considered generating a single file which:
  1. Includes all header files listed in the api.td.
  2. Lists cpp:Function pointers to all functions as required by api.td.

There definitely are problems that need to be solved with this approach. For example, how do you ensure all public header files are already present in the build/.../libc/include directory. But, the appeal of this approach is that one does not have to create explicit or implicit targets for "integration tests". There would be just one target which would test everything in one go.

• In general, I don't think we should expect that internal API to match the public API. For example, we might choose to give internal types names which are different from the public names. So, I think the second kind of tests you mention in your description is not required as it is restrictive.

• Using -nostdinc would require us to explicitly list path to the linux headers for entrypoints which depend on the linux header files, which will very likely bring all headers into the include path. Do you have any ideas on how to get around it?

• Using -nostdlib results in an incomplete link. For ELF, most linkers only give a warning about missing _start symbol, but the resulting exe is bad/incomplete. Do you have any plans to ensure that the link is complete, resulting in an exe which can be executed?

[abrachet]

In D79192#2015295, @sivachandra wrote:

Few high level comments and questions:

• Instead of creating a target for every entrypoint, have you considered generating a single file which:
  1. Includes all header files listed in the api.td.
  2. Lists cpp:Function pointers to all functions as required by api.td.

There definitely are problems that need to be solved with this approach. For example, how do you ensure all public header files are already present in the build/.../libc/include directory. But, the appeal of this approach is that one does not have to create explicit or implicit targets for "integration tests". There would be just one target which would test everything in one go.

+1 to doing this just from api.td. As for the includes, I think it would be wise for us to make the header files dependencies of llvmlibc or maybe better create a libc_headers target to generate all of our headers.

• In general, I don't think we should expect that internal API to match the public API. For example, we might choose to give internal types names which are different from the public names. So, I think the second kind of tests you mention in your description is not required as it is restrictive.

Agreed. I think there are a few places already we use an internal __llvm_libc types I think these would break. That being said because we are doing that, it is important to test that the external api is compatible with the internal one, I cant think of a great way to do that though.

libc/utils/HdrGen/PrototypeTestGen/PrototypeTestGen.cpp

48

It was just recently renamed to Function But I think it makes more semantic sense to use cpp::IsSame<[type], decltype(EntryPointNameOption)>

[PaulkaToast]

In D79192#2015295, @sivachandra wrote:

• Instead of creating a target for every entrypoint, have you considered generating a single file which:
  1. Includes all header files listed in the api.td.
  2. Lists cpp:Function pointers to all functions as required by api.td.

There definitely are problems that need to be solved with this approach. For example, how do you ensure all public header files are already present in the build/.../libc/include directory. But, the appeal of this approach is that one does not have to create explicit or implicit targets for "integration tests". There would be just one target which would test everything in one go.

Good idea, this is done and the new integration test file now looks like this:

...
#include <unistd.h>
#include <sys/mman.h>
#include <math.h>

int main() {
  static_assert(__llvm_libc::cpp::IsSame<int(int), decltype(raise)>::Value, "raise prototype in TableGen does not match public header");
  static_assert(__llvm_libc::cpp::IsSame<int(int, const struct sigaction *__restrict, struct sigaction *__restrict), decltype(sigaction)>::Value, "sigaction prototype in TableGen does not match public header");
  static_assert(__llvm_libc::cpp::IsSame<int(sigset_t *, int), decltype(sigdelset)>::Value, "sigdelset prototype in TableGen does not match public header");
...

Thanks to @abrachet's suggestion of using IsSame we also get nice error messages. As for ensuring all public header files are present I added a top-level target for the headers that we can depend on here (same as @abrachet mentioned)

• In general, I don't think we should expect that internal API to match the public API. For example, we might choose to give internal types names which are different from the public names. So, I think the second kind of tests you mention in your description is not required as it is restrictive.

Done, this only tests the public/libc user perspective now.

• Using -nostdinc would require us to explicitly list path to the linux headers for entrypoints which depend on the linux header files, which will very likely bring all headers into the include path. Do you have any ideas on how to get around it?

For now I've commented out -nostdinc with a TODO, the most straightforward way would be to have our public headers not depend on Linux dev headers but another approach could be to use our clang-tidy RestrictSystemIncludes check to ensure only the right Linux header gets included (this would add a dependency on clang-tidy for the integration tests though)

• Using -nostdlib results in an incomplete link. For ELF, most linkers only give a warning about missing _start symbol, but the resulting exe is bad/incomplete. Do you have any plans to ensure that the link is complete, resulting in an exe which can be executed?

My earlier iteration of this had us linking with the loader for a complete executable link. However, since we now check everything statically, I think it's enough if the integration test compiles and links as this ensures the libc symbols are present and the types are compatible. We could add the loader again if you think it's necessary.

libc/utils/HdrGen/PrototypeTestGen/PrototypeTestGen.cpp

48

Thanks!

[abrachet]

libc/utils/HdrGen/PrototypeTestGen/PrototypeTestGen.cpp

25

Is there a reason for this to be namespaced? Should we use __llvm_libc instead?

28

Unecessary.

31–33

Don't need the brackets

37–42

These could be combined to

auto it = G->FunctionSpecMap.find(entrypoint);
if (it == G->FunctionSpecMap.end()) {} // error
llvm::Record *functionSpec = *it;

so as to not search twice.

46–49

_Noreturn isn't a C++ keyword, otherwise the type system shouldn't care about attributes no matter where they show up. I think instead of removing these manually we should just include __llvm-libc-common.h which will define this as [[noreturn]] we might in the future use other macros in type names.

53

for (size_t i = 0, size = args.size(); i < size; ++i)? I believe the code standards prefers this

76

& not necessary, I believe.

libc/utils/HdrGen/PublicAPICommand.cpp

111

PublicHeaders.emplace(Header)

184

explicit maybe?

274–276

Is this simpler than just using std::make_unique<APIGenerator>(Records); wherever it would be used?

[PaulkaToast]

libc/utils/HdrGen/PrototypeTestGen/PrototypeTestGen.cpp

46–49

We are already expanding this

projects/libc/lib/public_integration_test.cpp:23:42: error: expected variable name or 'this' in lambda capture list
  static_assert(__llvm_libc::cpp::IsSame<_Noreturn void(void), decltype(abort)>::Value, "abort prototype in TableGen does not match public header");
                                         ^
projects/libc/include/__llvm-libc-common.h:21:20: note: expanded from macro '_Noreturn'
#define _Noreturn [[noreturn]]

It seems square bracket [[attributes]] cannot be used on plain types/templates/outside definitions, the compiler thinks we are trying to define a lambda function.
https://godbolt.org/z/bMFzfy

Trying to use [[attributes]] in a using statement results in a similar error.

libc/utils/HdrGen/PublicAPICommand.cpp

274–276

I didn't want to take the whole class and expose it, just the parts I needed.

[sivachandra]

In D79192#2024575, @PaulkaToast wrote:

...
#include <unistd.h>
#include <sys/mman.h>
#include <math.h>

int main() {
  static_assert(__llvm_libc::cpp::IsSame<int(int), decltype(raise)>::Value, "raise prototype in TableGen does not match public header");
  static_assert(__llvm_libc::cpp::IsSame<int(int, const struct sigaction *__restrict, struct sigaction *__restrict), decltype(sigaction)>::Value, "sigaction prototype in TableGen does not match public header");
  static_assert(__llvm_libc::cpp::IsSame<int(sigset_t *, int), decltype(sigdelset)>::Value, "sigdelset prototype in TableGen does not match public header");
...

Thanks to @abrachet's suggestion of using IsSame we also get nice error messages. As for ensuring all public header files are present I added a top-level target for the headers that we can depend on here (same as @abrachet mentioned)

LG

• Using -nostdinc would require us to explicitly list path to the linux headers for entrypoints which depend on the linux header files, which will very likely bring all headers into the include path. Do you have any ideas on how to get around it?

For now I've commented out -nostdinc with a TODO, the most straightforward way would be to have our public headers not depend on Linux dev headers but another approach could be to use our clang-tidy RestrictSystemIncludes check to ensure only the right Linux header gets included (this would add a dependency on clang-tidy for the integration tests though)

I think we should be complete here. Lets split this change into two changes:

1. First change will add a target to build the main object but not the full executable. In this change, we will make use of clang-tidy to ensure only LLVM-libc or platform or compiler headers are being included.
2. Second change will add a target to link this main object with the loader object making it a true integration test.

Before you make changes, I want to bring up few other high-level points for discussion:

1. The list of public headers should be determined by the target config. However, we don't have a way for a config to list the public headers it wants. We can take two approaches here again: provide a rule to list a target for public headers, or just list the targets for the headers in .txt file. I would prefer keeping it simple and just requiring .txt file which defines a CMake list of targets for public headers. The CMake file listing the integration test target will include this file to set up the dependencies. Something like this:

# config/linux/headers.txt:

set(PUBLIC_HEADERS
  libc.include.math
  libc.inlcude.signal
  ...
)

And:

# integration_test/CMakeLists.txt
include("config/${LIBC_TARGET_OS}/headers.txt")
...

2. The list of entrypoints that go into a .a file should also be determined by the target config. This can also be done by listing the entrypoint targets in a, say entrypoints.txt file. The file lib/CMakeLists.txt can include this file and get the entrypoints for the various static libraries, just like above. This mechanism can also help us skip building and testing entrypoint targets if a config doesn't list it in its entrypoints.txt file.

WDYT?

[PaulkaToast]

Sorry for the delayed response!

I like idea of a .txt for both the headers and the entrypoints. This is nice because it also prevents us from building and including platform specific headers e.g fcntl.h which is on Linux but not on Windows.
Will send out a revised diff soon. (:

As for the Linux dev headers I think instead of a work around using clang-tidy, it might just be better to remove our dependency on the dev headers all together, especially since the dev headers differ between platforms such as sigset_t on AARCH64 vs x86.
I can send out a separate patch out for this as well.

[sivachandra]

In D79192#2048277, @PaulkaToast wrote:

As for the Linux dev headers I think instead of a work around using clang-tidy, it might just be better to remove our dependency on the dev headers all together, especially since the dev headers differ between platforms such as sigset_t on AARCH64 vs x86.
I can send out a separate patch out for this as well.

What do you mean remove the dependency on dev headers? Does it mean that we make copies of linux definitions in LLVM libc tree?

[PaulkaToast]

In D79192#2048306, @sivachandra wrote:

In D79192#2048277, @PaulkaToast wrote:

As for the Linux dev headers I think instead of a work around using clang-tidy, it might just be better to remove our dependency on the dev headers all together, especially since the dev headers differ between platforms such as sigset_t on AARCH64 vs x86.
I can send out a separate patch out for this as well.

What do you mean remove the dependency on dev headers? Does it mean that we make copies of linux definitions in LLVM libc tree?

As we discussed offline, the approach I was thinking would involve duplicating a lot of definitions from the linux headers. So I decided to take the approach you suggested and use clang-tidy to ensure we're including the correct headers in the test.

[sivachandra]

libc/CMakeLists.txt

85 ↗

(On Diff #266719)

Why is this change required?

libc/cmake/modules/LLVMLibCLibraryRules.cmake

59

Call this entry point *name* list throughout to avoid confusion with target names.

libc/cmake/modules/LLVMLibCObjectRules.cmake

1 ↗

(On Diff #266719)

Seems to me like the changes in this file should be separated out into a different patch.

libc/config/linux/entrypoints.txt

1 ↗

(On Diff #266719)

This LGTM. But, do you think it makes sense to put this in a target machine specific directory (config/linux/x86_64/entrypoints.txt)? For example, almost everything under LIBC_ENTRYPOINTS currently is x86_64 specific. So, putting these in a machine specific list helps us build the libc for different machine types independent of each of other.

libc/config/linux/headers.txt

1 ↗

(On Diff #266719)

The same would be true for headers also I suppose but we can do that separately as it does not interfere with the current patch?

libc/lib/CMakeLists.txt

22 ↗

(On Diff #266719)

AFAICT, this is the only place this rule is being used. Do you see it being used anywhere else? If not, just inline everything here. May be even put it under the test directory and not here?

libc/utils/HdrGen/PublicAPICommand.h

42

I see only one specialization of this class. In which case, why is this need?

[PaulkaToast]

libc/CMakeLists.txt

85 ↗

(On Diff #266719)

Due to the changes below its no longer required (:

libc/cmake/modules/LLVMLibCObjectRules.cmake

1 ↗

(On Diff #266719)

Yes, I could pull it out into another patch but this is the only current use case for this change so maybe its better to leave it in this patch? wdyt?

libc/config/linux/entrypoints.txt

1 ↗

(On Diff #266719)

I'm not necessarily opposed to this, although I'm not sure I know of a case where the libc differs for different architectures on linux. If the need arises it should be a very simple change to make if/when we encounter it.

libc/config/linux/headers.txt

1 ↗

(On Diff #266719)

yes, we can evaluate this in a different patch.

libc/lib/CMakeLists.txt

22 ↗

(On Diff #266719)

Ah, yes this was useful for when I was testing individual libraries but now there is no need for the function so I'll inline it instead. This also removes the reordering issue from above. Thanks!

libc/utils/HdrGen/PublicAPICommand.h

42

Right now the APIGenerator is defined in the .cpp file, so this is used to expose the parts we want. The alternative is to expose the APIGenerator class in the header, but that'd be a larger refactor. It can be done though, which would you prefer?

[sivachandra]

libc/cmake/modules/LLVMLibCObjectRules.cmake

1 ↗

(On Diff #266719)

OK

libc/config/linux/entrypoints.txt

1 ↗

(On Diff #266719)

My comment is more for our convenience while we are building the libc for different target machine architectures. Having just one linux list means that while the implementation for other architectures is incomplete, our build infrastructure will keep failing at different places.

libc/utils/HdrGen/PublicAPICommand.h

42

Yes. Refactor it separately. It need not really be a refactor. I am OK even if you just make API generator public. As it stands in this patch, the specialization and the base class have no meaning as makeAPIReader is really not a factory method.

Splitting the CMake parts and the hdrgen parts into two patches makes it easier for me to review as well.

[PaulkaToast]

libc/config/linux/entrypoints.txt

1 ↗

(On Diff #266719)

Ah, this is a good point. If you are porting to another architecture and building it out its much more convenient to do it in pieces, a few entrpoints at a time, which multiple lists allow. Done.

libc/utils/HdrGen/PublicAPICommand.h

42

Okay, this is now pulled into a desperate patch: D80832.
When that patch gets merged in I'll pull it in here and update the diff.

[sivachandra]

libc/utils/HdrGen/PublicAPICommand.h

42

[This is very likely a typo but funny.]
No, my intention was not to make you feel desperate.

[PaulkaToast]

libc/utils/HdrGen/PublicAPICommand.h

42

I'm so sorry that was a typo, meant to write "separate". 😅

[sivachandra]

Accepting now. The one comment I added can be addressed separately.

libc/test/src/CMakeLists.txt

96 ↗

(On Diff #267708)

Why not include the loader here?

[PaulkaToast]

libc/test/src/CMakeLists.txt

96 ↗

(On Diff #267708)

Yes, this will be incorporated in part 2, as per this comment.

https://reviews.llvm.org/D79192#2027676

I think we should be complete here. Lets split this change into two changes:

1. First change will add a target to build the main object but not the full executable. In this change, we will make use of clang-tidy to ensure only LLVM-libc or platform or compiler headers are being included.
2. Second change will add a target to link this main object with the loader object making it a true integration test.

libc/cmake/modules/LLVMLibCLibraryRules.cmake

function(add_entrypoint_library target_name)
   )
   if(NOT ENTRYPOINT_LIBRARY_DEPENDS)
     message(FATAL_ERROR "'add_entrypoint_library' target requires a DEPENDS list "
                         "of 'add_entrypoint_object' targets.")
   endif()
 
   get_fq_deps_list(fq_deps_list ${ENTRYPOINT_LIBRARY_DEPENDS})
   get_object_files_for_entrypoint_library(obj_list ${fq_deps_list})
   foreach(dep IN LISTS fq_deps_list)
     get_target_property(dep_type ${dep} "TARGET_TYPE")
     if(NOT (${dep_type} STREQUAL ${ENTRYPOINT_OBJ_TARGET_TYPE}))
       message(FATAL_ERROR "Dependency '${dep}' of 'add_entrypoint_collection' is "
                           "not an 'add_entrypoint_object' target.")
     endif()
     get_target_property(objfile ${dep} "OBJECT_FILE")
     list(APPEND obj_list ${objfile})
   endforeach(dep)
   list(REMOVE_DUPLICATES obj_list)
 
   set(library_file "${CMAKE_CURRENT_BINARY_DIR}/${CMAKE_STATIC_LIBRARY_PREFIX}${target_name}${CMAKE_STATIC_LIBRARY_SUFFIX}")
   add_custom_command(
     OUTPUT ${library_file}
     COMMAND ${CMAKE_AR} -r ${library_file} ${obj_list}
     DEPENDS ${obj_list}
   )
   add_custom_target(
     ${target_name}
     ALL
     DEPENDS ${library_file}
   )
+  set_target_properties(
+    ${target_name}
+    PROPERTIES LIBRARY_FILE ${library_file}
+  )
 endfunction(add_entrypoint_library)
 
 # Rule to build a shared library of redirector objects.
 function(add_redirector_library target_name)
   cmake_parse_arguments(
     "REDIRECTOR_LIBRARY"
     ""
     ""

libc/cmake/modules/LLVMLibCTestRules.cmake

set_target_properties(${fq_target_name}
       PROPERTIES RUNTIME_OUTPUT_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})
 
   add_dependencies(
     ${fq_target_name}
     ${fq_deps_list}
   )
   add_dependencies(libc-fuzzer ${fq_target_name})
 endfunction(add_libc_fuzzer)
+
+# Rule to add a libc integration test.
+# Usage
+#    add_libc_integration_test(
+#      <target name>
+#      SUITE          <name of the suite this test belongs to>
+#      ENTRYPOINT     <name of entrypoint the test is for>
+#      DEPENDS        <list of dependencies>
+#      INTERNAL_HDR   <internal .h files for the test>
+#      PUBLIC_HDR     <public .h files for the test>
+#      LIB            <list of public libraries to link against>
+#    )
+function(add_libc_integration_tests target_name)
+  if(NOT LLVM_INCLUDE_TESTS)
+    return()
+  endif()
+
+  cmake_parse_arguments(
+    "LIBC_INTTEST"
+    "" # No optional arguments
+    "SUITE;ENTRYPOINT;INTERNAL_HDR;PUBLIC_HDR" # Single value arguments
+    "DEPENDS;LIB" # No Multi value arguments
+    ${ARGN}
+  )
+  if(NOT LIBC_INTTEST_ENTRYPOINT)
+    message(FATAL_ERROR "'add_libc_integration_test' target requires a ENTRYPOINT "
+                        "the name of the entrypoint.")
+  endif()
+  if(NOT LIBC_INTTEST_INTERNAL_HDR)
+    message(FATAL_ERROR "'add_libc_integration_test' target requires a INTERNAL_HDR "
+                        "a internal .h file.")
+  endif()
+  if(NOT LIBC_INTTEST_PUBLIC_HDR)
+    message(FATAL_ERROR "'add_libc_integration_test' target requires a PUBLIC_HDR "
+                        "a public .h file.")
+  endif()
+  if(NOT LIBC_INTTEST_LIB)
+    message(FATAL_ERROR "'add_libc_integration_test' target requires a LIB "
+                        "public library to link against.")
+  endif()
+
+  get_fq_target_name(${target_name} fq_target_name)
+
+  add_libc_public_integration_test(
+    ${fq_target_name}
+    ENTRYPOINT
+      ${LIBC_INTTEST_ENTRYPOINT}
+    PUBLIC_HDR
+      ${LIBC_INTTEST_PUBLIC_HDR}
+    LIB
+      ${LIBC_INTTEST_LIB}
+  )
+
+  add_libc_internal_integration_test(
+    ${fq_target_name}
+    ENTRYPOINT
+      ${LIBC_INTTEST_ENTRYPOINT}
+    PUBLIC_HDR
+      ${LIBC_INTTEST_PUBLIC_HDR}
+    INTERNAL_HDR
+      ${LIBC_INTTEST_INTERNAL_HDR}
+    DEPENDS
+    ${LIBC_INTTEST_DEPENDS}
+  )
+
+  if(LIBC_INTTEST_SUITE)
+  add_dependencies(
+    ${LIBC_INTTEST_SUITE}
+    ${fq_target_name}-internal
+    ${fq_target_name}-public
+  )
+endif()
+endfunction(add_libc_integration_tests)
+
+# Rule to add a libc public integration test.
+# Usage
+#    add_libc_public_integration_test(
+#      <target name>
+#      ENTRYPOINT     <name of entrypoint the test is for>
+#      PUBLIC_HDR     <public .h files for the test>
+#      LIB            <list of public libraries to link against>
+#    )
+function(add_libc_public_integration_test target_name)
+if(NOT LLVM_INCLUDE_TESTS)
+    return()
+  endif()
+
+  cmake_parse_arguments(
+    "LIBC_INTTEST"
+    "" # No option arguments
+    "ENTRYPOINT;PUBLIC_HDR" # Single value arguments
+    "LIB" # Multi value arguments
+    ${ARGN}
+  )
+
+  set(public_test ${CMAKE_CURRENT_BINARY_DIR}/${target_name}-public-test.cpp)
+
+  # Generate integration test souce code
+  add_custom_command(
+    OUTPUT ${public_test}
+    COMMAND $<TARGET_FILE:libc-prototype-testgen> -o ${public_test} --public
+            --header ${LIBC_INTTEST_PUBLIC_HDR}
+            --entrypoint ${LIBC_INTTEST_ENTRYPOINT}
+            --public-header ${LIBC_INTTEST_PUBLIC_HDR}
+            -I ${LIBC_SOURCE_DIR}
+            ${LIBC_SOURCE_DIR}/config/${LIBC_TARGET_OS}/api.td
+
+    DEPENDS ${LIBC_SOURCE_DIR}/config/${LIBC_TARGET_OS}/api.td
+            libc-prototype-testgen
+  )
+
+  add_executable(
+    ${target_name}-public
+    EXCLUDE_FROM_ALL
+    ${public_test}
+  )
+  # Put these into a different folder to not pollute the lib folder
+  set_target_properties(${target_name}-public PROPERTIES
+    LIBRARY_OUTPUT_DIRECTORY "${LIBC_BUILD_DIR}/integration_tests/"
+  )
+  # Blank out default include directories to prevent accidentally including
+  # system headers or our own internal headers.
+  set_target_properties(
+    ${target_name}-public
+    PROPERTIES
+    INCLUDE_DIRECTORIES ""
+  )
+  # Only includes we need are is the include for cpp::function
+  # and our generated public headers.
+  target_include_directories(
+    ${target_name}-public BEFORE
+    PRIVATE
+      "${LIBC_SOURCE_DIR}/utils/CPP"
+  )
+  target_include_directories(
+    ${target_name}-public SYSTEM BEFORE
+    PRIVATE
+      "${LIBC_BUILD_DIR}/include"
+  )
+  get_compiler_include_dir()
+  target_compile_options(
+    ${target_name}-public
+    PRIVATE "-nostdinc"
+    # There is currently a bug that prevents us from adding the compilers
+    # include directory using target_include_directories so we must add it in
+    # the compiler flags ourselves.
+    # See: https://gitlab.kitware.com/cmake/cmake/issues/19227
+    "-isystem" ${COMPILER_INCLUDE_PATH}
+  )
+  target_link_options(
+    ${target_name}-public
+    PRIVATE "-nostdlib"
+  )
+  set(library_files)
+  foreach(library_name IN LISTS LIBC_INTTEST_LIB)
+    get_target_property(library_file ${library_name} "LIBRARY_FILE")
+    list(APPEND library_files ${library_file})
+  endforeach()
+
+  target_link_libraries(${target_name}-public
+    PRIVATE
+    ${library_files}
+  )
+endfunction(add_libc_public_integration_test)
+
+# Rule to add a libc internal integration test.
+# Usage
+#    add_libc_internal_integration_test(
+#      <target name>
+#      ENTRYPOINT     <name of entrypoint the test is for>
+#      PUBLIC_HDR     <public .h files for the test>
+#      INTERNAL_HDR   <internal .h files for the test>
+#      DEPENDS        <list of dependencies>
+#    )
+function(add_libc_internal_integration_test target_name)
+  if(NOT LLVM_INCLUDE_TESTS)
+    return()
+  endif()
+
+  cmake_parse_arguments(
+    "LIBC_INTTEST"
+    "" # No option arguments
+    "ENTRYPOINT;INTERNAL_HDR;PUBLIC_HDR" # Single value arguments
+    "DEPENDS" # Multi value arguments
+    ${ARGN}
+  )
+
+  set(internal_test ${CMAKE_CURRENT_BINARY_DIR}/${target_name}-internal-test.cpp)
+
+  # Generate integration test souce code
+  add_custom_command(
+    OUTPUT ${internal_test}
+    COMMAND $<TARGET_FILE:libc-prototype-testgen> -o ${internal_test}
+            --header ${LIBC_INTTEST_INTERNAL_HDR}
+            --entrypoint ${LIBC_INTTEST_ENTRYPOINT}
+            --public-header ${LIBC_INTTEST_PUBLIC_HDR}
+            -I ${LIBC_SOURCE_DIR}
+            ${LIBC_SOURCE_DIR}/config/${LIBC_TARGET_OS}/api.td
+
+    DEPENDS ${LIBC_SOURCE_DIR}/config/${LIBC_TARGET_OS}/api.td
+            libc-prototype-testgen
+  )
+
+  add_executable(
+    ${target_name}-internal
+    EXCLUDE_FROM_ALL
+    ${internal_test}
+  )
+  # Put these into a different folder to not pollute the lib folder
+  set_target_properties(${target_name}-internal PROPERTIES
+         LIBRARY_OUTPUT_DIRECTORY "${LIBC_BUILD_DIR}/tests/integration_tests"
+  )
+  target_include_directories(
+    ${target_name}-internal BEFORE
+    PRIVATE
+      "${LIBC_SOURCE_DIR}/utils/CPP"
+      "${LIBC_BUILD_DIR}/include"
+      "${LIBC_SOURCE_DIR}"
+      "${LIBC_BUILD_DIR}"
+  )
+  get_fq_deps_list(fq_deps_list ${LIBC_INTTEST_DEPENDS})
+  get_object_files_for_test(link_object_files ${fq_deps_list})
+  target_link_libraries(${target_name}-internal PRIVATE ${link_object_files})
+endfunction(add_libc_internal_integration_test)
+
+# This function is needed because we want to compile with -nostdinc but still
+# have access to free-standing heaers provided by the compiler such as
+# stddef.h, stdatomic.h etc.
+#
+# We can't use -ibuiltininc because it is only available on clang 11.
+function(get_compiler_include_dir)
+  if(COMPILER_INCLUDE_PATH)
+    return()
+  endif()
+
+  # Use --print-resource-dir to find the compiler include path if this flag
+  # is supported by the compiler.
+  execute_process(
+    OUTPUT_STRIP_TRAILING_WHITESPACE
+    COMMAND ${CMAKE_CXX_COMPILER} --print-resource-dir
+    RESULT_VARIABLE COMMAND_RETURN_CODE
+    OUTPUT_VARIABLE COMPILER_RESOURCE_DIR
+  )
+  if(COMMAND_RETURN_CODE EQUAL 0)
+    set(COMPILER_INCLUDE_PATH
+      "${COMPILER_RESOURCE_DIR}/include" CACHE PATH "path to compiler include dir"
+    )
+    message(STATUS "Set COMPILER_INCLUDE_PATH to
+                    ${COMPILER_INCLUDE_PATH} using --print-resource-dir")
+    return()
+  endif()
+
+  # Fallback if --print-resource-dir is not supported by the compiler.
+  # TODO, see if there is a better way to do this.
+  find_path(stddef_file_loc stddef.h)
+  set(COMPILER_INCLUDE_PATH
+    "${stddef_file_loc}" CACHE PATH "path to compiler include dir"
+  )
+  message(STATUS "Set COMPILER_INCLUDE_PATH to
+                  ${COMPILER_INCLUDE_PATH} using location of stddef.h")
+
+endfunction(get_compiler_include_dir)

libc/test/src/assert/CMakeLists.txt

-add_libc_testsuite(libc_assert_unittests)
+add_libc_testsuite(libc_assert_tests)
 
 add_libc_unittest(
   assert_test
   SUITE
-    libc_assert_unittests
+    libc_assert_tests
   SRCS
     assert_test.cpp
   DEPENDS
     libc.src.assert.__assert_fail
     # These are necessary for now because dependencies are not properly added.
     libc.src.signal.raise
     libc.src.stdlib._Exit
     libc.src.stdlib.abort
 )

libc/test/src/math/CMakeLists.txt

-add_libc_testsuite(libc_math_unittests)
+add_libc_testsuite(libc_math_tests)
 
 function(add_math_unittest name)
   cmake_parse_arguments(
     "MATH_UNITTEST"
     "NEED_MPFR" # No optional arguments
     "" # Single value arguments
     "" # Multi-value arguments
     ${ARGN}
add_header_library(
   HDRS
     float.h
 )
 
 add_math_unittest(
   cosf_test
   NEED_MPFR
   SUITE
-    libc_math_unittests
+    libc_math_tests
   SRCS
     cosf_test.cpp
   HDRS
     sdcomp26094.h
   DEPENDS
     .float_utils
     libc.src.math.cosf
     libc.utils.CPP.standalone_cpp
 )
+add_libc_integration_tests(
+  cosf_integration_test
+  SUITE
+    libc_math_tests
+  ENTRYPOINT
+    cosf
+  DEPENDS
+    libc.src.math.cosf
+  PUBLIC_HDR
+    math.h
+  INTERNAL_HDR
+    src/math/cosf.h
+  LIB
+    llvmlibm
+    llvmlibc
+)
 
 add_math_unittest(
   sinf_test
   NEED_MPFR
   SUITE
-    libc_math_unittests
+    libc_math_tests
   SRCS
     sinf_test.cpp
   HDRS
     sdcomp26094.h
   DEPENDS
     .float_utils
     libc.src.math.sinf
     libc.utils.CPP.standalone_cpp
 )
+add_libc_integration_tests(
+  sinf_integration_test
+  SUITE
+    libc_math_tests
+  ENTRYPOINT
+    sinf
+  DEPENDS
+    libc.src.math.sinf
+  PUBLIC_HDR
+    math.h
+  INTERNAL_HDR
+    src/math/sinf.h
+  LIB
+    llvmlibm
+    llvmlibc
+)
 
 add_math_unittest(
   sincosf_test
   NEED_MPFR
   SUITE
-    libc_math_unittests
+    libc_math_tests
   SRCS
     sincosf_test.cpp
   HDRS
     sdcomp26094.h
   DEPENDS
     .float_utils
     libc.src.math.sincosf
     libc.utils.CPP.standalone_cpp
 )
+add_libc_integration_tests(
+  sincosf_integration_test
+  SUITE
+    libc_math_tests
+  ENTRYPOINT
+    sincosf
+  DEPENDS
+    libc.src.math.sincosf
+  PUBLIC_HDR
+    math.h
+  INTERNAL_HDR
+    src/math/sincosf.h
+  LIB
+    llvmlibm
+    llvmlibc
+)

libc/test/src/string/CMakeLists.txt

-add_libc_testsuite(libc_string_unittests)
+add_libc_testsuite(libc_string_tests)
 
 add_subdirectory(memory_utils)
 
 add_libc_unittest(
   strcat_test
   SUITE
-    libc_string_unittests
+    libc_string_tests
   SRCS
     strcat_test.cpp
   DEPENDS
     libc.src.string.strcat
 )
+add_libc_integration_tests(
+  strcat_integration_test
+  SUITE
+    libc_string_tests
+  ENTRYPOINT
+    strcat
+  DEPENDS
+    libc.src.string.strcat
+  PUBLIC_HDR
+    string.h
+  INTERNAL_HDR
+    src/string/strcat.h
+  LIB
+    llvmlibc
+)
 
 add_libc_unittest(
   strcpy_test
   SUITE
-    libc_string_unittests
+    libc_string_tests
   SRCS
     strcpy_test.cpp
   DEPENDS
     libc.src.string.strcpy
 )
+add_libc_integration_tests(
+  strcpy_integration_test
+  SUITE
+    libc_string_tests
+  ENTRYPOINT
+    strcpy
+  DEPENDS
+    libc.src.string.strcpy
+  PUBLIC_HDR
+    string.h
+  INTERNAL_HDR
+    src/string/strcpy.h
+  LIB
+    llvmlibc
+)
 
 add_libc_unittest(
   strlen_test
   SUITE
-    libc_string_unittests
+    libc_string_tests
   SRCS
     strlen_test.cpp
   DEPENDS
     libc.src.string.strlen
 )
+add_libc_integration_tests(
+  strlen_integration_test
+  SUITE
+    libc_string_tests
+  ENTRYPOINT
+    strlen
+  DEPENDS
+    libc.src.string.strlen
+  PUBLIC_HDR
+    string.h
+  INTERNAL_HDR
+    src/string/strlen.h
+  LIB
+    llvmlibc
+)
 
 # Tests all implementations of memcpy that can run on the host.
 get_property(memcpy_implementations GLOBAL PROPERTY memcpy_implementations)
 foreach(memcpy_config_name IN LISTS memcpy_implementations)
   get_target_property(require_cpu_features libc.src.string.${memcpy_config_name} REQUIRE_CPU_FEATURES)
   host_supports(can_run "${require_cpu_features}")
   if(can_run)
     add_libc_unittest(
       ${memcpy_config_name}_test
       SUITE
-        libc_string_unittests
+        libc_string_tests
       SRCS
         memcpy_test.cpp
       DEPENDS
         libc.src.string.${memcpy_config_name}
     )
   else()
     message(STATUS "Skipping test for '${memcpy_config_name}' insufficient host cpu features")
   endif()
 endforeach()
 
+add_libc_integration_tests(
+  memcpy_integration_test
+  SUITE
+    libc_string_tests
+  ENTRYPOINT
+    memcpy
+  DEPENDS
+    libc.src.string.memcpy
+  PUBLIC_HDR
+    string.h
+  INTERNAL_HDR
+    src/string/memcpy.h
+  LIB
+    llvmlibc
+)
+

libc/test/src/string/memory_utils/CMakeLists.txt

 add_libc_unittest(
   utils_test
   SUITE
-    libc_string_unittests
+    libc_string_tests
   SRCS
     utils_test.cpp
     memcpy_utils_test.cpp
   DEPENDS
     libc.src.string.memory_utils.memory_utils
     libc.utils.CPP.standalone_cpp
 )
 
 target_compile_definitions(
   libc.test.src.string.memory_utils.utils_test
   PRIVATE
   LLVM_LIBC_MEMCPY_MONITOR=memcpy_monitor
 )

libc/utils/HdrGen/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS Support)
 
 add_tablegen(libc-hdrgen llvm-libc
   Command.h
   Command.cpp
   Generator.cpp
   Generator.h
   IncludeFileCommand.cpp
   IncludeFileCommand.h
   Main.cpp
   PublicAPICommand.cpp
   PublicAPICommand.h
 )
+
+add_subdirectory(PrototypeTestGen)

libc/utils/HdrGen/PrototypeTestGen/CMakeLists.txt

+add_tablegen(libc-prototype-testgen llvm-libc
+  PrototypeTestGen.cpp
+  ../PublicAPICommand.cpp
+  ../Command.cpp
+)

libc/utils/HdrGen/PrototypeTestGen/PrototypeTestGen.cpp

+//===-- PrototypeTestGen.cpp ----------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "../PublicAPICommand.h"
+
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/TableGen/Main.h"
+#include "llvm/TableGen/Record.h"
+
+namespace {
+
+llvm::cl::opt<std::string> EntrypointNameOption(
+    "entrypoint",
+    llvm::cl::desc("The entrypoint we want to generate a test for."),
+    llvm::cl::Required);
+llvm::cl::opt<std::string> HeaderFileOption(
+    "header",
+    llvm::cl::desc("The header file to include in the generated test."),
+    llvm::cl::Required);
+llvm::cl::opt<std::string> PublicHeaderFileOption(
+    "public-header",
+    llvm::cl::desc("The public header file that contains this entrypoint."),
+    llvm::cl::Required);
+llvm::cl::opt<bool>
+    PublicOption("public",
+                 llvm::cl::desc("Generate test using public symbol name."));
+
+} // anonymous namespace
+
+namespace llvm_libc {
+
+bool TestGeneratorMain(llvm::raw_ostream &OS, llvm::RecordKeeper &records) {
+  OS << "#include \"Functional.h\"\n";
+  OS << "#include ";
+  if (PublicOption)
+    OS << '<' << PublicHeaderFileOption << '>';
+  else
+    OS << '"' << HeaderFileOption << '"';
+  OS << "\n\n";
+
+  OS << "int main() {\n";
+  OS << "  __llvm_libc::cpp::function<";
+  auto G = makeAPIReader(PublicHeaderFileOption, records);
+  if (G->FunctionSpecMap.count(EntrypointNameOption) == 0) {
+    llvm::errs() << "ERROR: entrypoint '" << EntrypointNameOption
+                 << "' could not be found in spec for public header '"
+                 << PublicHeaderFileOption << "'\n";
+    return true;
+  }
+
+  llvm::Record *functionSpec = G->FunctionSpecMap[EntrypointNameOption];
+  llvm::Record *retValSpec = functionSpec->getValueAsDef("Return");
+  llvm::Record *returnType = retValSpec->getValueAsDef("ReturnType");
+  OS << G->getTypeAsString(returnType);
+  OS << '(';
+  auto args = functionSpec->getValueAsListOfDefs("Args");
+  for (size_t i = 0; i < args.size(); ++i) {
+    llvm::Record *argType = args[i]->getValueAsDef("ArgType");
+    OS << G->getTypeAsString(argType);
+    if (i < args.size() - 1)
+      OS << ", ";
+  }
+  OS << ")> f(";
+  if (!PublicOption) {
+    OS << "__llvm_libc::";
+  }
+  OS << EntrypointNameOption << ");\n";
+
+  OS << "  (void) f;\n";
+  OS << "  return 0;\n";
+  OS << "}\n\n";
+
+  return false;
+}
+
+} // namespace llvm_libc
+
+int main(int argc, char *argv[]) {
+  llvm::cl::ParseCommandLineOptions(argc, argv);
+  return TableGenMain(argv[0], &llvm_libc::TestGeneratorMain);
+}

libc/utils/HdrGen/PublicAPICommand.h

 namespace llvm {
 
 class raw_ostream;
 class Record;
 class RecordKeeper;
 
 } // namespace llvm
 
+using NameToRecordMapping = std::unordered_map<std::string, llvm::Record *>;
+using NameSet = std::unordered_set<std::string>;
+
 namespace llvm_libc {
 
 class PublicAPICommand : public Command {
 public:
   static const char Name[];
 
   void run(llvm::raw_ostream &OS, const ArgVector &Args,
            llvm::StringRef StdHeader, llvm::RecordKeeper &Records,
            const Command::ErrorReporter &Reporter) const override;
 };
 
+class APIReader {
+public:
+  // Mapping from names to records defining them.
+  NameToRecordMapping MacroSpecMap;
+  NameToRecordMapping TypeSpecMap;
+  NameToRecordMapping EnumerationSpecMap;
+  NameToRecordMapping FunctionSpecMap;
+  NameToRecordMapping MacroDefsMap;
+  NameToRecordMapping TypeDeclsMap;
+
+  NameSet Structs;
+  NameSet Enumerations;
+  NameSet Functions;
+
+  virtual std::string getTypeAsString(llvm::Record *TypeRecord) = 0;
+
+  virtual ~APIReader() {}
+};
+
+std::unique_ptr<APIReader> makeAPIReader(llvm::StringRef StdHeader,
+                                         llvm::RecordKeeper &Records);
+
 } // namespace llvm_libc
 
 #endif // LLVM_LIBC_UTILS_HDRGEN_PUBLICAPICOMMAND_H

libc/utils/HdrGen/PublicAPICommand.cpp

if (IndentSize < shortest_indent)
       shortest_indent = IndentSize;
   }
   for (llvm::StringRef L : Lines) {
     if (L.size() >= shortest_indent)
       OS << L.drop_front(shortest_indent) << '\n';
   }
 }
 
-class APIGenerator {
+class APIGenerator : public llvm_libc::APIReader {
   llvm::StringRef StdHeader;
 
   // TableGen classes in spec.td.
   llvm::Record *NamedTypeClass;
   llvm::Record *PtrTypeClass;
   llvm::Record *RestrictedPtrTypeClass;
   llvm::Record *ConstTypeClass;
   llvm::Record *StructClass;
   llvm::Record *StandardSpecClass;
   llvm::Record *PublicAPIClass;
 
-  using NameToRecordMapping = std::unordered_map<std::string, llvm::Record *>;
-  using NameSet = std::unordered_set<std::string>;
-
-  // Mapping from names to records defining them.
-  NameToRecordMapping MacroSpecMap;
-  NameToRecordMapping TypeSpecMap;
-  NameToRecordMapping EnumerationSpecMap;
-  NameToRecordMapping FunctionSpecMap;
-  NameToRecordMapping MacroDefsMap;
-  NameToRecordMapping TypeDeclsMap;
-
-  NameSet Structs;
-  NameSet Enumerations;
-  NameSet Functions;
-
   bool isaNamedType(llvm::Record *Def) { return isa(Def, NamedTypeClass); }
 
   bool isaStructType(llvm::Record *Def) { return isa(Def, StructClass); }
 
   bool isaPtrType(llvm::Record *Def) { return isa(Def, PtrTypeClass); }
 
   bool isaConstType(llvm::Record *Def) { return isa(Def, ConstTypeClass); }
 
   bool isaRestrictedPtrType(llvm::Record *Def) {
     return isa(Def, RestrictedPtrTypeClass);
   }
 
   bool isaStandardSpec(llvm::Record *Def) {
     return isa(Def, StandardSpecClass);
   }
 
   bool isaPublicAPI(llvm::Record *Def) { return isa(Def, PublicAPIClass); }
 
-  std::string getTypeAsString(llvm::Record *TypeRecord) {
+  std::string getTypeAsString(llvm::Record *TypeRecord) override {
     if (isaNamedType(TypeRecord) || isaStructType(TypeRecord)) {
       return std::string(TypeRecord->getValueAsString("Name"));
     } else if (isaPtrType(TypeRecord)) {
       return getTypeAsString(TypeRecord->getValueAsDef("PointeeType")) + " *";
     } else if (isaConstType(TypeRecord)) {
       return std::string("const ") +
              getTypeAsString(TypeRecord->getValueAsDef("UnqualifiedType"));
     } else if (isaRestrictedPtrType(TypeRecord)) {
       return getTypeAsString(TypeRecord->getValueAsDef("PointeeType")) +
              " *__restrict";
     } else {
       llvm::PrintFatalError(TypeRecord->getLoc(), "Invalid type.\n");
     }
   }
 
   void indexStandardSpecDef(llvm::Record *StandardSpec) {
     auto HeaderSpecList = StandardSpec->getValueAsListOfDefs("Headers");
     for (llvm::Record *HeaderSpec : HeaderSpecList) {
       if (HeaderSpec->getValueAsString("Name") == StdHeader) {
         auto MacroSpecList = HeaderSpec->getValueAsListOfDefs("Macros");
         // TODO: Trigger a fatal error on duplicate specs.
         for (llvm::Record *MacroSpec : MacroSpecList)
           MacroSpecMap[std::string(MacroSpec->getValueAsString("Name"))] =
               MacroSpec;
 
         auto TypeSpecList = HeaderSpec->getValueAsListOfDefs("Types");
         for (llvm::Record *TypeSpec : TypeSpecList)
           TypeSpecMap[std::string(TypeSpec->getValueAsString("Name"))] =
               TypeSpec;
for (auto &Pair : DefsMap) {
       if (isaPublicAPI(Def)) {
         if (Def->getValueAsString("HeaderName") == StdHeader)
           indexPublicAPIDef(Def);
       }
     }
   }
 
 public:
   APIGenerator(llvm::StringRef Header, llvm::RecordKeeper &Records)
       : StdHeader(Header) {
     index(Records);
   }
 
   void write(llvm::raw_ostream &OS) {
     for (auto &Pair : MacroDefsMap) {
       const std::string &Name = Pair.first;
       if (MacroSpecMap.find(Name) == MacroSpecMap.end())
void PublicAPICommand::run(llvm::raw_ostream &OS, const ArgVector &Args,
   if (Args.size() != 0) {
     Reporter.printFatalError("public_api command does not take any arguments.");
   }
 
   APIGenerator G(StdHeader, Records);
   G.write(OS);
 }
 
+std::unique_ptr<APIReader> makeAPIReader(llvm::StringRef StdHeader,
+                                         llvm::RecordKeeper &Records) {
+  return std::make_unique<APIGenerator>(StdHeader, Records);
+}
+
 } // namespace llvm_libc