[fuzzer,CMake] Add config name for fuzzer lit test

[yingcong-wu]

Summary

Add config name for fuzzer lit test, to make it easier to identify failures are with which config.

Before this change, same lit tests with different configs will share the same test name.

********************
Failed Tests (2):
  libFuzzer :: fuzzer-flags.test
  libFuzzer :: fuzzer-flags.test

Actually this is a failure of two lit tests(two configs of the same test).

With this change, the names will be different.

********************
Failed Tests (2):
  libFuzzer-i386-default-Linux ::fuzzer-flags.test
  libFuzzer-x86_64-default-Linux :: fuzzer-flags.test

[MaskRay]

Can you put the lit output difference in the summary/commit message?

[yingcong-wu]

Update suffix

[MaskRay]

Currently we may see

% ninja check-fuzzer
...
Failed Tests (1):
  libFuzzer :: merge-posix.test

How does the change affect the output?

[yingcong-wu]

How does the change affect the output?

Actually, I think the currently check-fuzzer is not implemented well. If you run check-fuzzer, it actually runs all of its sub-check speraterly, which will have the following output(sperate check result cannot merge into one final result). With check-all those results will group together. I think we can do it like sanitizer_common, only add_lit_testsuite once, which runs all fuzzer tests with all configs. I plan to do it with another PR.

********************
********************
Failed Tests (1):
  libFuzzer :: fuzzer-flags.test


Testing Time: 19.80s
  Unsupported      :   7
  Passed           : 128
  Expectedly Failed:   3
  Failed           :   1
make[3]: *** [projects/compiler-rt/test/fuzzer/CMakeFiles/check-fuzzer-default-x86_64.dir/build.make:71: projects/compiler-rt/test/fuzzer/CMakeFiles/check-fuzzer-default-x86_64] Error 1
make[2]: *** [CMakeFiles/Makefile2:36745: projects/compiler-rt/test/fuzzer/CMakeFiles/check-fuzzer-default-x86_64.dir/all] Error 2
make[2]: *** Waiting for unfinished jobs....
--

********************
********************
Failed Tests (1):
  libFuzzer :: fuzzer-flags.test


Testing Time: 24.33s
  Unsupported:  21
  Passed     : 117
  Failed     :   1
make[3]: *** [projects/compiler-rt/test/fuzzer/CMakeFiles/check-fuzzer-default-i386.dir/build.make:71: projects/compiler-rt/test/fuzzer/CMakeFiles/check-fuzzer-default-i386] Error 1
make[2]: *** [CMakeFiles/Makefile2:36697: projects/compiler-rt/test/fuzzer/CMakeFiles/check-fuzzer-default-i386.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:36608: projects/compiler-rt/test/fuzzer/CMakeFiles/check-fuzzer.dir/rule] Error 2
make: *** [Makefile:10719: check-fuzzer] Error 2

[yingcong-wu]

Resolve merge

[MaskRay]

Is there an opportunity for more consistency?

test/fuzzer for Apple does set(LIBFUZZER_TEST_CONFIG_SUFFIX "-${arch}-${platform}").

test/asan does string(TOLOWER "-${arch}-${OS_NAME}" ASAN_TEST_CONFIG_SUFFIX)

What's is the possible value of ${stdlib_name} and why is it useful?

[yingcong-wu]

test/fuzzer for Apple

set(CONFIG_NAME "${PLATFORM_CAPITALIZED}${ARCH_UPPER_CASE}Config")
set(LIBFUZZER_TEST_CONFIG_SUFFIX "-${arch}-${platform}")

test/fuzzer

set(CONFIG_NAME ${ARCH_UPPER_CASE}${STDLIB_CAPITALIZED}${OS_NAME}Config)
set(LIBFUZZER_TEST_CONFIG_SUFFIX "-${arch}-${stdlib_name}-${OS_NAME}")

I think the config_suffix is aligned with the config_name. Tests for Apple have different config name.

As for possible value of ${stdlib_name},

macro(test_fuzzer stdlib)
# skip lines
endmacro()

test_fuzzer("default")
if("${CMAKE_SYSTEM_NAME}" STREQUAL "Linux")
  if(TARGET cxx_shared)
    test_fuzzer("libc++" DEPS cxx_shared)
  endif()
  if(TARGET cxx_static)
    test_fuzzer("static-libc++" DEPS cxx_static)
  endif()
endif()

[MaskRay]

Thanks!

[yingcong-wu]

Hi @MaskRay, I don't have commit access, could you help land patch this for me? Thanks.

compiler-rt/test/fuzzer/CMakeLists.txt

foreach(arch ${FUZZER_TEST_ARCH})
     set(LIBFUZZER_TEST_TARGET_ARCH ${arch})
     set(LIBFUZZER_TEST_APPLE_PLATFORM "osx")
     set(LIBFUZZER_TEST_MIN_DEPLOYMENT_TARGET_FLAG "${DARWIN_osx_MIN_VER_FLAG}")
 
     set(LIBFUZZER_TEST_STDLIB ${stdlib})
 
     string(TOUPPER ${arch} ARCH_UPPER_CASE)
     set(CONFIG_NAME ${ARCH_UPPER_CASE}${STDLIB_CAPITALIZED}${OS_NAME}Config)
+    set(LIBFUZZER_TEST_CONFIG_SUFFIX "-${arch}-${stdlib_name}-${OS_NAME}")
 
     # LIT-based libFuzzer tests.
     configure_lit_site_cfg(
       ${CMAKE_CURRENT_SOURCE_DIR}/lit.site.cfg.py.in
       ${CMAKE_CURRENT_BINARY_DIR}/${CONFIG_NAME}/lit.site.cfg.py
       )
 
     add_lit_testsuite(check-fuzzer-${stdlib_name}-${arch}