[LibFuzzer] Refactor declaration of tests in CMake.

[delcypher]

Summary

[LibFuzzer] Refactor declaration of tests in CMake.

Add a new CMake function (add_libfuzzer_test()) to simplify
declaration of executables for testing LibFuzzer and use it to
reorganise how tests are declared.

Note that configuration of the lit configuration files has been moved
as late as possible because we are going to need to disable some tests
for some platforms and we will need to propagate this information into
the lit configuration.

[kcc]

lib/Fuzzer/test/CMakeLists.txt

31 ↗

(On Diff #58707)

Let's start from not having this "NO_MAIN" complexity.
CustomMainTests is empty and could be safely removed.
The gunit unit test is a separate beast, one of its kind.

[delcypher]

@kcc @aizatsky: To give you more context as to how these changes are going to be used I have some WIP patches that depend on this one:

• fix weak linking issues on OSX:

https://github.com/delcypher/llvm/commit/63ba587cd1f94e1b7c29877f1f7afdff5efc477c

• Disable building DFSan tests when host compiler doesn't support it

https://github.com/delcypher/llvm/commit/c316fb26a8def458b51120e8cbdbf190fa8d0479

With those changes LibFuzzer can be compiled and the tests can be built on OSX. There are still test failures but this one step closer to bringing LibFuzzer to OSX.

Failing Tests (7):
    LLVMFuzzer :: fuzzer-leak.test
    LLVMFuzzer :: fuzzer-oom.test
    LLVMFuzzer :: fuzzer-trace-pc.test
    LLVMFuzzer :: fuzzer-traces.test
    LLVMFuzzer-Unittest :: LLVMFuzzer-Unittest/FuzzerDictionary.ParseDictionaryFile
    LLVMFuzzer-Unittest :: LLVMFuzzer-Unittest/FuzzerDictionary.ParseOneDictionaryEntry
    LLVMFuzzer-Unittest :: LLVMFuzzer-Unittest/FuzzerMutate.ShuffleBytes2

  Expected Passes    : 35
  Unsupported Tests  : 1
  Unexpected Failures: 7

[delcypher]

lib/Fuzzer/test/CMakeLists.txt

31 ↗

(On Diff #58707)

@kcc:

I kept "NO_MAIN" because

• I presumed the intention was to add tests to CustomMainTests at some point in the future. Is this not the intention?
• Using the NO_MAIN variant of add_libfuzzer_test() for the unit test is very useful because in a subsequent patch (https://github.com/delcypher/llvm/commit/63ba587cd1f94e1b7c29877f1f7afdff5efc477c) I am going to add logic to fix weak linking issues on OSX. If I don't use add_libfuzzer_test() for the unit test then I will have duplicate the logic that fixes the weak linking issue for the unit test which isn't desirable because I am trying to avoid as much boiler plate CMake code as possible.

It is your call but I would prefer to use add_libfuzzer_test() for the unit test so that my subsequent patch for weak linking is cleaner. What would you like to do?

[kcc]

lib/Fuzzer/test/CMakeLists.txt

31 ↗

(On Diff #58707)

Yes, I hope to not need CustomMainTests in future you can safely delete that part.

Sure, please have as much code reuse as possible for gnuint test, but I think you can do it w/o thie NO_MAIN

[delcypher]

• Remove use of NO_MAIN in add_libfuzzer_test(). This has added some code duplication when handling the unit test.
• Rebase

[delcypher]

@kcc : I've updated the patch and rebased. I noticed that when I rebased that the LibFuzzer tests are taking much longer to run (this is nothing to do with my patch). They used to take ~40 seconds on my machine now it takes roughly ~140 seconds. Is that intentional?

[kcc]

LGTM, assuming this does not change the behavior of 'check-fuzzer'.
I wonder, how you can verify that.
In the past I remember cmake changes that caused us to not run some tests.
We've learned about those in a hard way...

They used to take ~40 seconds on my machine now it takes roughly ~140 seconds. Is that intentional?

Not intentional... On my box they still take 25 seconds...

[delcypher]

@kcc :

LGTM, assuming this does not change the behavior of 'check-fuzzer'.
I wonder, how you can verify that.
In the past I remember cmake changes that caused us to not run some tests.
We've learned about those in a hard way...

I verified by doing a completely fresh build every time (to avoid lit running executables built previously) and checking that lit reports the same number of tests being executed.

[kcc]

Perfect.
And I've just reduced the test run-time (on my box) to 14 seconds.

[delcypher]

In D20706#441867, @kcc wrote:

LGTM, assuming this does not change the behavior of 'check-fuzzer'.
I wonder, how you can verify that.
In the past I remember cmake changes that caused us to not run some tests.
We've learned about those in a hard way...

They used to take ~40 seconds on my machine now it takes roughly ~140 seconds. Is that intentional?

Not intentional... On my box they still take 25 seconds...

Okay FYI the slow down is introduced in r270942. For that commit if I run

bin/llvm-lit lib/Fuzzer/test/fuzzer.test
-- Testing: 1 tests, 1 threads --
PASS: LLVMFuzzer :: fuzzer.test (1 of 1)
Testing Time: 126.83s
  Expected Passes    : 1

on the previous commit

bin/llvm-lit lib/Fuzzer/test/fuzzer.test
-- Testing: 1 tests, 1 threads --
PASS: LLVMFuzzer :: fuzzer.test (1 of 1)
Testing Time: 3.63s
  Expected Passes    : 1

My configure line is

CC=/home/dan/dev/llvm-upstream/ninja_build/bin/clang CXX=/home/dan/dev/llvm-upstream/ninja_build/bin/clang++ cmake -DCMAKE_BUILD_TYPE=RELWITHDEBINFO  -DLLVM_USE_SANITIZER=Address -DLLVM_USE_SANITIZE_COVERAGE=YES  ../src/ && make check-fuzzer -j8

and the clang I'm building with is clang version 3.9.0 (trunk 270188) which is a bit old. I should probably rebuild clang and see if this is still observable.

[kcc]

Is this linux or mac?

[delcypher]

In D20706#441909, @kcc wrote:

Is this linux or mac?

This is Linux (Arch Linux - Linux 4.5.1-1-ARCH #1 SMP PREEMPT Thu Apr 14 19:19:32 CEST 2016 x86_64 GNU/Linux). I am building libfuzzer and the tests with an old revision of clang though so I will rebuild clang and try again. It is probably worth noting that my version of libstdc++ is probably different than yours (it is the version that comes with gcc 5.3.0).

[kcc]

In D20706#441947, @delcypher wrote:

In D20706#441909, @kcc wrote:

Is this linux or mac?

This is Linux (Arch Linux - Linux 4.5.1-1-ARCH #1 SMP PREEMPT Thu Apr 14 19:19:32 CEST 2016 x86_64 GNU/Linux). I am building libfuzzer and the tests with an old revision of clang though so I will rebuild clang and try again. It is probably worth noting that my version of libstdc++ is probably different than yours (it is the version that comes with gcc 5.3.0).

Let's deal with it in a separate thread.
Since I don't see it on my box you'll have to do the initial investigation (just look at 'top' what test is running for so long)

llvm/trunk/lib/Fuzzer/test/CMakeLists.txt

foreach (flag ${BUILD_FLAGS_AS_LIST})
     endif()
   endforeach()
   set(${VARNAME} "${new_flags}")
 endforeach()
 
 # Enable the coverage instrumentation (it is disabled for the Fuzzer lib).
 set(CMAKE_CXX_FLAGS "${LIBFUZZER_FLAGS_BASE} -fsanitize-coverage=edge,indirect-calls")
 
-set(DFSanTests
-  MemcmpTest
-  SimpleCmpTest
-  StrcmpTest
-  StrncmpTest
-  SwitchTest
+# add_libfuzzer_test(<name>
+#   SOURCES source0.cpp [source1.cpp ...]
+#   )
+#
+#   Declares a LibFuzzer test executable with target name LLVMFuzzer-<name>.
+#
+#   One or more source files to be compiled into the binary must be declared
+#   after the SOURCES keyword.
+function(add_libfuzzer_test name)
+  set(multi_arg_options "SOURCES")
+  cmake_parse_arguments(
+    "add_libfuzzer_test" "" "" "${multi_arg_options}" ${ARGN})
+  if ("${add_libfuzzer_test_SOURCES}" STREQUAL "")
+    message(FATAL_ERROR "Source files must be specified")
+  endif()
+  add_executable(LLVMFuzzer-${name}
+    ${add_libfuzzer_test_SOURCES}
+    )
+  target_link_libraries(LLVMFuzzer-${name} LLVMFuzzer)
+  # Place binary where llvm-lit expects to find it
+  set_target_properties(LLVMFuzzer-${name}
+    PROPERTIES RUNTIME_OUTPUT_DIRECTORY
+    "${CMAKE_BINARY_DIR}/lib/Fuzzer/test"
     )
+  set(TestBinaries ${TestBinaries} LLVMFuzzer-${name} PARENT_SCOPE)
+endfunction()
+
+# Variable to keep track of all test targets
+set(TestBinaries)
+
+###############################################################################
+# Basic tests
+###############################################################################
 
 set(Tests
   AccumulateAllocationsTest
   BufferOverflowOnInput
   CallerCalleeTest
   CounterTest
   CustomMutatorTest
   EmptyTest
set(Tests
   SpamyTest
   StrcmpTest
   StrncmpTest
   SwitchTest
   ThreadedTest
   TimeoutTest
   )
 
-set(CustomMainTests
-  )
-
-set(UninstrumentedTests
-  UninstrumentedTest
-  )
-
-set(TraceBBTests
-  SimpleTest
-  )
-
-set(TracePCTests
-  FourIndependentBranchesTest
-  FullCoverageSetTest
-  )
-
-set(UbsanTests
-  SignedIntOverflowTest
-  )
-
-set(TestBinaries)
-
 foreach(Test ${Tests})
-  add_executable(LLVMFuzzer-${Test}
-    ${Test}.cpp
-    )
-  target_link_libraries(LLVMFuzzer-${Test}
-    LLVMFuzzer
-    )
-  set(TestBinaries ${TestBinaries} LLVMFuzzer-${Test})
+  add_libfuzzer_test(${Test} SOURCES ${Test}.cpp)
 endforeach()
 
-foreach(Test ${CustomMainTests})
-  add_executable(LLVMFuzzer-${Test}
-    ${Test}.cpp
-    )
-  target_link_libraries(LLVMFuzzer-${Test}
-    LLVMFuzzerNoMain
-    )
-  set(TestBinaries ${TestBinaries} LLVMFuzzer-${Test})
-endforeach()
-
-
-configure_lit_site_cfg(
-  ${CMAKE_CURRENT_SOURCE_DIR}/lit.site.cfg.in
-  ${CMAKE_CURRENT_BINARY_DIR}/lit.site.cfg
-  )
-
-configure_lit_site_cfg(
-  ${CMAKE_CURRENT_SOURCE_DIR}/unit/lit.site.cfg.in
-  ${CMAKE_CURRENT_BINARY_DIR}/unit/lit.site.cfg
-  )
-
-include_directories(..)
-include_directories(${LLVM_MAIN_SRC_DIR}/utils/unittest/googletest/include)
+###############################################################################
+# Unit tests
+###############################################################################
 
 add_executable(LLVMFuzzer-Unittest
   FuzzerUnittest.cpp
   FuzzerFnAdapterUnittest.cpp
-  $<TARGET_OBJECTS:LLVMFuzzerNoMainObjects>
   )
 
 target_link_libraries(LLVMFuzzer-Unittest
   gtest
   gtest_main
+  LLVMFuzzerNoMain
+  )
+
+target_include_directories(LLVMFuzzer-Unittest PRIVATE
+  "${LLVM_MAIN_SRC_DIR}/utils/unittest/googletest/include"
   )
 
 set(TestBinaries ${TestBinaries} LLVMFuzzer-Unittest)
+set_target_properties(LLVMFuzzer-Unittest
+  PROPERTIES RUNTIME_OUTPUT_DIRECTORY
+  "${CMAKE_CURRENT_BINARY_DIR}"
+)
+###############################################################################
+# Additional tests
+###############################################################################
 
+include_directories(..)
 add_subdirectory(dfsan)
-
-foreach(Test ${DFSanTests})
-  set(TestBinaries ${TestBinaries} LLVMFuzzer-${Test}-DFSan)
-endforeach()
-
 add_subdirectory(uninstrumented)
-
-foreach(Test ${UninstrumentedTests})
-  set(TestBinaries ${TestBinaries} LLVMFuzzer-${Test}-Uninstrumented)
-endforeach()
-
 add_subdirectory(ubsan)
-
-foreach(Test ${UbsanTests})
-  set(TestBinaries ${TestBinaries} LLVMFuzzer-${Test}-Ubsan)
-endforeach()
-
 add_subdirectory(trace-bb)
-
-foreach(Test ${TraceBBTests})
-  set(TestBinaries ${TestBinaries} LLVMFuzzer-${Test}-TraceBB)
-endforeach()
-
 add_subdirectory(trace-pc)
 
-foreach(Test ${TracePCTests})
-  set(TestBinaries ${TestBinaries} LLVMFuzzer-${Test}-TracePC)
-endforeach()
+###############################################################################
+# Configure lit to run the tests
+#
+# Note this is done after declaring all tests so we can inform lit if any tests
+# need to be disabled.
+###############################################################################
 
-set_target_properties(${TestBinaries}
-  PROPERTIES RUNTIME_OUTPUT_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
+configure_lit_site_cfg(
+  ${CMAKE_CURRENT_SOURCE_DIR}/lit.site.cfg.in
+  ${CMAKE_CURRENT_BINARY_DIR}/lit.site.cfg
+  )
+
+configure_lit_site_cfg(
+  ${CMAKE_CURRENT_SOURCE_DIR}/unit/lit.site.cfg.in
+  ${CMAKE_CURRENT_BINARY_DIR}/unit/lit.site.cfg
   )
 
 add_lit_testsuite(check-fuzzer "Running Fuzzer tests"
     ${CMAKE_CURRENT_BINARY_DIR}
     DEPENDS ${TestBinaries} FileCheck not
     )

llvm/trunk/lib/Fuzzer/test/dfsan/CMakeLists.txt

 # These tests depend on both coverage and dfsan instrumentation.
 
 set(CMAKE_CXX_FLAGS
   "${LIBFUZZER_FLAGS_BASE} -fno-sanitize=all -fsanitize=dataflow")
 
-foreach(Test ${DFSanTests})
-  add_executable(LLVMFuzzer-${Test}-DFSan
-    ../${Test}.cpp
-    )
-  target_link_libraries(LLVMFuzzer-${Test}-DFSan
-    LLVMFuzzer
-    )
+set(DFSanTests
+  MemcmpTest
+  SimpleCmpTest
+  StrcmpTest
+  StrncmpTest
+  SwitchTest
+
+foreach(Test ${DFSanTests})
+  add_libfuzzer_test(${Test}-DFSan SOURCES ../${Test}.cpp)
 endforeach()
 
+# Propagate value into parent directory
+set(TestBinaries ${TestBinaries} PARENT_SCOPE)

llvm/trunk/lib/Fuzzer/test/trace-bb/CMakeLists.txt

 # These tests are not instrumented with coverage.
 
 set(CMAKE_CXX_FLAGS
   "${LIBFUZZER_FLAGS_BASE} -fsanitize-coverage=edge,trace-bb")
 
-foreach(Test ${TraceBBTests})
-  add_executable(LLVMFuzzer-${Test}-TraceBB
-    ../${Test}.cpp
-    )
-  target_link_libraries(LLVMFuzzer-${Test}-TraceBB
-    LLVMFuzzer
-    )
+set(TraceBBTests
+  SimpleTest
+
+foreach(Test ${TraceBBTests})
+  add_libfuzzer_test(${Test}-TraceBB SOURCES ../${Test}.cpp)
 endforeach()
 
+# Propagate value into parent directory
+set(TestBinaries ${TestBinaries} PARENT_SCOPE)

llvm/trunk/lib/Fuzzer/test/trace-pc/CMakeLists.txt

 # These tests are not instrumented with coverage.
 
 set(CMAKE_CXX_FLAGS
   "${LIBFUZZER_FLAGS_BASE} -fno-sanitize-coverage=8bit-counters -fsanitize-coverage=trace-pc")
 
-foreach(Test ${TracePCTests})
-  add_executable(LLVMFuzzer-${Test}-TracePC
-    ../${Test}.cpp
-    )
-  target_link_libraries(LLVMFuzzer-${Test}-TracePC
-    LLVMFuzzer
-    )
+set(TracePCTests
+  FourIndependentBranchesTest
+  FullCoverageSetTest
+
+foreach(Test ${TracePCTests})
+  add_libfuzzer_test(${Test}-TracePC SOURCES ../${Test}.cpp)
 endforeach()
 
+# Propagate value into parent directory
+set(TestBinaries ${TestBinaries} PARENT_SCOPE)

llvm/trunk/lib/Fuzzer/test/ubsan/CMakeLists.txt

 # These tests are instrumented with ubsan in non-recovery mode.
 
 set(CMAKE_CXX_FLAGS
   "${LIBFUZZER_FLAGS_BASE} -fsanitize=undefined -fno-sanitize-recover=all")
 
-foreach(Test ${UbsanTests})
-  add_executable(LLVMFuzzer-${Test}-Ubsan
-    ../${Test}.cpp
-    )
-  target_link_libraries(LLVMFuzzer-${Test}-Ubsan
-    LLVMFuzzer
-    )
+set(UbsanTests
+  SignedIntOverflowTest
+
+foreach(Test ${UbsanTests})
+  add_libfuzzer_test(${Test}-Ubsan SOURCES ../${Test}.cpp)
 endforeach()
 
+# Propagate value into parent directory
+set(TestBinaries ${TestBinaries} PARENT_SCOPE)

llvm/trunk/lib/Fuzzer/test/uninstrumented/CMakeLists.txt

 # These tests are not instrumented with coverage.
 
 set(CMAKE_CXX_FLAGS
   "${LIBFUZZER_FLAGS_BASE} -fno-sanitize=all -fno-sanitize-coverage=edge,trace-cmp,indirect-calls,8bit-counters")
 
-foreach(Test ${UninstrumentedTests})
-  add_executable(LLVMFuzzer-${Test}-Uninstrumented
-    ../${Test}.cpp
-    )
-  target_link_libraries(LLVMFuzzer-${Test}-Uninstrumented
-    LLVMFuzzer
-    )
+set(UninstrumentedTests
+  UninstrumentedTest
+
+foreach(Test ${UninstrumentedTests})
+  add_libfuzzer_test(${Test}-Uninstrumented SOURCES ../${Test}.cpp)
 endforeach()
 
+# Propagate value into parent directory
+set(TestBinaries ${TestBinaries} PARENT_SCOPE)