Always build libfuzzer

[george.karpenkov]

Summary

Hi,

There are two reasons why users might want to build libfuzzer:

• To fuzz LLVM itself
• To get the libFuzzer.a archive file, so that they can attach it to their code

Current CMake file only supports the first case: in order to build it, the user has to compile the entire LLVM repo with coverage flags (which is not even possible using e.g. default mac os clang).

The proposed change always builds libfuzzer, and supports the second use case if the specified flag is set.

[kcc]

Why do you need this change?
Anyone who wants to build libFuzzer.a can simply run lib/Fuzzer/build.sh.
libFuzzer is independent of the rest of LLVM (except that it requires the target to be built with LLVM's coverage instrumentation)
and so the build is also independent of LLVM.

the check-fuzzer (i.e. runninng libFuzzer tests), as you note, does require special flags and what's more important -- the fresh clang.

When/if we move to monorepo, I want to change these build rules to depend on freshly-built clang and then libFuzzer and its tests will be built/run by default.

[kubamracek]

(George works with me and Anna and he's trying to make libFuzzer more available to non-compiler folks.)

The point of this patch is to have something that can potentially be shipped with the compiler, and this also ensures that the version of libFuzzer is correct to use with that compiler. The current "recommended" way of using libFuzzer involves checking out the llvm repo and building a .a file of libFuzzer – we'd like to remove those steps and have the .a file be already a part of the compiler build.

[kcc]

George works with me and Anna

I've figured that out. Welcome George!

So, ok, I don't mind, as long as we all agree that this is a partial solution: we build libFuzzer.a but we don't test it in the default mode.

BTW, if you can figure out how to actually solve this problem (and to make the libFuzzer tests depend on the fresh clang) w/o waiting for the monorepo -- would be great!

[kubamracek]

In D32096#728508, @kcc wrote:

George works with me and Anna

I've figured that out. Welcome George!

So, ok, I don't mind, as long as we all agree that this is a partial solution: we build libFuzzer.a but we don't test it in the default mode.

Right, this is just a small incremental change, and there are further patches coming. The goal is to get libFuzzer as easy to use as e.g. AddressSanitizer.

BTW, if you can figure out how to actually solve this problem (and to make the libFuzzer tests depend on the fresh clang) w/o waiting for the monorepo -- would be great!

Sounds doable, this is what compiler-rt unit tests are already doing.

[kcc]

Sounds doable, this is what compiler-rt unit tests are already doing.

but they are in compiler-rt, while this is in llvm

[george.karpenkov]

@kcc Kostya, could you take a look at http://lab.llvm.org:8011/builders/lld-x86_64-darwin13/builds/7107/steps/build_Lld/logs/stdio ?
Seems like you are using a compiler extension which is not supported by one of the machines in our CI farm.
Is this extension required? What would be a reasonable way to check for its presence?
LLVM webpage currently claims that LLVM should be buildable with those (http://llvm.org/docs/GettingStarted.html#software)

There is another failure on http://green.lab.llvm.org/green/job/clang-stage1-configure-RA_build/33461/console

Thanks!

[george.karpenkov]

Currently blocked by D32312 and D32311.

[george.karpenkov]

Whitelisting allowed systems due to too many build failures on obscure architectures.
The new file exhibits old behavior (build on demand) for all systems except for mac os and linux.

llvm/trunk/lib/Fuzzer/CMakeLists.txt

 include(CheckCXXSourceCompiles)
 
 CHECK_CXX_SOURCE_COMPILES("
     static thread_local int blah;
     int main() {
       return 0;
     }
     " HAS_THREAD_LOCAL)
 
 if( NOT HAS_THREAD_LOCAL )
     set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Dthread_local=__thread")
 endif()
 
+if( LLVM_USE_SANITIZE_COVERAGE )
   set(LIBFUZZER_FLAGS_BASE "${CMAKE_CXX_FLAGS}")
+
   # Disable the coverage and sanitizer instrumentation for the fuzzer itself.
   set(CMAKE_CXX_FLAGS "${LIBFUZZER_FLAGS_BASE} -fno-sanitize-coverage=trace-pc-guard,edge,trace-cmp,indirect-calls,8bit-counters -Werror")
-if( LLVM_USE_SANITIZE_COVERAGE )
-  if(NOT "${LLVM_USE_SANITIZER}" STREQUAL "Address")
-    message(FATAL_ERROR
-      "LibFuzzer and its tests require LLVM_USE_SANITIZER=Address and "
-      "LLVM_USE_SANITIZE_COVERAGE=YES to be set."
-      )
-  endif()
-  add_library(LLVMFuzzerNoMainObjects OBJECT
+
     FuzzerCrossOver.cpp
     FuzzerDriver.cpp
     FuzzerExtFunctionsDlsym.cpp
     FuzzerExtFunctionsDlsymWin.cpp
     FuzzerExtFunctionsWeak.cpp
     FuzzerExtraCounters.cpp
     FuzzerIO.cpp
     FuzzerIOPosix.cpp
     FuzzerIOWindows.cpp
     FuzzerLoop.cpp
     FuzzerMerge.cpp
     FuzzerMutate.cpp
     FuzzerSHA1.cpp
     FuzzerShmemPosix.cpp
     FuzzerShmemWindows.cpp
     FuzzerTracePC.cpp
     FuzzerTraceState.cpp
     FuzzerUtil.cpp
     FuzzerUtilDarwin.cpp
     FuzzerUtilLinux.cpp
     FuzzerUtilPosix.cpp
     FuzzerUtilWindows.cpp
     )
-  add_library(LLVMFuzzerNoMain STATIC
     $<TARGET_OBJECTS:LLVMFuzzerNoMainObjects>
     )
-  target_link_libraries(LLVMFuzzerNoMain ${LLVM_PTHREAD_LIB})
-  add_library(LLVMFuzzer STATIC
     FuzzerMain.cpp
     $<TARGET_OBJECTS:LLVMFuzzerNoMainObjects>
     )
-  target_link_libraries(LLVMFuzzer ${LLVM_PTHREAD_LIB})
 
+if( LLVM_USE_SANITIZE_COVERAGE )
+
+  if(NOT "${LLVM_USE_SANITIZER}" STREQUAL "Address")
+    message(FATAL_ERROR
+      "LibFuzzer tests require LLVM_USE_SANITIZER=Address and "
+      "LLVM_USE_SANITIZE_COVERAGE=YES to be set."
+      )
+  endif()
+
   if( LLVM_INCLUDE_TESTS )
     add_subdirectory(test)
   endif()
 endif()