Drop Windows support from libFuzzer tests

[kcc]

I don't mind, but please let Zack or Reid review this.
The question is: do we need to remove all traces of windows support from tests, or just disable check-fuzzer on the bot and remove only the parts that are blocking the refactoring.

[rnk]

First we should remove check-fuzzer from sanitizer-windows. That means patching zorg and a buildmaster restart. Otherwise, the bot will go red because of a missing target.

[george.karpenkov]

@rnk That's nothing I could do, right?
In any case, if we were to leave Windows support, I think redefining CMAKE_CXX_LINK_EXECUTABLE is definitely not the way to go.
It already caused a few problems (e.g. having to change the output directory), and will cause more.
If anything, even moving the compilation to LIT, so that the driver would be able to insert the appropriate libraries, is better.

[kcc]

Is there an option to make check-fuzzer a no-op on windows?
This way we won't have to touch the bot (and then touch it again, when windows is reinstated)

[george.karpenkov]

@kcc of course anything is possible. Should we though? Pretending that tests pass when they don't seems like a bad idea.

[rnk]

In D36205#829441, @george.karpenkov wrote:

@rnk That's nothing I could do, right?
In any case, if we were to leave Windows support, I think redefining CMAKE_CXX_LINK_EXECUTABLE is definitely not the way to go.

Why do you think that? If we want to keep the logic of where clang's compiler-rt libraries are within the clang driver, this is what we have to do. Otherwise we have to duplicate the logic that -fsanitize=foo uses to find clang's builtin libraries.

Which do you guys prefer, duplication or this complicated CMake logic to make Windows appear more like Posix?

It already caused a few problems (e.g. having to change the output directory), and will cause more.

I'm not familiar with this, you probably explained it somewhere else but I haven't had time to read.

If anything, even moving the compilation to LIT, so that the driver would be able to insert the appropriate libraries, is better.

That would be nice, it would make libFuzzer tests look more like compiler-rt sanitizer tests.

[kcc]

We will have to do something of this sort anyway once we make 'check-fuzzer' a part of 'check-all' on Linux and Mac.

[rnk]

In D36205#829441, @george.karpenkov wrote:

@rnk That's nothing I could do, right?

We have to wait for Galina to do master restarts, but the logic is probably pretty easy to remove from zorg/zorg/buildbot/builders/SanitizerBuilderWindows.py.

[george.karpenkov]

@rnk

That would be nice, it would make libFuzzer tests look more like compiler-rt sanitizer tests.

I think so too, I've just written an email arguing for it.

Which do you guys prefer

I prefer moving compilation commands to LIT.
Unit tests would still remain, but for them I think it's easier to duplicate the logic and specify all the dependencies in CMake, as done already in all sanitizers.

[kcc]

I prefer moving compilation commands to LIT.

And I actually don't mind.

[george.karpenkov]

@kcc Does it mean you would be OK with the change moving the compilation commands to LIT, and only compiling unit-tests in CMake?

@rnk

I'm not familiar with this, you probably explained it somewhere else but I haven't had time to read.

I was referring to https://reviews.llvm.org/D27870

[kcc]

In D36205#829472, @george.karpenkov wrote:

@kcc Does it mean you would be OK with the change moving the compilation commands to LIT, and only compiling unit-tests in CMake?

Yes. This will be quite a change though

[kcc]

I would still suggest to make check-fuzzer a no-op on Windows so that we don't need to touch the windows bot configs now.

[zturner]

Agreed, if both of you are blocked and just need to move on, and there's no definitive timeline for when a master restart will happen, then just making it a no-op until we can get the step actually removed seems like an immediate way to get unblocked.

[george.karpenkov]

@kcc Getting back to this revision, can this be committed as well then?

[kcc]

LGTM

[george.karpenkov]

OK great will probably land after https://reviews.llvm.org/D36295 to avoid conflicting with myself

lib/Fuzzer/CMakeLists.txt

if ( LLVM_USE_SANITIZE_COVERAGE OR CMAKE_SYSTEM_NAME MATCHES "Darwin|Linux" )
   target_link_libraries(LLVMFuzzerNoMain ${LLVM_PTHREAD_LIB})
   add_library(LLVMFuzzer STATIC
       FuzzerMain.cpp
       $<TARGET_OBJECTS:LLVMFuzzerNoMainObjects>
       )
   target_link_libraries(LLVMFuzzer ${LLVM_PTHREAD_LIB})
 endif()
 
-if( LLVM_USE_SANITIZE_COVERAGE AND LLVM_INCLUDE_TESTS )
+if( LLVM_USE_SANITIZE_COVERAGE AND LLVM_INCLUDE_TESTS)
+
+  # Libfuzzer tests are currently not supported on Windows.
+  if (NOT MSVC)
     add_subdirectory(test)
   endif()
+endif()

lib/Fuzzer/test/CMakeLists.txt

set(variables_to_filter
   )
 foreach (VARNAME ${variables_to_filter})
   string(REGEX REPLACE "([-/]O)[123s]" "\\10" ${VARNAME} "${${VARNAME}}")
 endforeach()
 
 # Enable the coverage instrumentation (it is disabled for the Fuzzer lib).
 set(CMAKE_CXX_FLAGS "${LIBFUZZER_FLAGS_BASE} -fsanitize-coverage=trace-pc-guard,indirect-calls,trace-cmp,trace-div,trace-gep -gline-tables-only")
 
-if(MSVC)
-  # For tests use the CRT specified for release build
-  # (asan doesn't support MDd and MTd)
-  if ("${LLVM_USE_CRT_RELEASE}" STREQUAL "")
-    set(CRT_FLAG " /MD ")
-  else()
-    set(CRT_FLAG " /${LLVM_USE_CRT_RELEASE} ")
-  endif()
-  # In order to use the sanitizers in Windows, we need to link against many
-  # runtime libraries which will depend on the target being created
-  # (executable or dll) and the c runtime library used (MT/MD).
-  # By default, cmake uses link.exe for linking, which fails because we don't
-  # specify the appropiate dependencies.
-  # As we don't want to consider all of that possible situations which depends
-  # on the implementation of the compiler-rt, the simplest option is to change
-  # the rules for linking executables and shared libraries, using the compiler
-  # instead of link.exe. Clang will consider the sanitizer flags, and
-  # automatically provide the required libraries to the linker.
-  set(CMAKE_CXX_LINK_EXECUTABLE "<CMAKE_CXX_COMPILER> <FLAGS> ${CMAKE_CXX_FLAGS} ${CRT_FLAG} <OBJECTS> -o <TARGET> <LINK_LIBRARIES> /link <CMAKE_CXX_LINK_FLAGS> <LINK_FLAGS>")
-  set(CMAKE_CXX_CREATE_SHARED_LIBRARY "<CMAKE_CXX_COMPILER> ${CMAKE_CXX_FLAGS} ${CRT_FLAG} /LD <CMAKE_SHARED_LIBRARY_CXX_FLAGS> <CMAKE_SHARED_LIBRARY_CREATE_CXX_FLAGS> <CMAKE_SHARED_LIBRARY_SONAME_CXX_FLAG> <TARGET_SONAME> -o <TARGET> <OBJECTS> <LINK_LIBRARIES> /link <LINK_FLAGS>")
-endif()
-
 add_custom_target(TestBinaries)
 
 # add_libfuzzer_test(<name>
 #   SOURCES source0.cpp [source1.cpp ...]
 #   )
 #
 #   Declares a LibFuzzer test executable with target name LLVMFuzzer-<name>.
 #
set(Tests
   ThreadedLeakTest
   ThreadedTest
   TimeoutTest
   TimeoutEmptyTest
   TraceMallocTest
   TwoDifferentBugsTest
   )
 
-if(APPLE OR MSVC)
+if(APPLE)
   # LeakSanitizer is not supported on OSX and Windows right now
   set(HAS_LSAN 0)
   message(WARNING "LeakSanitizer is not supported."
     " Building and running LibFuzzer LeakSanitizer tests is disabled."
     )
 else()
   set(HAS_LSAN 1)
 endif()
 
 foreach(Test ${Tests})
   add_libfuzzer_test(${Test} SOURCES ${Test}.cpp)
 endforeach()
 
-function(test_export_symbol target symbol)
-  if(MSVC)
-    set_target_properties(LLVMFuzzer-${target} PROPERTIES LINK_FLAGS
-        "-export:${symbol}")
-  endif()
-endfunction()
-
-test_export_symbol(FlagsTest "LLVMFuzzerInitialize")
-test_export_symbol(InitializeTest "LLVMFuzzerInitialize")
-test_export_symbol(BogusInitializeTest "LLVMFuzzerInitialize")
-test_export_symbol(CustomCrossOverTest "LLVMFuzzerCustomCrossOver")
-test_export_symbol(CustomMutatorTest "LLVMFuzzerCustomMutator")
 
 ###############################################################################
 # Unit tests
 ###############################################################################
 
 add_executable(LLVMFuzzer-Unittest
   FuzzerUnittest.cpp
   )

 ###############################################################################
 
 include_directories(..)
 
 # add_subdirectory(uninstrumented)
 add_subdirectory(no-coverage)
 add_subdirectory(trace-pc)
 add_subdirectory(ubsan)
-if (NOT MSVC)
-  add_subdirectory(inline-8bit-counters)
-endif()
 
 add_library(LLVMFuzzer-DSO1 SHARED DSO1.cpp)
 add_library(LLVMFuzzer-DSO2 SHARED DSO2.cpp)
 
 add_executable(LLVMFuzzer-DSOTest
   DSOTestMain.cpp
   DSOTestExtra.cpp)
 
 target_link_libraries(LLVMFuzzer-DSOTest
   LLVMFuzzer-DSO1
   LLVMFuzzer-DSO2
   LLVMFuzzer
   )
 
 set_target_properties(LLVMFuzzer-DSOTest PROPERTIES RUNTIME_OUTPUT_DIRECTORY
   "${CMAKE_BINARY_DIR}/lib/Fuzzer/test")
 
-if(MSVC)
-  set_output_directory(LLVMFuzzer-DSO1
-    BINARY_DIR "${CMAKE_BINARY_DIR}/lib/Fuzzer/test"
-    LIBRARY_DIR "${CMAKE_BINARY_DIR}/lib/Fuzzer/test")
-  set_output_directory(LLVMFuzzer-DSO2
-    BINARY_DIR "${CMAKE_BINARY_DIR}/lib/Fuzzer/test"
-    LIBRARY_DIR "${CMAKE_BINARY_DIR}/lib/Fuzzer/test")
-else(MSVC)
-  set_output_directory(LLVMFuzzer-DSO1
-    LIBRARY_DIR "${CMAKE_BINARY_DIR}/lib/Fuzzer/lib")
-  set_output_directory(LLVMFuzzer-DSO2
-    LIBRARY_DIR "${CMAKE_BINARY_DIR}/lib/Fuzzer/lib")
-endif()
 
 add_dependencies(TestBinaries LLVMFuzzer-DSOTest)
 
 ###############################################################################
 # Configure lit to run the tests
 #
 # Note this is done after declaring all tests so we can inform lit if any tests
 # need to be disabled.
 ###############################################################################
-set(LIBFUZZER_POSIX 1)
-if (MSVC)
-  set(LIBFUZZER_POSIX 0)
-endif()
 
 configure_lit_site_cfg(
   ${CMAKE_CURRENT_SOURCE_DIR}/lit.site.cfg.in
   ${CMAKE_CURRENT_BINARY_DIR}/lit.site.cfg
   )
 
 configure_lit_site_cfg(
   ${CMAKE_CURRENT_SOURCE_DIR}/unit/lit.site.cfg.in

lib/Fuzzer/test/afl-driver-extra-stats.test

-REQUIRES: posix
-
 ; Test that not specifying an extra stats file isn't broken.
 RUN: unset AFL_DRIVER_EXTRA_STATS_FILENAME
 RUN: AFLDriverTest
 
 ; Test that specifying an invalid extra stats file causes a crash.
 RUN: ASAN_OPTIONS= AFL_DRIVER_EXTRA_STATS_FILENAME=%T not --crash AFLDriverTest
 
 ; Test that specifying a corrupted stats file causes a crash.

lib/Fuzzer/test/afl-driver-stderr.test

-REQUIRES: posix
-
 ; Test that not specifying a stderr file isn't broken.
 RUN: unset AFL_DRIVER_STDERR_DUPLICATE_FILENAME
 RUN: AFLDriverTest
 
 ; Test that specifying an invalid file causes a crash.
 RUN: ASAN_OPTIONS= AFL_DRIVER_STDERR_DUPLICATE_FILENAME="%T" not --crash AFLDriverTest
 
 ; Test that a file is created when specified as the duplicate stderr.
 RUN: AFL_DRIVER_STDERR_DUPLICATE_FILENAME=%t AFLDriverTest
 RUN: stat %t

lib/Fuzzer/test/equivalence-signals.test

-REQUIRES: posix
 # Run EquivalenceATest against itself with a small timeout
 # to stress the signal handling and ensure that shmem doesn't mind
 # the signals.
 
 RUN: LLVMFuzzer-EquivalenceATest -timeout=1 -run_equivalence_server=EQUIV_SIG_TEST & export APID=$!
 RUN: sleep 3
 RUN: LLVMFuzzer-EquivalenceATest -timeout=1 -use_equivalence_server=EQUIV_SIG_TEST -runs=500000 2>&1
 RUN: kill -9 $APID

lib/Fuzzer/test/equivalence.test

-REQUIRES: posix
-
 RUN: LLVMFuzzer-EquivalenceATest -run_equivalence_server=EQUIV_TEST & export APID=$!
 RUN: sleep 3
 RUN: not LLVMFuzzer-EquivalenceBTest -use_equivalence_server=EQUIV_TEST -max_len=4096 2>&1 | FileCheck %s
 CHECK: ERROR: libFuzzer: equivalence-mismatch. Sizes: {{.*}}; offset 2
 CHECK: SUMMARY: libFuzzer: equivalence-mismatch
 RUN: kill -9 $APID

lib/Fuzzer/test/lit.cfg

else:
   lit_config.note('lsan feature unavailable')
 
 if sys.platform.startswith('win') or sys.platform.startswith('cygwin'):
   config.available_features.add('windows')
 
 if sys.platform.startswith('darwin'):
   config.available_features.add('darwin')
 
-if config.is_posix:
-  config.available_features.add('posix')
-
 if sys.platform.startswith('linux'):
   # Note the value of ``sys.platform`` is not consistent
   # between python 2 and 3, hence the use of ``.startswith()``.
   lit_config.note('linux feature available')
   config.available_features.add('linux')
 else:
   lit_config.note('linux feature unavailable')

lib/Fuzzer/test/lit.site.cfg.in

 config.test_exec_root = "@CMAKE_CURRENT_BINARY_DIR@"
 config.llvm_tools_dir = "@LLVM_TOOLS_DIR@"
 config.has_lsan = True if @HAS_LSAN@ == 1 else False
-config.is_posix = @LIBFUZZER_POSIX@
 lit_config.load_config(config, "@CMAKE_CURRENT_SOURCE_DIR@/lit.cfg")

lib/Fuzzer/test/merge-posix.test

-REQUIRES: posix
-
 RUN: rm -rf  %tmp/T1 %tmp/T2
 RUN: mkdir -p %tmp/T1 %tmp/T2
 
 RUN: echo F..... > %tmp/T1/1
 RUN: echo .U.... > %tmp/T1/2
 RUN: echo ..Z... > %tmp/T1/3
 
 RUN: echo .....F > %tmp/T2/1

lib/Fuzzer/test/ulimit.test

-REQUIRES: posix
-
 RUN: ulimit -s 1000
 RUN: not LLVMFuzzer-SimpleTest