Drop Windows support from libFuzzer tests

[kcc]

I don't mind, but please let Zack or Reid review this.
The question is: do we need to remove all traces of windows support from tests, or just disable check-fuzzer on the bot and remove only the parts that are blocking the refactoring.

[rnk]

First we should remove check-fuzzer from sanitizer-windows. That means patching zorg and a buildmaster restart. Otherwise, the bot will go red because of a missing target.

[george.karpenkov]

@rnk That's nothing I could do, right?
In any case, if we were to leave Windows support, I think redefining CMAKE_CXX_LINK_EXECUTABLE is definitely not the way to go.
It already caused a few problems (e.g. having to change the output directory), and will cause more.
If anything, even moving the compilation to LIT, so that the driver would be able to insert the appropriate libraries, is better.

[kcc]

Is there an option to make check-fuzzer a no-op on windows?
This way we won't have to touch the bot (and then touch it again, when windows is reinstated)

[george.karpenkov]

@kcc of course anything is possible. Should we though? Pretending that tests pass when they don't seems like a bad idea.

[rnk]

In D36205#829441, @george.karpenkov wrote:

@rnk That's nothing I could do, right?
In any case, if we were to leave Windows support, I think redefining CMAKE_CXX_LINK_EXECUTABLE is definitely not the way to go.

Why do you think that? If we want to keep the logic of where clang's compiler-rt libraries are within the clang driver, this is what we have to do. Otherwise we have to duplicate the logic that -fsanitize=foo uses to find clang's builtin libraries.

Which do you guys prefer, duplication or this complicated CMake logic to make Windows appear more like Posix?

It already caused a few problems (e.g. having to change the output directory), and will cause more.

I'm not familiar with this, you probably explained it somewhere else but I haven't had time to read.

If anything, even moving the compilation to LIT, so that the driver would be able to insert the appropriate libraries, is better.

That would be nice, it would make libFuzzer tests look more like compiler-rt sanitizer tests.

[kcc]

We will have to do something of this sort anyway once we make 'check-fuzzer' a part of 'check-all' on Linux and Mac.

[rnk]

In D36205#829441, @george.karpenkov wrote:

@rnk That's nothing I could do, right?

We have to wait for Galina to do master restarts, but the logic is probably pretty easy to remove from zorg/zorg/buildbot/builders/SanitizerBuilderWindows.py.

[george.karpenkov]

@rnk

That would be nice, it would make libFuzzer tests look more like compiler-rt sanitizer tests.

I think so too, I've just written an email arguing for it.

Which do you guys prefer

I prefer moving compilation commands to LIT.
Unit tests would still remain, but for them I think it's easier to duplicate the logic and specify all the dependencies in CMake, as done already in all sanitizers.

[kcc]

I prefer moving compilation commands to LIT.

And I actually don't mind.

[george.karpenkov]

@kcc Does it mean you would be OK with the change moving the compilation commands to LIT, and only compiling unit-tests in CMake?

@rnk

I'm not familiar with this, you probably explained it somewhere else but I haven't had time to read.

I was referring to https://reviews.llvm.org/D27870

[kcc]

In D36205#829472, @george.karpenkov wrote:

@kcc Does it mean you would be OK with the change moving the compilation commands to LIT, and only compiling unit-tests in CMake?

Yes. This will be quite a change though

[kcc]

I would still suggest to make check-fuzzer a no-op on Windows so that we don't need to touch the windows bot configs now.

[zturner]

Agreed, if both of you are blocked and just need to move on, and there's no definitive timeline for when a master restart will happen, then just making it a no-op until we can get the step actually removed seems like an immediate way to get unblocked.

[george.karpenkov]

@kcc Getting back to this revision, can this be committed as well then?

[kcc]

LGTM

[george.karpenkov]

OK great will probably land after https://reviews.llvm.org/D36295 to avoid conflicting with myself

llvm/trunk/lib/Fuzzer/test/CMakeLists.txt

-if(MSVC)
-  # For tests use the CRT specified for release build
-  # (asan doesn't support MDd and MTd)
-  if ("${LLVM_USE_CRT_RELEASE}" STREQUAL "")
-    set(CRT_FLAG " /MD ")
-  else()
-    set(CRT_FLAG " /${LLVM_USE_CRT_RELEASE} ")
-  endif()
-  # In order to use the sanitizers in Windows, we need to link against many
-  # runtime libraries which will depend on the target being created
-  # (executable or dll) and the c runtime library used (MT/MD).
-  # By default, cmake uses link.exe for linking, which fails because we don't
-  # specify the appropiate dependencies.
-  # As we don't want to consider all of that possible situations which depends
-  # on the implementation of the compiler-rt, the simplest option is to change
-  # the rules for linking executables and shared libraries, using the compiler
-  # instead of link.exe. Clang will consider the sanitizer flags, and
-  # automatically provide the required libraries to the linker.
-  set(CMAKE_CXX_LINK_EXECUTABLE "<CMAKE_CXX_COMPILER> <FLAGS> ${CMAKE_CXX_FLAGS} ${CRT_FLAG} <OBJECTS> -o <TARGET> <LINK_LIBRARIES> /link <CMAKE_CXX_LINK_FLAGS> <LINK_FLAGS>")
-  set(CMAKE_CXX_CREATE_SHARED_LIBRARY "<CMAKE_CXX_COMPILER> ${CMAKE_CXX_FLAGS} ${CRT_FLAG} /LD <CMAKE_SHARED_LIBRARY_CXX_FLAGS> <CMAKE_SHARED_LIBRARY_CREATE_CXX_FLAGS> <CMAKE_SHARED_LIBRARY_SONAME_CXX_FLAG> <TARGET_SONAME> -o <TARGET> <OBJECTS> <LINK_LIBRARIES> /link <LINK_FLAGS>")
-endif()
 
-if(APPLE OR MSVC)
+if(APPLE)
   # LeakSanitizer is not supported on OSX and Windows right now
   set(HAS_LSAN 0)
   message(WARNING "LeakSanitizer is not supported."
     " Building and running LibFuzzer LeakSanitizer tests is disabled."
     )
 else()
   set(HAS_LSAN 1)
 endif()

 include_directories(..)
 
 ###############################################################################
 # Configure lit to run the tests
 #
 # Note this is done after declaring all tests so we can inform lit if any tests
 # need to be disabled.
 ###############################################################################
-set(LIBFUZZER_POSIX 1)
-if (MSVC)
-  set(LIBFUZZER_POSIX 0)
-endif()
 
 # Use just-built Clang to compile/link tests on all platforms, except for
 # Windows where we need to use clang-cl instead.
-if(NOT MSVC)
-  set(LIBFUZZER_TEST_COMPILER ${LLVM_RUNTIME_OUTPUT_INTDIR}/clang)
-  set(LIBFUZZER_TEST_CXX_COMPILER ${LLVM_RUNTIME_OUTPUT_INTDIR}/clang++)
-else()
-  set(LIBFUZZER_TEST_COMPILER ${LLVM_RUNTIME_OUTPUT_INTDIR}/clang.exe)
-  set(LIBFUZZER_TEST_CXX_COMPILER ${LLVM_RUNTIME_OUTPUT_INTDIR}/clang++.exe)
-endif()
 
 # LIT-based libFuzzer tests.
 configure_lit_site_cfg(
   ${CMAKE_CURRENT_SOURCE_DIR}/lit.site.cfg.in
   ${CMAKE_CURRENT_BINARY_DIR}/lit.site.cfg
   )
 
 # libFuzzer unit tests.
 configure_lit_site_cfg(
   ${CMAKE_CURRENT_SOURCE_DIR}/unit/lit.site.cfg.in
   ${CMAKE_CURRENT_BINARY_DIR}/unit/lit.site.cfg
   )
 
 add_lit_testsuite(check-fuzzer "Running Fuzzer tests"
     ${CMAKE_CURRENT_BINARY_DIR}
-    DEPENDS LLVMFuzzer-Unittest
-    )
+    DEPENDS LLVMFuzzer-Unittest)
 
-
-# Don't add dependencies on Windows. The linker step would fail on Windows,
-# since cmake will use link.exe for linking and won't include compiler-rt libs.
-if(NOT MSVC)
-  add_dependencies(check-fuzzer LLVMFuzzer asan clang llvm-symbolizer FileCheck sancov not)
-endif()

llvm/trunk/lib/Fuzzer/test/afl-driver-extra-stats.test

-REQUIRES: posix
-
 RUN: %no_fuzzer_cpp_compiler -fno-sanitize-coverage=edge,trace-cmp,indirect-calls,8bit-counters,trace-pc-guard %S/AFLDriverTest.cpp %S/../afl/afl_driver.cpp -o %t-AFLDriverTest
 
 ; Test that not specifying an extra stats file isn't broken.
 RUN: unset AFL_DRIVER_EXTRA_STATS_FILENAME
 RUN: %t-AFLDriverTest
 
 ; Test that specifying an invalid extra stats file causes a crash.
 RUN: ASAN_OPTIONS= AFL_DRIVER_EXTRA_STATS_FILENAME=%T not --crash %t-AFLDriverTest

llvm/trunk/lib/Fuzzer/test/afl-driver-stderr.test

-REQUIRES: posix
-
 RUN: %no_fuzzer_cpp_compiler -fno-sanitize-coverage=edge,trace-cmp,indirect-calls,8bit-counters,trace-pc-guard %S/AFLDriverTest.cpp %S/../afl/afl_driver.cpp -o %t-AFLDriverTest
 
 ; Test that not specifying a stderr file isn't broken.
 RUN: unset AFL_DRIVER_STDERR_DUPLICATE_FILENAME
 RUN: %t-AFLDriverTest
 
 ; Test that specifying an invalid file causes a crash.
 RUN: ASAN_OPTIONS= AFL_DRIVER_STDERR_DUPLICATE_FILENAME="%T" not --crash %t-AFLDriverTest
 
 ; Test that a file is created when specified as the duplicate stderr.
 RUN: AFL_DRIVER_STDERR_DUPLICATE_FILENAME=%t %t-AFLDriverTest
 RUN: stat %t

llvm/trunk/lib/Fuzzer/test/equivalence-signals.test

-REQUIRES: posix
 # Run EquivalenceATest against itself with a small timeout
 # to stress the signal handling and ensure that shmem doesn't mind
 # the signals.
 
 RUN: %cpp_compiler %S/EquivalenceATest.cpp -o %t-LLVMFuzzer-EquivalenceATest
 RUN: %t-LLVMFuzzer-EquivalenceATest -timeout=1 -run_equivalence_server=EQUIV_SIG_TEST & export APID=$!
 RUN: sleep 3
 RUN: %t-LLVMFuzzer-EquivalenceATest -timeout=1 -use_equivalence_server=EQUIV_SIG_TEST -runs=500000 2>&1
 RUN: kill -9 $APID

llvm/trunk/lib/Fuzzer/test/equivalence.test

-REQUIRES: posix
 RUN: %cpp_compiler %S/EquivalenceATest.cpp -o %t-LLVMFuzzer-EquivalenceATest
 RUN: %cpp_compiler %S/EquivalenceBTest.cpp -o %t-LLVMFuzzer-EquivalenceBTest
 
 RUN: %t-LLVMFuzzer-EquivalenceATest -run_equivalence_server=EQUIV_TEST & export APID=$!
 RUN: sleep 3
 RUN: not %t-LLVMFuzzer-EquivalenceBTest -use_equivalence_server=EQUIV_TEST -max_len=4096 2>&1 | FileCheck %s
 CHECK: ERROR: libFuzzer: equivalence-mismatch. Sizes: {{.*}}; offset 2
 CHECK: SUMMARY: libFuzzer: equivalence-mismatch
 RUN: kill -9 $APID

llvm/trunk/lib/Fuzzer/test/lit.cfg

else:
   lit_config.note('lsan feature unavailable')
 
 if sys.platform.startswith('win') or sys.platform.startswith('cygwin'):
   config.available_features.add('windows')
 
 if sys.platform.startswith('darwin'):
   config.available_features.add('darwin')
 
-if config.is_posix:
-  config.available_features.add('posix')
-
 if sys.platform.startswith('linux'):
   # Note the value of ``sys.platform`` is not consistent
   # between python 2 and 3, hence the use of ``.startswith()``.
   lit_config.note('linux feature available')
   config.available_features.add('linux')
 else:
   lit_config.note('linux feature unavailable')
 

llvm/trunk/lib/Fuzzer/test/lit.site.cfg.in

 config.test_exec_root = "@CMAKE_CURRENT_BINARY_DIR@"
 config.llvm_tools_dir = "@LLVM_TOOLS_DIR@"
 config.has_lsan = True if @HAS_LSAN@ == 1 else False
-config.is_posix = @LIBFUZZER_POSIX@
 config.cpp_compiler = "@LIBFUZZER_TEST_CXX_COMPILER@"
 config.c_compiler = "@LIBFUZZER_TEST_COMPILER@"
 config.osx_sysroot = "@CMAKE_OSX_SYSROOT@"
 config.cmake_binary_dir = "@CMAKE_BINARY_DIR@"
 config.target_triple = "@TARGET_TRIPLE@"
 lit_config.load_config(config, "@CMAKE_CURRENT_SOURCE_DIR@/lit.cfg")

llvm/trunk/lib/Fuzzer/test/merge-posix.test

-REQUIRES: posix
-
 RUN: %cpp_compiler %S/FullCoverageSetTest.cpp -o %t-LLVMFuzzer-FullCoverageSetTest
 
 RUN: rm -rf  %tmp/T1 %tmp/T2
 RUN: mkdir -p %tmp/T1 %tmp/T2
 
 RUN: echo F..... > %tmp/T1/1
 RUN: echo .U.... > %tmp/T1/2
 RUN: echo ..Z... > %tmp/T1/3

llvm/trunk/lib/Fuzzer/test/ulimit.test

-REQUIRES: posix
-
 RUN: %cpp_compiler %S/SimpleTest.cpp -o %t-LLVMFuzzer-SimpleTest
 RUN: ulimit -s 1000
 RUN: not %t-LLVMFuzzer-SimpleTest