[compiler-rt] Make compiler-rt tests work with relocatable SDKs on OS X

[kubamracek]

Summary

Hi,

I am trying to play a bit with ASan on OS X and I found out that on recent OS X releases, compiler-rt tests fail to build and run with CMake. That's because OS X now provides relocatable SDKs with Xcode and has no system-wide standard headers/libs. I am aware that currently compiler-rt tests are supported only with CMake.

There are actually two issues:

• Newly built clang binary can't find the platform SDK. Either the SDKROOT env has to be set or -isysroot passed on command line.
• Newly build clang binary can't find c++ headers. It expects to find them in "../include/c++" relative to its binary. This issue goes away if you put libcxx into "llvm/projects/" and build libcxx simultaneously with llvm.

My proposed patch adds OS X detection into CMake to add the -isysroot parameter and it creates a symlink pointing from the build directory's include/c++ into Xcode's toolchain (but only if you don't simultaneously build libcxx). It also moves a similar SDKROOT detection from clang's lit config into a util function and uses it in compiler-rt lit tests as well.

One other solution to make the tests build and pass would be to enforce OS X users to set their SDKROOT env property (most OS X users don't have it set) and lit would need to be changed in a way to propagate this env property (it doesn't now). Installing a SDK into a system-wide location (e.g. /usr/include) is discouraged and deprecated on OS X, although manually installing Xcode Command Line Tools does that. Note that this will not solve the other issue (missing c++ headers).

The attached patch is three files, which should go into llvm, compiler-rt and clang projects respectively.

Kuba Brecka

[samsonov]

compiler-rt/trunk/CMakeLists.txt

379 ↗

(On Diff #10188)

I don't like the idea of adding a symlink - it seems too brittle and unexpected that compiler-rt build would for some reason create a symlink in the build tree that would point to c++ headers. If you can't provide path to c++ headers in CMAKE_CXX_FLAGS when you configure a build tree, consider just modifying the compile flags using the output of xcrun.

compiler-rt/trunk/cmake/Modules/AddCompilerRT.cmake

151 ↗

(On Diff #10188)

Please don't use ASan-specific variables here.

llvm/trunk/utils/lit/lit/util.py

171 ↗

(On Diff #10188)

Please send this path to llvm-commits separately, and let lit maintainers decide if they want to add this functionality to lit.

[glider]

Sorry, haven't had time to look yet.
Will try to do that tomorrow.

[kubamracek]

compiler-rt/trunk/CMakeLists.txt

379 ↗

(On Diff #10188)

Providing the path in a variable (either externally or inside cmake) will make compiler-rt build and pass tests, however the output clang binary would still be unable to compile C++ source out of the box:

$ ../llvm-cmake/bin/clang a.cc
a.cc:4:10: fatal error: 'string' file not found
#include <string>
         ^
1 error generated.

On other platforms this is not an issue, because you usually have /usr/include/c++. If this is expected/intentional, we should be more specific with the build instructions for OS X at http://clang.llvm.org/get_started.html.

compiler-rt/trunk/cmake/Modules/AddCompilerRT.cmake

151 ↗

(On Diff #10188)

is there something like COMPILER_RT_COMMON_LINKFLAGS? if not, can I introduce such a variable?

[kubamracek]

Moved the lit-related stuff to http://reviews.llvm.org/D4072.

[glider]

Looks mostly good (see the comments). I'm not sure this doesn't break on 10.8, but let's see if any problems appear.

compiler-rt/trunk/cmake/Modules/AddCompilerRT.cmake

143 ↗

(On Diff #10257)

This comment is misleading. This build config is for compiler-rt, not unit tests.

compiler-rt/trunk/lib/asan/tests/CMakeLists.txt

68 ↗

(On Diff #10257)

Why remove this line? The tests actually do depend on Foundation.

[glider]

I've recently deinstalled LLVM from our Mavericks machine and am seeing the same problem now.
But I'm not convinced it's a good idea to make all these symlink changes in compiler-rt CMake files. Instead this should probably be done in Clang.

[kubamracek]

In D4047#11, @glider wrote:

I've recently deinstalled LLVM from our Mavericks machine and am seeing the same problem now.
But I'm not convinced it's a good idea to make all these symlink changes in compiler-rt CMake files. Instead this should probably be done in Clang.

Yeah, the automatic symlink creation is weird. How about just having a detection that there are c++ headers available, if not then we'll just stop building/testing informing you to either copy or symlink libcxx into your build directory.

For the SDKROOT stuff I still think we should detect and set this automatically, mostly because other parts of LLVM/Clang are doing it already.

[glider]

Yeah, the automatic symlink creation is weird. How about just having a detection that there are c++ headers available, if not then we'll just stop building/testing informing you to either copy or symlink libcxx into your build directory.

Can't this be done by installing the Xcode command line tools? (I've failed to, but perhaps I was doing something wrong)

For the SDKROOT stuff I still think we should detect and set this automatically, mostly because other parts of LLVM/Clang are doing it already.

Any pointers? A quick grep doesn't show any additional -isysroot flags.

compiler-rt/trunk/cmake/Modules/AddCompilerRT.cmake

151 ↗

(On Diff #10257)

I think ASAN_UNITTEST_COMMON_LINKFLAGS should be set in the corresponding build config.

[kubamracek]

For the SDKROOT stuff I still think we should detect and set this automatically, mostly because other parts of LLVM/Clang are doing it already.

Any pointers? A quick grep doesn't show any additional -isysroot flags.

lit tests (utils/lit/lit/util.py, tools/clang/test/lit.cfg, projects/compiler-rt/test/lit.common.cfg). It's only needed when you want to *run* the just built clang without doing stuff like "make install". And compiler-rt uses the newly build clang during its build phase of check-asan, so that's why we're seeing this issue.

[kubamracek]

Can't this be done by installing the Xcode command line tools? (I've failed to, but perhaps I was doing something wrong)

Probably. But the whole point of having relocatable SDKs and toolchains on OS X is so you don't have to install CL Tools, and so you can switch between different compilers/Xcodes/SDKs.

[kubamracek]

Another try to solve the issue with compiler-rt unit tests and relocatable SDKs on OS X. This patch adds a check into the build process that:

• before using the just-built clang, tries to compile a single-line #include <iostream>
• if that fails, prints out a user-friendly message that they should copy or symlink existing C++ headers into the LLVM output directory

[glider]

I like this more. There's yet another option which I ended up with when I've encountered this problem on our OSX bot that bootstraps Clang.
It is to run:

cd ${LLVM_BINARY_DIR} && make -C ${LLVM_CHECKOUT}/projects/libcxx installheaders HEADER_DIR=${PWD}/include

, which installs the libcxx headers from the Clang checkout for of the newly built Clang.

[kubamracek]

Changing the error message to also suggest checking out libcxx from llvm.org and doing "installheaders" into the build directory.

[• ramosian.glider]

LGTM

[glider]

Sorry, the wrong Glider

[kubamracek]

Landed in r217523.

cmake/Modules/CompilerRTCompile.cmake

 include(LLVMParseArguments)
 
 # Compile a source into an object file with COMPILER_RT_TEST_COMPILER using
 # a provided compile flags and dependenices.
 # clang_compile(<object> <source>
 #               CFLAGS <list of compile flags>
 #               DEPS <list of dependencies>)
 macro(clang_compile object_file source)
   parse_arguments(SOURCE "CFLAGS;DEPS" "" ${ARGN})
   get_filename_component(source_rpath ${source} REALPATH)
   if(NOT COMPILER_RT_STANDALONE_BUILD)
     list(APPEND SOURCE_DEPS clang)
   endif()
+  if (TARGET CompilerRTUnitTestCheckCxx)
+    list(APPEND SOURCE_DEPS CompilerRTUnitTestCheckCxx)
+  endif()
   string(REGEX MATCH "[.](cc|cpp)$" is_cxx ${source_rpath})
   if(is_cxx)
     string(REPLACE " " ";" global_flags "${CMAKE_CXX_FLAGS}")
   else()
     string(REPLACE " " ";" global_flags "${CMAKE_C_FLAGS}")
   endif()
   # On Windows, CMAKE_*_FLAGS are built for MSVC but we use the GCC clang.exe
   # which doesn't support flags starting with "/smth". Replace those with
macro(clang_compile object_file source)
   add_custom_command(
     OUTPUT ${object_file}
     COMMAND ${COMPILER_RT_TEST_COMPILER} ${compile_flags} -c
             -o "${object_file}"
             ${source_rpath}
     MAIN_DEPENDENCY ${source}
     DEPENDS ${SOURCE_DEPS})
 endmacro()
+
+# On Darwin, there are no system-wide C++ headers and the just-built clang is
+# therefore not able to compile C++ files unless they are copied/symlinked into
+# ${LLVM_BINARY_DIR}/include/c++
+# The just-built clang is used to build compiler-rt unit tests. Let's detect
+# this before we try to build the tests and print out a suggestion how to fix
+# it.
+# On other platforms, this is currently not an issue.
+macro(clang_compiler_add_cxx_check)
+  if (APPLE)
+    set(CMD
+      "echo '#include <iostream>' | ${COMPILER_RT_TEST_COMPILER} -E -x c++ - > /dev/null"
+      "if [ $? != 0 ] "
+      "  then echo"
+      "  echo 'Your just-built clang cannot find C++ headers, which are needed to build and run compiler-rt tests.'"
+      "  echo 'You should copy or symlink your system C++ headers into ${LLVM_BINARY_DIR}/include/c++'"
+      "  if [ -d $(dirname $(dirname $(xcrun -f clang)))/include/c++ ]"
+      "    then echo 'e.g. with:'"
+      "    echo '  cp -r' $(dirname $(dirname $(xcrun -f clang)))/include/c++ '${LLVM_BINARY_DIR}/include/'"
+      "  elif [ -d $(dirname $(dirname $(xcrun -f clang)))/lib/c++ ]"
+      "    then echo 'e.g. with:'"
+      "    echo '  cp -r' $(dirname $(dirname $(xcrun -f clang)))/lib/c++ '${LLVM_BINARY_DIR}/include/'"
+      "  fi"
+      "  echo 'This can also be fixed by checking out the libcxx project from llvm.org and installing the headers'"
+      "  echo 'into your build directory:'"
+      "  echo '  cd ${LLVM_SOURCE_DIR}/projects && svn co http://llvm.org/svn/llvm-project/libcxx/trunk libcxx'"
+      "  echo '  cd ${LLVM_BINARY_DIR} && make -C ${LLVM_SOURCE_DIR}/projects/libcxx installheaders HEADER_DIR=${LLVM_BINARY_DIR}/include'"
+      "  echo"
+      "  false"
+      "fi"
+      )
+    add_custom_target(CompilerRTUnitTestCheckCxx
+      COMMAND bash -c "${CMD}"
+      COMMENT "Checking that just-built clang can find C++ headers..."
+      DEPENDS clang
+      VERBATIM)
+  endif()
+endmacro()

lib/sanitizer_common/tests/CMakeLists.txt

 include(CompilerRTCompile)
 
+clang_compiler_add_cxx_check()
+
 set(SANITIZER_UNITTESTS
   sanitizer_allocator_test.cc
   sanitizer_atomic_test.cc
   sanitizer_bitvector_test.cc
   sanitizer_bvgraph_test.cc
   sanitizer_common_test.cc
   sanitizer_deadlock_detector_test.cc
   sanitizer_flags_test.cc