[sanitizer][symbolizer] Checkout zlib

[vitalybuka]

Summary

And make build script incremental.

We will eventually convert the script into ExternalProject_Add.

[MaskRay]

compiler-rt/lib/sanitizer_common/symbolizer/scripts/build_symbolizer.sh

55

$PWD

[vitalybuka]

PWD

[hoy]

It looks like without ZLIB_SRC there isn't a clear way to specify a custom zlib source. Also the git clone action may not work in an environment without internet access. Can you advise how I should work around those? Thanks.

[MaskRay]

In D158044#4655715, @hoy wrote:

It looks like without ZLIB_SRC there isn't a clear way to specify a custom zlib source. Also the git clone action may not work in an environment without internet access. Can you advise how I should work around those? Thanks.

build_symbolizer.sh is a convenience script that is optional. It's not used if COMPILER_RT_ENABLE_INTERNAL_SYMBOLIZER (default to OFF) is off.
For a constrained environment without internet access, just use the default COMPILER_RT_ENABLE_INTERNAL_SYMBOLIZER=OFF :)

[hoy]

In D158044#4655725, @MaskRay wrote:

In D158044#4655715, @hoy wrote:

It looks like without ZLIB_SRC there isn't a clear way to specify a custom zlib source. Also the git clone action may not work in an environment without internet access. Can you advise how I should work around those? Thanks.

build_symbolizer.sh is a convenience script that is optional. It's not used if COMPILER_RT_ENABLE_INTERNAL_SYMBOLIZER (default to OFF) is off.
For a constrained environment without internet access, just use the default COMPILER_RT_ENABLE_INTERNAL_SYMBOLIZER=OFF :)

I see. But we do want to use the internal symbolizer :) Do you think we can add ZLIB_SRC back while still keeping the git clone there?

[MaskRay]

In D158044#4655726, @hoy wrote:

In D158044#4655725, @MaskRay wrote:

In D158044#4655715, @hoy wrote:

It looks like without ZLIB_SRC there isn't a clear way to specify a custom zlib source. Also the git clone action may not work in an environment without internet access. Can you advise how I should work around those? Thanks.

build_symbolizer.sh is a convenience script that is optional. It's not used if COMPILER_RT_ENABLE_INTERNAL_SYMBOLIZER (default to OFF) is off.
For a constrained environment without internet access, just use the default COMPILER_RT_ENABLE_INTERNAL_SYMBOLIZER=OFF :)

I see. But we do want to use the internal symbolizer :) Do you think we can add ZLIB_SRC back while still keeping the git clone there?

Yes. I think restoring ZLIB_SRC is fine:)

Internally we use Bazel to reimplement the script and don't use the script.
The script is for sanitizer build bots and other users who build the internal symbolizer. I agree that having ZLIB_SRC is useful.

[hoy]

In D158044#4655732, @MaskRay wrote:

In D158044#4655726, @hoy wrote:

In D158044#4655725, @MaskRay wrote:

In D158044#4655715, @hoy wrote:

It looks like without ZLIB_SRC there isn't a clear way to specify a custom zlib source. Also the git clone action may not work in an environment without internet access. Can you advise how I should work around those? Thanks.

build_symbolizer.sh is a convenience script that is optional. It's not used if COMPILER_RT_ENABLE_INTERNAL_SYMBOLIZER (default to OFF) is off.
For a constrained environment without internet access, just use the default COMPILER_RT_ENABLE_INTERNAL_SYMBOLIZER=OFF :)

I see. But we do want to use the internal symbolizer :) Do you think we can add ZLIB_SRC back while still keeping the git clone there?

Yes. I think restoring ZLIB_SRC is fine:)

Internally we use Bazel to reimplement the script and don't use the script.
The script is for sanitizer build bots and other users who build the internal symbolizer. I agree that having ZLIB_SRC is useful.

Sounds good. I can work on the restoration.

compiler-rt/lib/sanitizer_common/symbolizer/scripts/build_symbolizer.sh

 #!/usr/bin/env bash
 #
-# Run as: CLANG=bin/clang ZLIB_SRC=src/zlib \
-#             build_symbolizer.sh runtime_build/lib/clang/4.0.0/lib/linux/
+# Run as: CLANG=bin/clang build_symbolizer.sh out.o
 # zlib can be downloaded from http://www.zlib.net.
 #
 # Script compiles self-contained object file with symbolization code and injects
 # it into the given set of runtime libraries. Script updates only libraries
 # which has unresolved __sanitizer_symbolize_* symbols and matches architecture.
 # Object file is be compiled from LLVM sources with dependencies like libc++ and
 # zlib. Then it internalizes symbols in the file, so that it can be linked
 # into arbitrary programs, avoiding conflicts with the program own symbols and

 
 SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd)
 SRC_DIR=$(readlink -f $SCRIPT_DIR/..)
 TARGE_DIR=$(readlink -f $1)
 COMPILER_RT_SRC=$(readlink -f ${SCRIPT_DIR}/../../../..)
 LLVM_SRC=${LLVM_SRC:-${COMPILER_RT_SRC}/../llvm}
 LLVM_SRC=$(readlink -f $LLVM_SRC)
 
-if [[ "$ZLIB_SRC" == ""  ||
-      ! -x "${ZLIB_SRC}/configure" ||
-      ! -f "${ZLIB_SRC}/zlib.h" ]]; then
-  echo "Missing or incomplete ZLIB_SRC"
-  exit 1
-fi
-ZLIB_SRC=$(readlink -f $ZLIB_SRC)
-
 CLANG="${CLANG:-`which clang`}"
 CLANG_DIR=$(readlink -f $(dirname "$CLANG"))
 
-rm -rf symbolizer
-BUILD_DIR=$(readlink -f ./symbolizer)
-mkdir -p $BUILD_DIR
-cd $BUILD_DIR
-
 CC=$CLANG_DIR/clang
 CXX=$CLANG_DIR/clang++
 TBLGEN=$CLANG_DIR/llvm-tblgen
 OPT=$CLANG_DIR/opt
 AR=$CLANG_DIR/llvm-ar
 LINK=$CLANG_DIR/llvm-link
 
 for F in $CC $CXX $TBLGEN $LINK $OPT $AR; do
   if [[ ! -x "$F" ]]; then
     echo "Missing $F"
      exit 1
   fi
 done
 
+BUILD_DIR=${PWD}/symbolizer
+mkdir -p $BUILD_DIR
+cd $BUILD_DIR
+
 ZLIB_BUILD=${BUILD_DIR}/zlib
 LIBCXX_BUILD=${BUILD_DIR}/libcxx
 LLVM_BUILD=${BUILD_DIR}/llvm
 SYMBOLIZER_BUILD=${BUILD_DIR}/symbolizer
 
 FLAGS=${FLAGS:-}
 TARGET_TRIPLE=$($CC -print-target-triple $FLAGS)
 if [[ "$FLAGS" =~ "-m32" ]] ; then
   # Avoid new wrappers.
   FLAGS+=" -U_FILE_OFFSET_BITS"
 fi
 FLAGS+=" -fPIC -flto -Oz -g0 -DNDEBUG -target $TARGET_TRIPLE -Wno-unused-command-line-argument"
 FLAGS+=" -include ${SRC_DIR}/../sanitizer_redefine_builtins.h -DSANITIZER_COMMON_REDEFINE_BUILTINS_IN_STD -Wno-language-extension-token"
 
 LINKFLAGS="-fuse-ld=lld -target $TARGET_TRIPLE"
 
 # Build zlib.
-mkdir -p ${ZLIB_BUILD}
+[[ -d ${ZLIB_BUILD} ]] || git clone https://github.com/madler/zlib ${ZLIB_BUILD}
 cd ${ZLIB_BUILD}
-cp -r ${ZLIB_SRC}/* .
 AR="${AR}" CC="${CC}" CFLAGS="$FLAGS -Wno-deprecated-non-prototype" RANLIB=/bin/true ./configure --static
 make -j libz.a
 
 # Build and install libcxxabi and libcxx.
 if [[ ! -d ${LIBCXX_BUILD} ]]; then
   mkdir -p ${LIBCXX_BUILD}
   cd ${LIBCXX_BUILD}
   LIBCXX_FLAGS="${FLAGS} -Wno-macro-redefined"