[zorg] Add sanitizer-bootstrap-lite bot.

[earthdok]

Summary

Add a new sanitizer bot. The host compiler is clang fixed at a specific
revision; it will not be rebuilt unless the revision number is changed in the
bot config. The host compiler does incremental builds of (A,M)San-instrumented
LLVM/clang, which are then tested with check-llvm, check-clang.

[earthdok]

buildbot/osuosl/master/config/slaves.py

161

No idea what "jobs" does.

[kcc]

mostly looks great

zorg/buildbot/builders/sanitizers/buildbot_bootstrap_lite.sh

2 ↗

(On Diff #17354)

This it not bootstrap at all, I'd prefer another name (fast-sanitizer-build?).

[earthdok]

zorg/buildbot/builders/sanitizers/buildbot_bootstrap_lite.sh

2 ↗

(On Diff #17354)

You're right. How about sanitizer-x86_64-linux-fast?

[earthdok]

• rename
• add a missing bit

[earthdok]

Another thing I don't understand is how fatal and non-fatal steps are configured. I.e. here:

http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/14600

we fail to build the host compiler, so all remaining steps are skipped. On the other hand, when a test fails, we still run the remaining steps:

http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-bootstrap/builds/5327

Yet, it looks like in both cases the script looks exactly the same:

(run_step) || echo @@@STEP_FAILURE@@@

[eugenis]

In D6689#102388, @earthdok wrote:

Another thing I don't understand is how fatal and non-fatal steps are configured. I.e. here:

http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/14600

we fail to build the host compiler, so all remaining steps are skipped. On the other hand, when a test fails, we still run the remaining steps:

http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-bootstrap/builds/5327

Yet, it looks like in both cases the script looks exactly the same:

(run_step) || echo @@@STEP_FAILURE@@@

No, in the first case the line is:
(cd clang_build && cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo \

${CMAKE_COMMON_OPTIONS} $LLVM_CHECKOUT)

And the script sets "-e", so a non-zero exit code in this line is fatal.

This could be made more consistent.

[earthdok]

In D6689#102420, @eugenis wrote:

No, in the first case the line is:
(cd clang_build && cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo \

${CMAKE_COMMON_OPTIONS} $LLVM_CHECKOUT)

But then we have:

(cd clang_build && make clang -j$MAKE_JOBS) || echo @@@STEP_FAILURE@@@

So a cmake failure is fatal, but a build failure is not. Ugh!

[earthdok]

• fix the "build stage1 clang" step

[earthdok]

Committed r224445

buildbot/osuosl/master/config/builders.py

def _get_sanitizer_builders():
            'builddir': "sanitizer-x86_64-linux",
            'factory': SanitizerBuilder.getSanitizerBuildFactory()},
 
           {'name': "sanitizer-x86_64-linux-bootstrap",
            'slavenames' :["sanitizer-buildbot2"],
            'builddir': "sanitizer-x86_64-linux-bootstrap",
            'factory': SanitizerBuilder.getSanitizerBuildFactory()},
 
+          {'name': "sanitizer-x86_64-linux-fast",
+           'slavenames' :["sanitizer-buildbot3"],
+           'builddir': "sanitizer-x86_64-linux-fast",
+           'factory': SanitizerBuilder.getSanitizerBuildFactory()},
+
           #{'name': "llvm-clang-lld-x86_64-ubuntu-sanitize-address",
           # 'slavenames':["hexagon-build-03"],
           # 'builddir':"llvm-clang-lld-x86_64-ubuntu-sanitize-address",
           # 'factory': ClangAndLLDBuilder.getClangAndLLDBuildFactory(
           #                                   buildWithSanitizerOptions=['-fsanitize=address'],
           #                                   env={'PATH':'/usr/local/bin:/usr/bin:/bin'})},
 
           {'name': "sanitizer_x86_64-freebsd",

buildbot/osuosl/master/config/slaves.py

32},
         create_slave("aarch64-foundation", properties={'jobs': 4}, max_builds=1),
 
         # AArch64 Qemu LNT
         create_slave("aarch64-qemu-lnt", properties={'jobs': 4}, max_builds=1),
 
         # AArch64 Clang, Juno ARM Development Platform
         create_slave("juno-aarch64-01", properties={'jobs': 4}, max_builds=1),
 
-        # Ubuntu x86-64, 12x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
+        # Debian 7.7 x86_64 16-core GCE instance
         create_slave("sanitizer-buildbot1", properties={'jobs': 8}, max_builds=1),
-        # Ubuntu x86-64, 24x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
+        # Debian 7.7 x86_64 16-core GCE instance
         create_slave("sanitizer-buildbot2", properties={'jobs': 8}, max_builds=1),
+        # Debian 7.7 x86_64 8-core GCE instance
+        create_slave("sanitizer-buildbot3", properties={'jobs': 8}, max_builds=1),
 
         # zEnterprise 196 (s390x), SLES 11 SP2
         create_slave("systemz-1", properties={'jobs': 4}, max_builds=1),
 
         # Ubuntu 14.04 x86_64, Intel(R) Xeon(R) CPU E5-1650 0 @ 3.20GHz
         create_slave("ericwf-buildslave", properties={'jobs': 4}, max_builds=2),
         # Ubuntu 14.10 x86_64, Intel(R) Xeon(R) CPU E3-1245 V2 @ 3.40GHz
         create_slave('ericwf-buildslave2', properties={'jobs': 4}, max_builds=2),

zorg/buildbot/builders/sanitizers/buildbot_fast.sh

+#!/usr/bin/env bash
+
+set -x
+set -e
+set -u
+
+HERE="$(cd $(dirname $0) && pwd)"
+. ${HERE}/buildbot_functions.sh
+
+ROOT=`pwd`
+PLATFORM=`uname`
+export PATH="/usr/local/bin:$PATH"
+
+if [ "$BUILDBOT_CLOBBER" != "" ]; then
+  echo @@@BUILD_STEP clobber@@@
+  rm -rf llvm
+  rm -rf llvm_build0
+fi
+
+HOST_CLANG_REVISION=223108
+
+MEMORY_SANITIZER_KIND="Memory"
+BUILDBOT_MSAN_ORIGINS=${BUILDBOT_MSAN_ORIGINS:-}
+if [ "$BUILDBOT_MSAN_ORIGINS" != "" ]; then
+    MEMORY_SANITIZER_KIND="MemoryWithOrigins"
+fi
+
+MAKE_JOBS=${MAX_MAKE_JOBS:-16}
+LLVM=$ROOT/llvm
+
+type -a gcc
+type -a g++
+CMAKE_COMMON_OPTIONS="-GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_PARALLEL_LINK_JOBS=3"
+CMAKE_STAGE1_OPTIONS="${CMAKE_COMMON_OPTIONS}"
+
+# Stage 1
+
+if  [ -r host_clang_revision ] && \
+    [ "$(cat host_clang_revision)" == $HOST_CLANG_REVISION ]
+then
+  # Do nothing.
+  echo @@@BUILD_STEP using pre-built stage1 clang at r$HOST_CLANG_REVISION@@@
+else
+  echo @@@BUILD_STEP sync to r$HOST_CLANG_REVISION@@@
+  REAL_BUILDBOT_REVISION=$BUILDBOT_REVISION
+  BUILDBOT_REVISION=$HOST_CLANG_REVISION
+  buildbot_update
+  
+  echo @@@BUILD_STEP build stage1 clang at r$HOST_CLANG_REVISION@@@
+
+  rm -rf host_clang_revision
+
+  # CMake does not notice that the compiler itself has changed. Anyway,
+  # incremental builds of stage2 don't make sense if stage1 compiler has
+  # changed. Clobber the build trees.
+  rm -rf libcxx_build_msan
+  rm -rf llvm_build_msan
+  rm -rf llvm_build_asan
+
+  if [ ! -d llvm_build0 ]; then
+    mkdir llvm_build0
+  fi
+  # Build cxx/cxxabi to fool the compiler check in MSan stage 2.
+  (cd llvm_build0 && cmake ${CMAKE_STAGE1_OPTIONS} $LLVM && \
+    ninja clang compiler-rt llvm-symbolizer)
+  echo $HOST_CLANG_REVISION > host_clang_revision
+
+  BUILDBOT_REVISION=$REAL_BUILDBOT_REVISION
+fi
+
+echo @@@BUILD_STEP update@@@
+buildbot_update
+
+CLANG_PATH=$ROOT/llvm_build0/bin
+CMAKE_STAGE2_COMMON_OPTIONS="\
+  ${CMAKE_COMMON_OPTIONS} \
+  -DCMAKE_C_COMPILER=${CLANG_PATH}/clang \
+  -DCMAKE_CXX_COMPILER=${CLANG_PATH}/clang++ \
+  "
+LLVM_SYMBOLIZER_PATH=${CLANG_PATH}/llvm-symbolizer
+export ASAN_SYMBOLIZER_PATH=${LLVM_SYMBOLIZER_PATH}
+export MSAN_SYMBOLIZER_PATH=${LLVM_SYMBOLIZER_PATH}
+
+
+# Stage 2 / Memory Sanitizer
+
+echo @@@BUILD_STEP build libcxx/msan@@@
+if [ ! -d libcxx_build_msan ]; then
+  mkdir libcxx_build_msan
+fi
+
+(cd libcxx_build_msan && \
+  cmake \
+    ${CMAKE_STAGE2_COMMON_OPTIONS} \
+    -DLLVM_USE_SANITIZER=${MEMORY_SANITIZER_KIND} \
+    $LLVM && \
+  ninja cxx cxxabi) || echo @@@STEP_FAILURE@@@
+
+
+echo @@@BUILD_STEP build clang/msan@@@
+if [ ! -d llvm_build_msan ]; then
+  mkdir llvm_build_msan
+fi
+
+MSAN_LDFLAGS="-lc++abi -Wl,--rpath=${ROOT}/libcxx_build_msan/lib -L${ROOT}/libcxx_build_msan/lib"
+# See http://llvm.org/bugs/show_bug.cgi?id=19071, http://www.cmake.org/Bug/view.php?id=15264
+CMAKE_BUG_WORKAROUND_CFLAGS="$MSAN_LDFLAGS -fsanitize=memory -w"
+MSAN_CFLAGS="-I${ROOT}/libcxx_build_msan/include -I${ROOT}/libcxx_build_msan/include/c++/v1 $CMAKE_BUG_WORKAROUND_CFLAGS"
+
+(cd llvm_build_msan && \
+ cmake ${CMAKE_STAGE2_COMMON_OPTIONS} \
+   -DLLVM_USE_SANITIZER=${MEMORY_SANITIZER_KIND} \
+   -DLLVM_ENABLE_LIBCXX=ON \
+   -DCMAKE_C_FLAGS="${MSAN_CFLAGS}" \
+   -DCMAKE_CXX_FLAGS="${MSAN_CFLAGS}" \
+   -DCMAKE_EXE_LINKER_FLAGS="${MSAN_LDFLAGS}" \
+   $LLVM && \
+ ninja clang) || echo @@@STEP_FAILURE@@@
+
+echo @@@BUILD_STEP check-llvm msan@@@
+
+(cd llvm_build_msan && ninja check-llvm) || echo @@@STEP_WARNINGS@@@
+
+echo @@@BUILD_STEP check-clang msan@@@
+
+(cd llvm_build_msan && ninja check-clang) || echo @@@STEP_FAILURE@@@
+
+
+# Stage 2 / AddressSanitizer
+
+echo @@@BUILD_STEP build clang/asan@@@
+
+# Turn on init-order checker as ASan runtime option.
+export ASAN_OPTIONS="check_initialization_order=true:detect_stack_use_after_return=1:detect_leaks=1"
+CMAKE_ASAN_OPTIONS=" \
+  ${CMAKE_STAGE2_COMMON_OPTIONS} \
+  -DLLVM_USE_SANITIZER=Address \
+  "
+
+if [ ! -d llvm_build_asan ]; then
+  mkdir llvm_build_asan
+fi
+
+(cd llvm_build_asan && \
+ cmake ${CMAKE_ASAN_OPTIONS} $LLVM && \
+ ninja clang) || echo @@@STEP_FAILURE@@@
+
+
+echo @@@BUILD_STEP check-llvm asan@@@
+
+(cd llvm_build_asan && ninja check-llvm) || echo @@@STEP_FAILURE@@@
+
+
+echo @@@BUILD_STEP check-clang asan@@@
+
+(cd llvm_build_asan && ninja check-clang) || echo @@@STEP_FAILURE@@@

zorg/buildbot/builders/sanitizers/buildbot_selector.py

BOT_ASSIGNMENT = {
     'linux-chrome-asan': bash('buildbot_chrome_asan.sh'),
     'linux-chrome-tsan': bash('buildbot_chrome_tsan.sh'),
     'linux-perf-asan': bash('buildbot_perf_asan.sh'),
     'mac10.9-cmake': bash('buildbot_cmake.sh'),
     'mac10.9': bash('buildbot_standard.sh'),
     'sanitizer-ppc64-linux1': bash('buildbot_cmake.sh'),
     'sanitizer-x86_64-linux': bash('buildbot_cmake.sh'),
     'sanitizer-x86_64-linux-bootstrap': bash('buildbot_bootstrap.sh'),
+    'sanitizer-x86_64-linux-fast': bash('buildbot_fast.sh'),
 }
 
 BOT_ADDITIONAL_ENV = {
     'win': {},
     'linux': { 'CHECK_TSAN': '1', 'BUILD_ASAN_ANDROID' : '1' },
     'linux-bootstrap': {},
     'linux-chrome-asan': {},
     'linux-chrome-tsan': {},
     'linux-perf-asan': {},
     'mac10.9-cmake': { 'MAX_MAKE_JOBS': '8' },
     'mac10.9': { 'MAX_MAKE_JOBS': '8' },
     'sanitizer-ppc64-linux1': { 'HAVE_NINJA': '0' },
     'sanitizer-x86_64-linux': { 'BUILD_ANDROID' : '1', 'RUN_ANDROID' : '1', 'HAVE_NINJA' : '1' },
     'sanitizer-x86_64-linux-bootstrap': {},
+    'sanitizer-x86_64-linux-fast': {},
 }
 
 def Main():
   builder = os.environ.get('BUILDBOT_BUILDERNAME')
   print "builder name: %s" % (builder)
   cmd = BOT_ASSIGNMENT.get(builder)
   if not cmd:
     sys.stderr.write('ERROR - unset/invalid builder name\n')