Move sanitizer bots's configs from an external repo to zorg

[kcc]

Summary

Currently, we store the configs for sanitizer bots in a separate repository:
https://code.google.com/p/address-sanitizer/source/browse/#svn%2Ftrunk%2Fbuild%2Fscripts%2Fslave
We are planing to add more hardware to the bots, and before that we would like to
perform some cleanup.
First (this change) we want to move the config files from the external repo to zorg.
Next, we'd like to change zorg/buildbot/builders/SanitizerBuilder.py to use the configs
from zorg instead of the extarnal repo.
Then we'll add some new bots.

[rengolin]

Hi Kostya, thanks for adding these. I haven't reviewed the code yet, but I
fully support this change!

Cheers,
Renato
Hi gkistanova, samsonov, eugenis,

Currently, we store the configs for sanitizer bots in a separate repository:
https://code.google.com/p/address-sanitizer/source/browse/#svn%2Ftrunk%2Fbuild%2Fscripts%2Fslave
We are planing to add more hardware to the bots, and before that we would
like to
perform some cleanup.
First (this change) we want to move the config files from the external repo
to zorg.
Next, we'd like to change zorg/buildbot/builders/SanitizerBuilder.py to use
the configs
from zorg instead of the extarnal repo.
Then we'll add some new bots.

http://reviews.llvm.org/D5531

Files:

zorg/buildbot/builders/sanitizers/
zorg/buildbot/builders/sanitizers/buildbot_bootstrap.sh
zorg/buildbot/builders/sanitizers/buildbot_chrome_asan.sh
zorg/buildbot/builders/sanitizers/buildbot_chrome_tsan.sh
zorg/buildbot/builders/sanitizers/buildbot_cmake.sh
zorg/buildbot/builders/sanitizers/buildbot_functions.sh
zorg/buildbot/builders/sanitizers/buildbot_perf_asan.sh
zorg/buildbot/builders/sanitizers/buildbot_selector.py
zorg/buildbot/builders/sanitizers/buildbot_standard.bat
zorg/buildbot/builders/sanitizers/buildbot_standard.sh
zorg/buildbot/builders/sanitizers/slaveutil.py
zorg/buildbot/builders/sanitizers/test_tsan.sh

---

llvm-commits mailing list
[e-mail]
http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits

[rengolin]

Kostya,

Stupid question, but did you have those scripts before Zorg had any builders? They seem completely disconnected with anything Zorg does. I was expecting a Builder class, like Buildbot docs describe...

cheers,
--renato

[eugenis]

No, this is just a flexible way of writing buildbot scripts. Is not it easier to write a straightforward shell (or python, or any other language) script than muck with ShellCommand and friends?

The idea, and part of implementation, was stolen from the Chromium buildbot.

[rnk]

This seems like a reasonable place to put these scripts to me.

In D5531#7, @rengolin wrote:

Stupid question, but did you have those scripts before Zorg had any builders? They seem completely disconnected with anything Zorg does. I was expecting a Builder class, like Buildbot docs describe...

Yeah, see http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/builders/SanitizerBuilder.py?view=markup . We would change that to point the svn checkout at zorg instead of some Google Code svn repo.

zorg/buildbot/builders/sanitizers/buildbot_chrome_asan.sh

1

I doubt we need these *_chrome*.sh scripts.

zorg/buildbot/builders/sanitizers/slaveutil.py

1

This file looks dead, even in the ASan repo. We probably don't need it.

[rnk]

I should also mention that using these funky shell scripts in this way makes it possible for any committer to make changes the buildbot script and have it go live on the next build without waiting for a master restart.

[kcc]

remove 3 files from the change list

zorg/buildbot/builders/sanitizers/buildbot_chrome_asan.sh

1

These bots (currently private) are using Chromium trunk to test LLVM trunk.
I've removed them for now, but eventually we may still want them here -- Chromium is a great test for LLVM :)

zorg/buildbot/builders/sanitizers/slaveutil.py

1

removed

[rengolin]

In D5531#11, @rnk wrote:

I should also mention that using these funky shell scripts in this way makes it possible for any committer to make changes the buildbot script and have it go live on the next build without waiting for a master restart.

I wanted this before, but I was sceptical to its value. I don't have a strong opinion about separating the scripts to make bot changes easier, but I'd prefer less bash and more python.

If the need to have bash is so it can work better at the shell level, than make the bash part the minimum necessary. Right now, you're duplicating a lot of the functionality on both shell and batch versions, things that the Builder library already does it better.

For example, we're in the process of unifying the repository checkout (svn, git, etc) for the ClangCMakeBuilder and this could be replicated across all other bots (since they all do the same thing), but with your current implementation you won't benefit from it and will increase the maintenance cost.

Also, please use:

#!/usr/bin/env bash

instead of:

#!/bin/bash

to make sure it'll work *everywhere*.

[eugenis]

Is it possible to teach the common checkout code to update these
scripts as well? It should be easier now that they are in the same
repository.

[gkistanova]

For a full picture I'm missing of how these scripts are going to be used with the buildbot. Devil is in the details.
But I could wait with this till the second set of the patches.

The major questions are

1. What build properties and parameters will be supported?
2. How properties will be propagated through the buildbot infrastructure?
3. If more than one script is used with one build, then the build seems to be a bit heavy. If there is only one script for a build, then I'd prefer seeing build steps instead of just one step with a heavy log to get a more granular and precise failure point from the bot. Something similar to what we did for the dragon egg with the scripted builder, perhaps?

[kcc]

These are the exact same scripts that are already used by the existing bots (e.g. lab.llvm.org:8011/builders/sanitizer-x86_64-linux),
but currently they are pulled from an external repo. see /zorg/buildbot/builders/SanitizerBuilder.py:

sanitizer_buildbot = "sanitizer_buildbot"
# Get sanitizer buildbot scripts.
f.addStep(SVN(name='svn-sanitizer-buildbot',
              mode='update',
              baseURL='http://address-sanitizer.googlecode.com/svn/',
              defaultBranch='trunk',
              workdir=sanitizer_buildbot,
              alwaysUseLatest=True))

sanitizer_script = os.path.join("..", sanitizer_buildbot, "build",
                                "scripts", "slave",
                                "buildbot_selector.py")

I am just moving them to zorg svn.
Once they are in zorg svn, we will modify SanitizerBuilder.py to not use the external repo any more.
The current structure: buildbot_selector.py selects which .sh script to invoke, one .sh script for the builder.
(I hope Alexey or Evgeniy can add some details to this)

[gkistanova]

I understand this.
The above questions are for when you will be changing the sanitizer builder.
I'd like to see those questions addressed then.

Looks good to me, assuming you will fix the scripts to reference to the shell on more generic way as Renato suggested.

[samsonov]

Galina,

I'm not sure I'm answering the correct question, but: buildbot configuration for each builder is determined by buildbot_selector.py - it defines which shell script (and with which additional configuration options) should be run on each builder. This file should be updated for every new builder. Note, however, that in order to switch builder to a different configuration (or enable different set of flags there), it's enough to just commit a change to buildbot_selector.py, master restart will not be necessary.

[samsonov]

Oh, regarding your question (3). The log for sanitizer builders is granular enough (see http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/13223), thanks to AnnotatedFactory. We're running a single shell script, but it contains lines like

@@@BUILD_STEP check-asan@@@

which form separate cells in build log.

[kcc]

Changed /usr/bin/bash to "!/usr/bin/env bash"
and udated the scripts to their current version from
code.google.com/p/address-sanitizer

PTAL

[gkistanova]

Thanks, everyone!
LGTM

[kcc]

Closed by commit rL219136 (authored by @kcc).

zorg/buildbot/builders/sanitizers/buildbot_bootstrap.sh

+#!/bin/bash
+
+set -x
+set -e
+set -u
+
+HERE="$(cd $(dirname $0) && pwd)"
+. ${HERE}/buildbot_functions.sh
+
+ROOT=`pwd`
+PLATFORM=`uname`
+export PATH="/usr/local/bin:$PATH"
+
+if [ "$BUILDBOT_CLOBBER" != "" ]; then
+  echo @@@BUILD_STEP clobber@@@
+  rm -rf llvm
+  rm -rf llvm_build0
+fi
+
+# CMake does not notice that the compiler itself has changed.
+# Anyway, incremental builds of stage2 and stage3 compilers don't make sense.
+# Clobber the build trees.
+rm -rf libcxx_build_msan
+rm -rf llvm_build_msan
+rm -rf llvm_build2_msan
+rm -rf llvm_build_asan
+rm -rf llvm_build2_asan
+rm -rf llvm_build_ubsan
+
+MAKE_JOBS=${MAX_MAKE_JOBS:-16}
+LLVM=$ROOT/llvm
+LIBCXX=$LLVM/projects/libcxx
+
+type -a gcc
+type -a g++
+CMAKE_COMMON_OPTIONS="-GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON"
+CMAKE_STAGE1_OPTIONS="${CMAKE_COMMON_OPTIONS}"
+
+echo @@@BUILD_STEP update@@@
+buildbot_update
+
+# Stage 1
+
+echo @@@BUILD_STEP build stage1 clang@@@
+if [ ! -d llvm_build0 ]; then
+  mkdir llvm_build0
+fi
+(cd llvm_build0 && cmake ${CMAKE_STAGE1_OPTIONS} $LLVM && \
+  ninja clang && ninja compiler-rt && ninja llvm-symbolizer) || \
+  echo @@@STEP_FAILURE@@@
+
+CLANG_PATH=$ROOT/llvm_build0/bin
+CMAKE_STAGE2_COMMON_OPTIONS="\
+  ${CMAKE_COMMON_OPTIONS} \
+  -DCMAKE_C_COMPILER=${CLANG_PATH}/clang \
+  -DCMAKE_CXX_COMPILER=${CLANG_PATH}/clang++ \
+  "
+LLVM_SYMBOLIZER_PATH=${CLANG_PATH}/llvm-symbolizer
+export ASAN_SYMBOLIZER_PATH=${LLVM_SYMBOLIZER_PATH}
+export MSAN_SYMBOLIZER_PATH=${LLVM_SYMBOLIZER_PATH}
+
+# Stage 2 / Memory Sanitizer
+
+echo @@@BUILD_STEP build libcxx/msan@@@
+if [ ! -d libcxx_build_msan ]; then
+  mkdir libcxx_build_msan
+fi
+
+LIBCXX_INST=${LIBCXX}/inst
+(cd libcxx_build_msan && \
+  cmake ${CMAKE_STAGE2_COMMON_OPTIONS} \
+    -DLLVM_USE_SANITIZER=Memory \
+    -DLIBCXX_CXX_ABI=libstdc++ \
+    -DLIBCXX_LIBSUPCXX_INCLUDE_PATHS="/usr/local/include/c++/4.9.1;/usr/local/include/c++/4.9.1/x86_64-unknown-linux-gnu" \
+    -DCMAKE_INSTALL_PREFIX=${LIBCXX_INST} \
+    ${LIBCXX} && \
+  ninja install) || echo @@@STEP_FAILURE@@@
+
+echo @@@BUILD_STEP build clang/msan@@@
+if [ ! -d llvm_build_msan ]; then
+  mkdir llvm_build_msan
+fi
+
+MSAN_INCLUDE_FLAGS="-I${LIBCXX_INST}/include/c++/v1"
+MSAN_LINK_FLAGS="-lc++ -Wl,--rpath=${LIBCXX_INST}/lib -L${LIBCXX_INST}/lib"
+
+(cd llvm_build_msan && \
+ cmake ${CMAKE_STAGE2_COMMON_OPTIONS} \
+   -DLLVM_USE_SANITIZER=Memory \
+   -DCMAKE_C_FLAGS="${MSAN_INCLUDE_FLAGS}" \
+   -DCMAKE_CXX_FLAGS="${MSAN_INCLUDE_FLAGS}" \
+   -DCMAKE_EXE_LINKER_FLAGS="${MSAN_LINK_FLAGS}" \
+   $LLVM && \
+ ninja clang) || echo @@@STEP_FAILURE@@@
+
+echo @@@BUILD_STEP check-llvm msan@@@
+
+(cd llvm_build_msan && ninja check-llvm) || echo @@@STEP_FAILURE@@@
+
+echo @@@BUILD_STEP check-clang msan@@@
+
+(cd llvm_build_msan && ninja check-clang) || echo @@@STEP_FAILURE@@@
+
+
+# Stage 3 / MemorySanitizer
+
+echo @@@BUILD_STEP build stage3/msan clang@@@
+
+if [ ! -d llvm_build2_msan ]; then
+  mkdir llvm_build2_msan
+fi
+
+CLANG_MSAN_PATH=$ROOT/llvm_build_msan/bin
+CMAKE_STAGE3_COMMON_OPTIONS="${CMAKE_COMMON_OPTIONS}"
+CMAKE_STAGE3_MSAN_OPTIONS="${CMAKE_STAGE3_COMMON_OPTIONS} -DCMAKE_C_COMPILER=${CLANG_MSAN_PATH}/clang -DCMAKE_CXX_COMPILER=${CLANG_MSAN_PATH}/clang++"
+
+(cd llvm_build2_msan && cmake ${CMAKE_STAGE3_MSAN_OPTIONS} $LLVM && ninja) || \
+  echo @@@STEP_FAILURE@@@
+
+
+echo @@@BUILD_STEP check-all stage3/msan@@@
+
+(cd llvm_build2_msan && ninja check-all) || echo @@@STEP_FAILURE@@@
+
+
+# Stage 2 / AddressSanitizer
+
+echo @@@BUILD_STEP build clang/asan@@@
+
+# Turn on init-order checker as ASan runtime option.
+export ASAN_OPTIONS="check_initialization_order=true:detect_stack_use_after_return=1:detect_leaks=1"
+CMAKE_ASAN_OPTIONS=" \
+  ${CMAKE_STAGE2_COMMON_OPTIONS} \
+  -DLLVM_USE_SANITIZER=Address \
+  "
+
+if [ ! -d llvm_build_asan ]; then
+  mkdir llvm_build_asan
+fi
+
+(cd llvm_build_asan && \
+ cmake ${CMAKE_ASAN_OPTIONS} $LLVM && \
+ ninja clang) || echo @@@STEP_FAILURE@@@
+
+
+echo @@@BUILD_STEP check-llvm asan@@@
+
+(cd llvm_build_asan && ninja check-llvm) || echo @@@STEP_FAILURE@@@
+
+
+echo @@@BUILD_STEP check-clang asan@@@
+
+(cd llvm_build_asan && ninja check-clang) || echo @@@STEP_FAILURE@@@
+
+
+# Stage 3 / AddressSanitizer
+
+echo @@@BUILD_STEP build stage3/asan clang@@@
+
+if [ ! -d llvm_build2_asan ]; then
+  mkdir llvm_build2_asan
+fi
+
+CLANG_ASAN_PATH=$ROOT/llvm_build_asan/bin
+CMAKE_STAGE3_COMMON_OPTIONS="${CMAKE_STAGE2_COMMON_OPTIONS}"
+CMAKE_STAGE3_ASAN_OPTIONS="${CMAKE_STAGE3_COMMON_OPTIONS} -DCMAKE_C_COMPILER=${CLANG_ASAN_PATH}/clang -DCMAKE_CXX_COMPILER=${CLANG_ASAN_PATH}/clang++"
+
+(cd llvm_build2_asan && cmake ${CMAKE_STAGE3_ASAN_OPTIONS} $LLVM && ninja) || \
+  echo @@@STEP_FAILURE@@@
+
+
+echo @@@BUILD_STEP check-all stage3/asan@@@
+
+(cd llvm_build2_asan && ninja check-all) || echo @@@STEP_FAILURE@@@
+
+# Stage 2 / UndefinedBehaviorSanitizer
+echo @@@BUILD_STEP build clang/ubsan@@@
+
+export UBSAN_OPTIONS="external_symbolizer_path=${LLVM_SYMBOLIZER_PATH}:print_stacktrace=1"
+CMAKE_UBSAN_OPTIONS=" \
+  ${CMAKE_STAGE2_COMMON_OPTIONS} \
+  -DCMAKE_BUILD_TYPE=Debug \
+  -DLLVM_USE_SANITIZER=Undefined \
+  "
+
+if [ ! -d llvm_build_ubsan ]; then
+  mkdir llvm_build_ubsan
+fi
+
+(cd llvm_build_ubsan &&
+  cmake ${CMAKE_UBSAN_OPTIONS} $LLVM && \
+  ninja clang) || echo @@@STEP_FAILURE@@@
+
+echo @@@BUILD_STEP check-llvm ubsan@@@
+(cd llvm_build_ubsan && ninja check-llvm) || echo @@@STEP_FAILURE@@@
+
+echo @@@BUILD_STEP check-clang ubsan@@@
+(cd llvm_build_ubsan && ninja check-clang) || echo @@@STEP_FAILURE@@@

zorg/buildbot/builders/sanitizers/buildbot_chrome_asan.sh

+#!/bin/bash
+
+set -x
+set -e
+set -u
+
+HERE="$(cd $(dirname $0) && pwd)"
+. ${HERE}/buildbot_functions.sh
+
+ROOT=`pwd`
+PLATFORM=`uname`
+# for CMake
+export PATH="/usr/local/bin:$PATH"
+GCC_BUILD=/usr/local/gcc-4.8.2
+export PATH="$GCC_BUILD/bin:$PATH"
+export LD_LIBRARY_PATH=$GCC_BUILD/lib64
+
+LLVM_CHECKOUT=$ROOT/llvm
+CLANG_BUILD=$ROOT/clang_build
+CHROME_CHECKOUT=$ROOT/chrome
+##ASAN_TESTS="base_unittests net_unittests remoting_unittests media_unittests unit_tests browser_tests content_browsertests"
+TESTS_NO_SANDBOX="base_unittests cacheinvalidation_unittests cc_unittests cast_unittests crypto_unittests gpu_unittests url_unittests jingle_unittests device_unittests net_unittests ppapi_unittests printing_unittests ipc_tests sync_unit_tests sql_unittests ui_unittests content_unittests remoting_unittests media_unittests unit_tests"
+TESTS_MAYBE_SANDBOX="browser_tests content_browsertests"
+TESTS_NEED_SANDBOX="sandbox_linux_unittests"
+CHROME_TESTS="${TESTS_NO_SANDBOX} ${TESTS_MAYBE_SANDBOX} ${TESTS_NEED_SANDBOX}"
+
+type -a gcc
+type -a g++
+CMAKE_COMMON_OPTIONS="-GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON"
+
+echo @@@BUILD_STEP update@@@
+buildbot_update
+
+# Chrome builder requires depot_tools to be present in $PATH.
+# LLVM build also requires ninja.
+
+echo @@@BUILD_STEP fetch depot_tools@@@
+fetch_depot_tools $ROOT
+
+echo @@@BUILD_STEP build fresh clang@@@
+(
+if [ ! -d $CLANG_BUILD ]; then
+  mkdir $CLANG_BUILD
+fi
+cd $CLANG_BUILD
+export PATH="$PATH:$ROOT/../../../ninja"
+cmake -DCMAKE_BUILD_TYPE=Release ${CMAKE_COMMON_OPTIONS} \
+  -DCMAKE_C_COMPILER=$(which gcc) -DCMAKE_CXX_COMPILER=$(which g++) \
+  $LLVM_CHECKOUT
+ninja clang || echo @@@STEP_FAILURE@@@
+# TODO(glider): build other targets depending on the platform.
+# See https://code.google.com/p/address-sanitizer/wiki/HowToBuild.
+ninja clang_rt.asan-x86_64 clang_rt.asan-i386 clang_rt.asan_cxx-x86_64 clang_rt.asan_cxx-i386 llvm-symbolizer compiler-rt-headers || echo @@@STEP_FAILURE@@@
+)
+
+
+echo @@@BUILD_STEP check out Chromium@@@
+check_out_chromium $CHROME_CHECKOUT
+
+echo @@@BUILD_STEP gclient runhooks@@@
+CUSTOM_GYP_DEFINES="asan=1 lsan=1"
+gclient_runhooks $CHROME_CHECKOUT $CLANG_BUILD "$CUSTOM_GYP_DEFINES"
+
+echo @@@BUILD_STEP clean Chromium build@@@
+(
+cd $CHROME_CHECKOUT/src
+ninja -C out/Release $CHROME_TESTS
+) || exit 1
+
+set_chrome_suid_sandbox
+#export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/debug
+
+GTEST_FLAGS="--brave-new-test-launcher --test-launcher-bot-mode --test-launcher-batch-limit=1 --verbose --test-launcher-print-test-stdio=always --gtest_print_time"
+for test_name in ${TESTS_NO_SANDBOX} ${TESTS_MAYBE_SANDBOX}
+do
+  echo @@@BUILD_STEP running $test_name under ASan + LSan@@@
+  (
+    set +x
+    cd $CHROME_CHECKOUT/src
+    export LLVM_SYMBOLIZER_PATH=$CLANG_BUILD/bin/llvm-symbolizer
+    # See http://dev.chromium.org/developers/testing/addresssanitizer for the
+    # instructions to run ASan.
+    export ASAN_OPTIONS="strict_memcmp=0 replace_intrin=0 detect_leaks=1 symbolize=1"
+    export LSAN_OPTIONS="verbosity=1:suppressions=tools/lsan/suppressions.txt"
+    export ASAN_SYMBOLIZER_PATH="${LLVM_SYMBOLIZER_PATH}"
+    # Without --server-args="-screen 0 1024x768x24" at least some of the Chrome
+    # tests hang: http://crbug.com/242486
+    xvfb-run --server-args="-screen 0 1024x768x24" out/Release/$test_name ${GTEST_FLAGS} --no-sandbox || echo @@@STEP_FAILURE@@@
+    ##((${PIPESTATUS[0]})) && echo @@@STEP_FAILURE@@@ || true
+  )
+done
+
+GTEST_FLAGS="--brave-new-test-launcher --test-launcher-bot-mode --test-launcher-batch-limit=1 --verbose --test-launcher-print-test-stdio=always --gtest_print_time"
+for test_name in ${TESTS_MAYBE_SANDBOX} ${TESTS_NEED_SANDBOX}
+do
+  echo @@@BUILD_STEP running $test_name under ASan w/ sandbox@@@
+  (
+    set +x
+    cd $CHROME_CHECKOUT/src
+    set_chrome_suid_sandbox
+    export LLVM_SYMBOLIZER_PATH=$CLANG_BUILD/bin/llvm-symbolizer
+    # See http://dev.chromium.org/developers/testing/addresssanitizer for the
+    # instructions to run ASan.
+    export ASAN_OPTIONS="strict_memcmp=0 replace_intrin=0 symbolize=false"
+    # Without --server-args="-screen 0 1024x768x24" at least some of the Chrome
+    # tests hang: http://crbug.com/242486
+    xvfb-run --server-args="-screen 0 1024x768x24" out/Release/$test_name ${GTEST_FLAGS} 2>&1 | tools/valgrind/asan/asan_symbolize.py | c++filt 
+    ((${PIPESTATUS[0]})) && echo @@@STEP_FAILURE@@@ || true
+  )
+done

zorg/buildbot/builders/sanitizers/buildbot_chrome_tsan.sh

+#!/bin/bash
+
+# TODO(glider): merge this with buildbot_chrome_asan.sh
+
+set -x
+# set -e # terminate if someone returns 1
+set -u
+
+HERE="$(cd $(dirname $0) && pwd)"
+. ${HERE}/buildbot_functions.sh
+
+ROOT=`pwd`
+PLATFORM=`uname`
+# for CMake
+export PATH="/usr/local/bin:$PATH"
+GCC_BUILD=/usr/local/gcc-4.8.2
+export PATH="$GCC_BUILD/bin:$PATH"
+export LD_LIBRARY_PATH=$GCC_BUILD/lib64
+
+LLVM_CHECKOUT=$ROOT/llvm
+CLANG_BUILD=$ROOT/clang_build
+CHROME_CHECKOUT=$ROOT/chrome
+CHROME_TESTS="base_unittests net_unittests remoting_unittests media_unittests unit_tests browser_tests content_browsertests cast_unittests"
+#CHROME_TESTS="base_unittests net_unittests"
+
+CMAKE_COMMON_OPTIONS="-GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON"
+
+echo @@@BUILD_STEP update@@@
+buildbot_update
+
+# Chrome builder requires depot_tools to be present in $PATH.
+# LLVM build also requires ninja.
+
+echo @@@BUILD_STEP fetch depot_tools@@@
+fetch_depot_tools $ROOT
+
+echo @@@BUILD_STEP build fresh clang@@@
+(
+if [ ! -d $CLANG_BUILD ]; then
+  mkdir $CLANG_BUILD
+fi
+cd $CLANG_BUILD
+export PATH="$PATH:$ROOT/../../../ninja"
+cmake -DCMAKE_BUILD_TYPE=Release ${CMAKE_COMMON_OPTIONS} \
+  -DCMAKE_C_COMPILER=$(which gcc) -DCMAKE_CXX_COMPILER=$(which g++) \
+  $LLVM_CHECKOUT
+ninja clang || echo @@@STEP_FAILURE@@@
+# TODO(glider): build other targets depending on the platform.
+# See https://code.google.com/p/address-sanitizer/wiki/HowToBuild.
+ninja clang_rt.tsan-x86_64 llvm-symbolizer compiler-rt-headers || echo @@@STEP_FAILURE@@@
+)
+
+
+echo @@@BUILD_STEP check out Chromium@@@
+check_out_chromium $CHROME_CHECKOUT
+
+echo @@@BUILD_STEP gclient runhooks@@@
+# See http://dev.chromium.org/developers/testing/threadsanitizer-tsan-v2
+CUSTOM_GYP_DEFINES="tsan=1 disable_nacl=1"
+gclient_runhooks $CHROME_CHECKOUT $CLANG_BUILD "$CUSTOM_GYP_DEFINES"
+
+echo @@@BUILD_STEP clean Chromium build@@@
+(
+cd $CHROME_CHECKOUT/src
+export TSAN_OPTIONS=report_thread_leaks=0  # suppress reports in the host binaries
+ninja -C out/Release $CHROME_TESTS
+) || exit 1
+
+GTEST_FLAGS="--brave-new-test-launcher --test-launcher-bot-mode --test-launcher-batch-limit=1 --verbose --test-launcher-print-test-stdio=always --gtest_print_time"
+for test_name in $CHROME_TESTS
+do
+  echo @@@BUILD_STEP running $test_name@@@
+  (
+    set -e
+    cd $CHROME_CHECKOUT/src
+    # See http://dev.chromium.org/developers/testing/threadsanitizer-tsan-v2
+    # for the instructions to run TSan.
+    export TSAN_OPTIONS="history_size=7 external_symbolizer_path=third_party/llvm-build/Release+Asserts/bin/llvm-symbolizer suppressions=tools/valgrind/tsan_v2/suppressions.txt report_signal_unsafe=0 report_thread_leaks=0" 
+    # Without --server-args="-screen 0 1024x768x24" at least some of the Chrome
+    # tests hang: http://crbug.com/242486
+    xvfb-run --server-args="-screen 0 1024x768x24" out/Release/$test_name ${GTEST_FLAGS} --no-sandbox --child-clean-exit 2>&1
+    ((${PIPESTATUS[0]})) && echo @@@STEP_FAILURE@@@ || true
+  )
+done

zorg/buildbot/builders/sanitizers/buildbot_cmake.sh

+#!/bin/bash
+
+set -x
+set -e
+set -u
+
+HERE="$(dirname $0)"
+. ${HERE}/buildbot_functions.sh
+
+ROOT=`pwd`
+PLATFORM=`uname`
+ARCH=`uname -m`
+export PATH="/usr/local/bin:$PATH"
+
+if [ "$BUILDBOT_CLOBBER" != "" ]; then
+  echo @@@BUILD_STEP clobber@@@
+  rm -rf llvm
+  rm -rf clang_build
+fi
+
+# Always clobber bootstrap build trees.
+rm -rf compiler_rt_build
+rm -rf llvm_build64
+rm -rf llvm_build_ninja
+
+SUPPORTS_32_BITS=${SUPPORTS_32_BITS:-1}
+MAKE_JOBS=${MAX_MAKE_JOBS:-16}
+LLVM_CHECKOUT=$ROOT/llvm
+COMPILER_RT_CHECKOUT=$LLVM_CHECKOUT/projects/compiler-rt
+CMAKE_COMMON_OPTIONS="-DLLVM_ENABLE_ASSERTIONS=ON"
+ENABLE_LIBCXX_FLAG=
+if [ "$PLATFORM" == "Darwin" ]; then
+  CMAKE_COMMON_OPTIONS="${CMAKE_COMMON_OPTIONS} -DPYTHON_EXECUTABLE=/usr/bin/python"
+  ENABLE_LIBCXX_FLAG="-DLLVM_ENABLE_LIBCXX=ON"
+fi
+
+echo @@@BUILD_STEP update@@@
+buildbot_update
+
+
+echo @@@BUILD_STEP lint@@@
+CHECK_LINT=${COMPILER_RT_CHECKOUT}/lib/sanitizer_common/scripts/check_lint.sh
+(LLVM_CHECKOUT=${LLVM_CHECKOUT} ${CHECK_LINT}) || echo @@@STEP_WARNINGS@@@
+
+# Use both gcc and just-built Clang as a host compiler for sanitizer tests.
+# Assume that self-hosted build tree should compile with -Werror.
+echo @@@BUILD_STEP build fresh clang@@@
+if [ ! -d clang_build ]; then
+  mkdir clang_build
+fi
+(cd clang_build && cmake -DCMAKE_BUILD_TYPE=Release \
+    ${CMAKE_COMMON_OPTIONS} $LLVM_CHECKOUT)
+(cd clang_build && make clang -j$MAKE_JOBS) || echo @@@STEP_FAILURE@@@
+
+# If we're building with libcxx, install the headers to clang_build/include.
+if [ ! -z ${ENABLE_LIBCXX_FLAG} ]; then
+(cd clang_build && make -C ${LLVM_CHECKOUT}/projects/libcxx installheaders \
+  HEADER_DIR=${PWD}/include) || echo @@@STEP_FAILURE@@@
+fi
+
+# Do a sanity check on Linux: build and test sanitizers using gcc as a host
+# compiler.
+if [ "$PLATFORM" == "Linux" ]; then
+  echo @@@BUILD_STEP run sanitizer tests in gcc build@@@
+  (cd clang_build && make -j$MAKE_JOBS check-sanitizer) || echo @@@STEP_FAILURE@@@
+  (cd clang_build && make -j$MAKE_JOBS check-asan) || echo @@@STEP_FAILURE@@@
+  (cd clang_build && make -j$MAKE_JOBS check-lsan) || echo @@@STEP_FAILURE@@@
+  (cd clang_build && make -j$MAKE_JOBS check-msan) || echo @@@STEP_FAILURE@@@
+  (cd clang_build && make -j$MAKE_JOBS check-tsan) || echo @@@STEP_FAILURE@@@
+  (cd clang_build && make -j$MAKE_JOBS check-ubsan) || echo @@@STEP_WARNINGS@@@
+  (cd clang_build && make -j$MAKE_JOBS check-dfsan) || echo @@@STEP_WARNINGS@@@
+fi
+
+### From now on we use just-built Clang as a host compiler ###
+CLANG_PATH=${ROOT}/clang_build/bin
+# Build self-hosted tree with fresh Clang and -Werror.
+CMAKE_CLANG_OPTIONS="${CMAKE_COMMON_OPTIONS} -DLLVM_ENABLE_WERROR=ON -DCMAKE_C_COMPILER=${CLANG_PATH}/clang -DCMAKE_CXX_COMPILER=${CLANG_PATH}/clang++"
+BUILD_TYPE=Release
+
+echo @@@BUILD_STEP bootstrap clang@@@
+if [ ! -d llvm_build64 ]; then
+  mkdir llvm_build64
+fi
+(cd llvm_build64 && cmake -DCMAKE_BUILD_TYPE=$BUILD_TYPE \
+    ${CMAKE_CLANG_OPTIONS} -DLLVM_BUILD_EXTERNAL_COMPILER_RT=ON \
+    ${ENABLE_LIBCXX_FLAG} $LLVM_CHECKOUT)
+# First, build only Clang.
+(cd llvm_build64 && make -j$MAKE_JOBS clang) || echo @@@STEP_FAILURE@@@
+
+# If needed, install the headers to clang_build/include.
+if [ ! -z ${ENABLE_LIBCXX_FLAG} ]; then
+(cd llvm_build64 && make -C ${LLVM_CHECKOUT}/projects/libcxx installheaders \
+  HEADER_DIR=${PWD}/include) || echo @@@STEP_FAILURE@@@
+fi
+
+# Now build everything else.
+(cd llvm_build64 && make -j$MAKE_JOBS) || echo @@@STEP_FAILURE@@@
+FRESH_CLANG_PATH=${ROOT}/llvm_build64/bin
+COMPILER_RT_BUILD_PATH=projects/compiler-rt/src/compiler-rt-build
+
+echo @@@BUILD_STEP run asan tests@@@
+(cd llvm_build64 && make -j$MAKE_JOBS check-asan) || echo @@@STEP_FAILURE@@@
+
+if [ "$PLATFORM" == "Linux" -a "$ARCH" == "x86_64" ]; then
+  echo @@@BUILD_STEP run msan unit tests@@@
+  (cd llvm_build64 && make -j$MAKE_JOBS check-msan) || echo @@@STEP_FAILURE@@@
+fi
+
+if [ "$PLATFORM" == "Linux" -a "$ARCH" == "x86_64" ]; then
+  echo @@@BUILD_STEP run 64-bit tsan unit tests@@@
+  (cd llvm_build64 && make -j$MAKE_JOBS check-tsan) || echo @@@STEP_FAILURE@@@
+fi
+
+if [ "$PLATFORM" == "Linux" -a "$ARCH" == "x86_64" ]; then
+  echo @@@BUILD_STEP run 64-bit lsan unit tests@@@
+  (cd llvm_build64 && make -j$MAKE_JOBS check-lsan) || echo @@@STEP_FAILURE@@@
+fi
+
+echo @@@BUILD_STEP run sanitizer_common tests@@@
+(cd llvm_build64 && make -j$MAKE_JOBS check-sanitizer) || echo @@@STEP_FAILURE@@@
+
+echo @@@BUILD_STEP build standalone compiler-rt@@@
+if [ ! -d compiler_rt_build ]; then
+  mkdir compiler_rt_build
+fi
+(cd compiler_rt_build && cmake -DCMAKE_BUILD_TYPE=$BUILD_TYPE \
+  -DCMAKE_C_COMPILER=${FRESH_CLANG_PATH}/clang \
+  -DCMAKE_CXX_COMPILER=${FRESH_CLANG_PATH}/clang++ \
+  -DCOMPILER_RT_INCLUDE_TESTS=ON \
+  -DCOMPILER_RT_ENABLE_WERROR=ON \
+  -DLLVM_CONFIG_PATH=${FRESH_CLANG_PATH}/llvm-config \
+  $COMPILER_RT_CHECKOUT)
+(cd compiler_rt_build && make -j$MAKE_JOBS) || echo @@@STEP_FAILURE@@@
+
+echo @@@BUILD_STEP test standalone compiler-rt@@@
+(cd compiler_rt_build && make -j$MAKE_JOBS check-all) || echo @@@STEP_FAILURE@@@
+
+HAVE_NINJA=${HAVE_NINJA:-1}
+if [ "$PLATFORM" == "Linux" -a $HAVE_NINJA == 1 ]; then
+  echo @@@BUILD_STEP run tests in ninja build tree@@@
+  if [ ! -d llvm_build_ninja ]; then
+    mkdir llvm_build_ninja
+  fi
+  CMAKE_NINJA_OPTIONS="${CMAKE_CLANG_OPTIONS} -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -G Ninja"
+  (cd llvm_build_ninja && cmake -DCMAKE_BUILD_TYPE=$BUILD_TYPE \
+      ${CMAKE_NINJA_OPTIONS} $LLVM_CHECKOUT)
+  ln -sf llvm_build_ninja/compile_commands.json $LLVM_CHECKOUT
+  (cd llvm_build_ninja && ninja check-asan) || echo @@@STEP_FAILURE@@@
+  (cd llvm_build_ninja && ninja check-sanitizer) || echo @@@STEP_FAILURE@@@
+  (cd llvm_build_ninja && ninja check-tsan) || echo @@@STEP_FAILURE@@@
+  (cd llvm_build_ninja && ninja check-msan) || echo @@@STEP_FAILURE@@@
+  (cd llvm_build_ninja && ninja check-lsan) || echo @@@STEP_FAILURE@@@
+  (cd llvm_build_ninja && ninja check-ubsan) || echo @@@STEP_WARNINGS@@@
+  (cd llvm_build_ninja && ninja check-dfsan) || echo @@@STEP_WARNINGS@@@
+fi
+
+BUILD_ANDROID=${BUILD_ANDROID:-0}
+if [ $BUILD_ANDROID == 1 ] ; then
+    echo @@@BUILD_STEP build Android runtime and tests@@@
+    ANDROID_TOOLCHAIN=$ROOT/../../../android-ndk/standalone
+    ANDROID_BUILD_DIR=compiler_rt_build_android
+
+    # Always clobber android build tree.
+    # It has a hidden dependency on clang (through CXX) which is not known to
+    # the build system.
+    rm -rf $ANDROID_BUILD_DIR
+    mkdir $ANDROID_BUILD_DIR
+    ANDROID_FLAGS="--target=arm-linux-androideabi --sysroot=$ANDROID_TOOLCHAIN/sysroot -B$ANDROID_TOOLCHAIN"
+    (cd $ANDROID_BUILD_DIR && \
+        cmake -GNinja -DCMAKE_BUILD_TYPE=$BUILD_TYPE \
+        -DCMAKE_C_COMPILER=$ROOT/llvm_build64/bin/clang \
+        -DCMAKE_CXX_COMPILER=$ROOT/llvm_build64/bin/clang++ \
+        -DLLVM_CONFIG_PATH=$ROOT/llvm_build64/bin/llvm-config \
+        -DCOMPILER_RT_INCLUDE_TESTS=ON \
+        -DCOMPILER_RT_ENABLE_WERROR=ON \
+        -DCMAKE_C_FLAGS="$ANDROID_FLAGS" \
+        -DCMAKE_CXX_FLAGS="$ANDROID_FLAGS" \
+        -DANDROID=1 \
+        -DCOMPILER_RT_TEST_TARGET_TRIPLE=arm-linux-androideabi \
+        -DCOMPILER_RT_TEST_COMPILER_CFLAGS="$ANDROID_FLAGS" \
+        ${CMAKE_COMMON_OPTIONS} \
+        $LLVM_CHECKOUT/projects/compiler-rt)
+    (cd $ANDROID_BUILD_DIR && ninja AsanUnitTests SanitizerUnitTests) || \
+        echo @@@STEP_FAILURE@@@
+fi
+
+RUN_ANDROID=${RUN_ANDROID:-0}
+if [ $RUN_ANDROID == 1 ] ; then
+    echo @@@BUILD_STEP device setup@@@
+    ADB=$ROOT/../../../android-sdk-linux/platform-tools/adb
+    DEVICE_ROOT=/data/local/asan_test
+
+    echo "Rebooting the device"
+    $ADB reboot
+    $ADB wait-for-device
+    sleep 5
+
+    $ADB devices
+
+    ASAN_RT_LIB=libclang_rt.asan-arm-android.so
+    ASAN_RT_LIB_PATH=`find $ANDROID_BUILD_DIR/lib -name $ASAN_RT_LIB`
+    echo "ASan runtime: $ASAN_RT_LIB_PATH"
+    ADB=$ADB $LLVM_CHECKOUT/projects/compiler-rt/lib/asan/scripts/asan_device_setup \
+        --lib $ASAN_RT_LIB_PATH
+    sleep 2
+
+    $ADB shell rm -rf $DEVICE_ROOT
+    $ADB shell mkdir $DEVICE_ROOT
+
+    # Copy asan-rt into toolchain build dir.
+    # Eventually this should be done in the build system.
+    ASAN_RT_INSTALL_DIR=$(ls -d llvm_build64/lib/clang/*/lib/linux/ | tail -1)
+    cp $ASAN_RT_LIB_PATH $ASAN_RT_INSTALL_DIR
+
+    echo @@@BUILD_STEP run asan lit tests [Android]@@@
+
+    (cd $ANDROID_BUILD_DIR && ninja check-asan) || \
+        echo @@@STEP_FAILURE@@@
+
+    echo @@@BUILD_STEP run sanitizer_common tests [Android]@@@
+
+    $ADB push $ANDROID_BUILD_DIR/lib/sanitizer_common/tests/SanitizerTest $DEVICE_ROOT/
+
+    $ADB shell "$DEVICE_ROOT/SanitizerTest; \
+        echo \$? >$DEVICE_ROOT/error_code"
+    $ADB pull $DEVICE_ROOT/error_code error_code && (exit `cat error_code`) || echo @@@STEP_FAILURE@@@
+
+    echo @@@BUILD_STEP run asan tests [Android]@@@
+
+    $ADB push $ANDROID_BUILD_DIR/lib/asan/tests/AsanTest $DEVICE_ROOT/
+    $ADB push $ANDROID_BUILD_DIR/lib/asan/tests/AsanNoinstTest $DEVICE_ROOT/
+
+    NUM_SHARDS=7
+    for ((SHARD=0; SHARD < $NUM_SHARDS; SHARD++)); do
+        $ADB shell "ASAN_OPTIONS=start_deactivated=1 \
+          GTEST_TOTAL_SHARDS=$NUM_SHARDS \
+          GTEST_SHARD_INDEX=$SHARD \
+          asanwrapper $DEVICE_ROOT/AsanTest; \
+          echo \$? >$DEVICE_ROOT/error_code"
+        $ADB pull $DEVICE_ROOT/error_code error_code && echo && (exit `cat error_code`) || echo @@@STEP_FAILURE@@@
+        $ADB shell " \
+          GTEST_TOTAL_SHARDS=$NUM_SHARDS \
+          GTEST_SHARD_INDEX=$SHARD \
+          $DEVICE_ROOT/AsanNoinstTest; \
+          echo \$? >$DEVICE_ROOT/error_code"
+        $ADB pull $DEVICE_ROOT/error_code error_code && echo && (exit `cat error_code`) || echo @@@STEP_FAILURE@@@
+    done
+fi

zorg/buildbot/builders/sanitizers/buildbot_functions.sh

+#!/bin/bash
+
+function buildbot_update {
+    REV_ARG=
+    if [ "$BUILDBOT_REVISION" != "" ]; then
+        REV_ARG="-r$BUILDBOT_REVISION"
+    fi
+    if [ -d llvm ]; then
+        svn cleanup llvm
+    fi
+    for subtree in llvm/tools/clang llvm/projects/compiler-rt llvm/projects/libcxx llvm/projects/libcxxabi
+    do
+      if [ -d ${subtree} ]; then
+        svn cleanup "${subtree}"
+      fi
+    done
+
+    if [ -d llvm -a -d llvm/projects/libcxxabi ]; then
+        svn up llvm $REV_ARG
+        if [ "$REV_ARG" == "" ]; then
+            REV_ARG="-r"$(svn info llvm | grep '^Revision:' | awk '{print $2}')
+        fi
+        for subtree in llvm/tools/clang llvm/projects/compiler-rt llvm/projects/libcxx llvm/projects/libcxxabi
+        do
+          svn up "${subtree}" $REV_ARG
+        done
+    else
+        svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm $REV_ARG
+        if [ "$REV_ARG" == "" ]; then
+            REV_ARG="-r"$(svn info llvm | grep '^Revision:' | awk '{print $2}')
+        fi
+        svn co http://llvm.org/svn/llvm-project/cfe/trunk llvm/tools/clang $REV_ARG
+        svn co http://llvm.org/svn/llvm-project/compiler-rt/trunk llvm/projects/compiler-rt $REV_ARG
+        svn co http://llvm.org/svn/llvm-project/libcxx/trunk llvm/projects/libcxx $REV_ARG
+        svn co http://llvm.org/svn/llvm-project/libcxxabi/trunk llvm/projects/libcxxabi $REV_ARG
+    fi
+}
+
+function set_chrome_suid_sandbox {
+  export CHROME_DEVEL_SANDBOX=/usr/local/sbin/chrome-devel-sandbox
+}
+
+function fetch_depot_tools {
+  ROOT=$1
+  (
+    cd $ROOT
+    if [ ! -d depot_tools ]; then
+      git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
+    fi
+  )
+  export PATH="$ROOT/depot_tools:$PATH"
+}
+
+function check_out_chromium {
+  CHROME_CHECKOUT=$1
+  (
+  if [ ! -d $CHROME_CHECKOUT ]; then
+    mkdir $CHROME_CHECKOUT
+    pushd $CHROME_CHECKOUT
+    fetch --nohooks chromium --nosvn=True 
+
+    # Sync to LKGR, see http://crbug.com/109191
+    mv .gclient .gclient-tmp
+    cat .gclient-tmp  | \
+        sed 's/"safesync_url": ""/"safesync_url": "https:\/\/chromium-status.appspot.com\/git-lkgr"/' > .gclient
+    rm .gclient-tmp
+    popd
+  fi
+  cd $CHROME_CHECKOUT/src
+  git checkout master
+  git pull
+  gclient sync --nohooks --jobs=16
+  )
+}
+
+function gclient_runhooks {
+  CHROME_CHECKOUT=$1
+  CLANG_BUILD=$2
+  CUSTOM_GYP_DEFINES=$3
+  (
+  cd $CHROME_CHECKOUT/src
+  
+  # Clobber Chromium to catch possible LLVM regressions early.
+  rm -rf out/Release
+  
+  export COMMON_GYP_DEFINES="use_allocator=none use_aura=1 clang_use_chrome_plugins=0 component=static_library"
+  export GYP_DEFINES="$CUSTOM_GYP_DEFINES $COMMON_GYP_DEFINES"
+  export GYP_GENERATORS=ninja
+  export CLANG_BIN=$CLANG_BUILD/bin
+  export CC="$CLANG_BIN/clang"
+  export CXX="$CLANG_BIN/clang++"
+  
+  gclient runhooks
+  )
+}

zorg/buildbot/builders/sanitizers/buildbot_perf_asan.sh

+#!/bin/bash
+
+set -x
+set -e
+set -u
+
+HERE="$(cd $(dirname $0) && pwd)"
+. ${HERE}/buildbot_functions.sh
+
+ROOT=`pwd`
+PLATFORM=`uname`
+export PATH="/usr/local/bin:$PATH"
+
+LLVM_CHECKOUT=$ROOT/llvm
+CLANG_BUILD=$ROOT/clang_build
+SPEC_DIRNAME=SPEC_CPU2006v1.2
+SPEC_SRC="${ROOT}/${SPEC_DIRNAME}"
+SPEC_RUNNER=./run_spec_clang_asan.sh
+#SPEC_TESTS='perlbench bzip2'
+SPEC_TESTS='perlbench bzip2 gcc mcf gobmk hmmer sjeng libquantum h264ref omnetpp astar xalancbmk'
+
+CMAKE_COMMON_OPTIONS="-GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON"
+
+echo @@@BUILD_STEP update@@@
+buildbot_update
+
+# LLVM build requires ninja.
+# TODO(glider): make a common function to fetch ninja.
+
+echo @@@BUILD_STEP fetch depot_tools@@@
+(
+  cd $ROOT
+  if [ ! -d depot_tools ]; then
+    git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
+  fi
+)
+export PATH="$ROOT/depot_tools:$PATH"
+
+echo @@@BUILD_STEP build fresh clang@@@
+(
+if [ ! -d $CLANG_BUILD ]; then
+  mkdir $CLANG_BUILD
+fi
+cd $CLANG_BUILD
+export PATH="$PATH:$ROOT/../../../ninja"
+cmake -DCMAKE_BUILD_TYPE=Release ${CMAKE_COMMON_OPTIONS} $LLVM_CHECKOUT
+ninja clang || echo @@@STEP_FAILURE@@@
+# TODO(glider): build other targets depending on the platform.
+# See https://code.google.com/p/address-sanitizer/wiki/HowToBuild.
+ninja clang_rt.asan-x86_64 clang_rt.asan-i386
+)
+
+
+echo @@@BUILD_STEP unpack, patch and install SPEC@@@
+(
+cd $ROOT
+if [ ! -d $SPEC_SRC ]; then
+  (
+  tar -jxf "$ROOT/../../../SPEC/${SPEC_DIRNAME}.tar.bz2"
+  cd $SPEC_SRC
+  wget https://address-sanitizer.googlecode.com/svn/trunk/spec/spec2006-asan.patch
+  yes | ./install.sh
+  patch -p1 -i spec2006-asan.patch
+  )
+fi
+
+if [ ! -e $SPEC_SRC/$SPEC_RUNNER ]; then
+  (
+  cd $SPEC_SRC
+  wget https://address-sanitizer.googlecode.com/svn/trunk/spec/run_spec_clang_asan.sh
+  chmod a+x $SPEC_RUNNER
+  )
+fi
+)
+
+export ASAN_BIN=$CLANG_BUILD/bin
+#export CC="$ASAN_BIN/clang"
+#export CXX="$ASAN_BIN/clang++"
+export PATH="$ASAN_BIN:$PATH"
+
+for test_name in $SPEC_TESTS
+do
+  echo @@@BUILD_STEP running $test_name@@@
+  (
+    set +x
+    cd $SPEC_SRC
+    name=asan-spec
+    size=test
+    SPEC_WRAPPER="perf record -q -o `pwd`/perf.data" $SPEC_RUNNER $name $size $test_name 2>&1 | tee asan-$test_name.log
+    grep "ERROR: AddressSanitizer" asan-$test_name.log && echo @@@STEP_FAILURE@@@
+    perf report
+  )
+done

zorg/buildbot/builders/sanitizers/buildbot_selector.py

+#!/usr/bin/python
+
+import os
+import subprocess
+import sys
+
+THIS_DIR=os.path.dirname(sys.argv[0])
+
+
+def bash(path):
+    return 'bash ' + os.path.join(THIS_DIR, path)
+
+def cmd_call(path):
+    return 'call ' + os.path.join(THIS_DIR, path)
+
+BOT_ASSIGNMENT = {
+    'win': cmd_call('buildbot_standard.bat'),
+    'linux': bash('buildbot_standard.sh'),
+    'linux-bootstrap': bash('buildbot_bootstrap.sh'),
+    'linux-chrome-asan': bash('buildbot_chrome_asan.sh'),
+    'linux-chrome-tsan': bash('buildbot_chrome_tsan.sh'),
+    'linux-perf-asan': bash('buildbot_perf_asan.sh'),
+    'mac10.9-cmake': bash('buildbot_cmake.sh'),
+    'mac10.9': bash('buildbot_standard.sh'),
+    'sanitizer-ppc64-linux1': bash('buildbot_cmake.sh'),
+    'sanitizer-x86_64-linux': bash('buildbot_cmake.sh'),
+    'sanitizer-x86_64-linux-bootstrap': bash('buildbot_bootstrap.sh'),
+}
+
+BOT_ADDITIONAL_ENV = {
+    'win': {},
+    'linux': { 'CHECK_TSAN': '1', 'BUILD_ASAN_ANDROID' : '1' },
+    'linux-bootstrap': {},
+    'linux-chrome-asan': {},
+    'linux-chrome-tsan': {},
+    'linux-perf-asan': {},
+    'mac10.9-cmake': { 'MAX_MAKE_JOBS': '8' },
+    'mac10.9': { 'MAX_MAKE_JOBS': '8' },
+    'sanitizer-ppc64-linux1': { 'HAVE_NINJA': '0' },
+    'sanitizer-x86_64-linux': { 'BUILD_ANDROID' : '1', 'RUN_ANDROID' : '1' },
+    'sanitizer-x86_64-linux-bootstrap': {},
+}
+
+def Main():
+  builder = os.environ.get('BUILDBOT_BUILDERNAME')
+  print "builder name: %s" % (builder)
+  cmd = BOT_ASSIGNMENT.get(builder)
+  if not cmd:
+    sys.stderr.write('ERROR - unset/invalid builder name\n')
+    sys.exit(1)
+
+  print "%s runs: %s\n" % (builder, cmd)
+  sys.stdout.flush()
+
+  bot_env = os.environ
+  add_env = BOT_ADDITIONAL_ENV.get(builder)
+  for var in add_env:
+    bot_env[var] = add_env[var]
+  if 'TMPDIR' in bot_env:
+    del bot_env['TMPDIR']
+
+  retcode = subprocess.call(cmd, env=bot_env, shell=True)
+  sys.exit(retcode)
+
+
+if __name__ == '__main__':
+  Main()

zorg/buildbot/builders/sanitizers/buildbot_standard.bat

+@echo off
+
+:: TODO(timurrrr) echo @@@BUILD_STEP clobber@@@
+
+echo @@@BUILD_STEP update@@@
+:: TODO(timurrrr)
+::if [ "$BUILDBOT_CLOBBER" != "" ]; then
+::  echo @@@BUILD_STEP clobber build@@@
+::  rmdir /S /Q llvm || goto :DIE
+::  rmdir /S /Q llvm-build || goto :DIE
+::  mkdir llvm-build || goto :DIE
+::fi
+
+set REV_ARG=
+if NOT "%BUILDBOT_REVISION%" == "" set REV_ARG="-r%BUILDBOT_REVISION%"
+
+:: call -> because "svn" might be a batch script, ouch
+call svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm %REV_ARG% || goto :DIE
+call svn co http://llvm.org/svn/llvm-project/cfe/trunk llvm/tools/clang %REV_ARG% || goto :DIE
+call svn co http://llvm.org/svn/llvm-project/compiler-rt/trunk llvm/projects/compiler-rt %REV_ARG% || goto :DIE
+
+set ROOT=%cd%
+
+echo @@@BUILD_STEP cmake llvm@@@
+mkdir llvm-build
+cd llvm-build || goto :DIE
+cmake -GNinja -DLLVM_ENABLE_ASSERTIONS=ON -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD=X86 ..\llvm || goto :DIE
+
+echo @@@BUILD_STEP build compiler-rt@@@
+:: Clean compiler-rt to get all the compile-time warnings,
+:: then rebuild it separately before anything else to help us find ASan RTL
+:: compile-time bugs quicker.
+ninja -t clean compiler-rt
+ninja compiler-rt
+
+echo @@@BUILD_STEP build llvm@@@
+ninja || goto :DIE
+
+echo @@@BUILD_STEP run tests@@@
+ninja check-asan check-sanitizer || goto :DIE
+
+cd %ROOT%
+
+:: TODO(timurrrr)
+:: echo @@@BUILD_STEP build asan RTL with clang@@@
+
+echo "ALL DONE"
+goto :EOF
+
+:DIE
+:: TODO(timurrrr) : get the current process's PID?
+taskkill /F /IM cl.exe /T 2>err
+taskkill /F /IM clang.exe /T 2>err
+taskkill /F /IM clang-cl.exe /T 2>err
+taskkill /F /IM cmake.exe /T 2>err
+taskkill /F /IM MSBuild.exe /T 2>err
+taskkill /F /IM WerFault.exe /T 2>err
+exit /b 42

zorg/buildbot/builders/sanitizers/buildbot_standard.sh

+#!/bin/bash
+
+set -x
+set -e
+set -u
+
+HERE="$(dirname $0)"
+. ${HERE}/buildbot_functions.sh
+
+GCC_BUILD=/usr/local/gcc-4.8.2
+export PATH="$GCC_BUILD/bin:$PATH"
+export LD_LIBRARY_PATH=$GCC_BUILD/lib64
+
+if [ "$BUILDBOT_CLOBBER" != "" ]; then
+  echo @@@BUILD_STEP clobber@@@
+  rm -rf llvm
+  rm -rf llvm-build
+fi
+
+PLATFORM=`uname`
+MAKE_JOBS=${MAX_MAKE_JOBS:-16}
+BUILD_ASAN_ANDROID=${BUILD_ASAN_ANDROID:-0}
+CHECK_TSAN=${CHECK_TSAN:-0}
+
+echo @@@BUILD_STEP update@@@
+buildbot_update
+
+echo @@@BUILD_STEP build llvm@@@
+if [ ! -d llvm-build ]; then
+  mkdir llvm-build
+fi
+cd llvm-build
+../llvm/configure --enable-optimized
+make -j$MAKE_JOBS
+cd ..
+BUILD_ROOT=`pwd`
+CLANG_BUILD=$BUILD_ROOT/llvm-build/Release+Asserts
+
+echo @@@BUILD_STEP test llvm@@@
+cd llvm-build
+make check-all || echo @@@STEP_WARNINGS@@@
+
+echo @@@BUILD_STEP sanity check for sanitizer tools@@@
+CLANGXX_BINARY=$CLANG_BUILD/bin/clang++
+echo -e "#include <stdio.h>\nint main(){ return 0; }" > temp.cc
+for xsan in address undefined; do
+  $CLANGXX_BINARY -fsanitize=$xsan -m64 temp.cc -o a.out
+  ./a.out
+  $CLANGXX_BINARY -fsanitize=$xsan -m32 temp.cc -o a.out
+  ./a.out
+done
+if [ "$PLATFORM" == "Linux" ]; then
+  for xsan in thread memory; do
+    $CLANGXX_BINARY -fsanitize=$xsan -m64 temp.cc -o a.out
+    ./a.out
+  done
+fi
+
+if [ $BUILD_ASAN_ANDROID == 1 ] ; then
+  echo @@@BUILD_STEP build asan/android runtime@@@
+  make -j$MAKE_JOBS -C tools/clang/runtime/ \
+      LLVM_ANDROID_TOOLCHAIN_DIR=$BUILD_ROOT/../../../android-ndk/standalone
+fi
+
+if [ $CHECK_TSAN == 1 ] ; then
+  echo @@@BUILD_STEP prepare for testing tsan@@@
+
+  TSAN_PATH=$BUILD_ROOT/llvm/projects/compiler-rt/lib/tsan/
+  (cd $TSAN_PATH && make -f Makefile.old install_deps)
+
+  export PATH=$CLANG_BUILD/bin:$GCC_BUILD/bin:$PATH
+  export MAKEFLAGS=-j$MAKE_JOBS
+  gcc -v 2>tmp && grep "version" tmp
+  clang -v 2>tmp && grep "version" tmp
+
+  cd $BUILD_ROOT
+  if [ -d tsanv2 ]; then
+    (cd tsanv2 && svn up --ignore-externals)
+  else
+    svn co http://data-race-test.googlecode.com/svn/trunk/ tsanv2
+  fi
+  export RACECHECK_UNITTEST_PATH=$BUILD_ROOT/tsanv2/unittest
+
+  cp $BUILD_ROOT/../../../scripts/slave/test_tsan.sh $TSAN_PATH
+  (cd $TSAN_PATH && ./test_tsan.sh)
+fi

zorg/buildbot/builders/sanitizers/slaveutil.py

+#!/usr/bin/python
+
+import os
+import sys
+import socket
+
+def slave_name():
+    hostname = socket.gethostname()
+    dirname = os.path.split(sys.argv[0])[-1]
+    print hostname, dirname
+    if dirname.startswith('slave'):
+        dirname = dirname[5:]
+        if dirname:
+            hostname = hostname + '-' + dirname
+    return hostname
+
+def slave_password():
+    return 'password'

zorg/buildbot/builders/sanitizers/test_tsan.sh

+#!/bin/bash
+
+set -x
+set -e
+set -u
+
+echo @@@BUILD_STEP tsan build debug-clang@@@
+make -f Makefile.old clean
+make -f Makefile.old DEBUG=1 CC=clang CXX=clang++
+
+echo @@@BUILD_STEP tsan test debug-clang@@@
+./tsan_test
+
+echo @@@BUILD_STEP tsan stats/output@@@
+make -f Makefile.old clean
+make -f Makefile.old DEBUG=1 CC=clang CXX=clang++ CFLAGS="-DTSAN_COLLECT_STATS=1 -DTSAN_DEBUG_OUTPUT=2"
+
+echo @@@BUILD_STEP tsan build SHADOW_COUNT=4@@@
+make -f Makefile.old clean
+make -f Makefile.old DEBUG=1 CC=clang CXX=clang++ CFLAGS=-DTSAN_SHADOW_COUNT=4
+
+echo @@@BUILD_STEP tsan test SHADOW_COUNT=4@@@
+./tsan_test
+
+echo @@@BUILD_STEP tsan build SHADOW_COUNT=2@@@
+make -f Makefile.old clean
+make -f Makefile.old DEBUG=1 CC=clang CXX=clang++ CFLAGS=-DTSAN_SHADOW_COUNT=2
+
+echo @@@BUILD_STEP tsan test SHADOW_COUNT=2@@@
+./tsan_test
+
+echo @@@BUILD_STEP tsan build release-gcc@@@
+make -f Makefile.old clean
+make -f Makefile.old DEBUG=0 CC=gcc CXX=g++
+
+echo @@@BUILD_STEP tsan test release-gcc@@@
+./tsan_test
+
+echo @@@BUILD_STEP tsan output_tests@@@
+(cd ../../test/tsan && ./test_output.sh)
+
+echo @@@BUILD_STEP tsan analyze@@@
+./check_analyze.sh
+
+echo @@@BUILD_STEP tsan Go runtime@@@
+(cd go && ./buildgo.sh)
+
+echo @@@BUILD_STEP tsan racecheck_unittest@@@
+TSAN_PATH=`pwd`
+LIBTSAN_A=$TSAN_PATH/rtl/libtsan.a
+SUPPRESS_WARNINGS="-Wno-format-security -Wno-null-dereference -Wno-unused-private-field"
+EXTRA_COMPILER_FLAGS="-fsanitize=thread -DTHREAD_SANITIZER -fPIC -g -O2 $SUPPRESS_WARNINGS"
+(cd $RACECHECK_UNITTEST_PATH && \
+make clean && \
+OMIT_DYNAMIC_ANNOTATIONS_IMPL=1 make l64 -j16 CC=clang CXX=clang++ LDOPT="-pie -Wl,--whole-archive $LIBTSAN_A -Wl,--no-whole-archive -ldl" OMIT_CPP0X=1 EXTRA_CFLAGS="$EXTRA_COMPILER_FLAGS" EXTRA_CXXFLAGS="$EXTRA_COMPILER_FLAGS" && \
+bin/racecheck_unittest-linux-amd64-O0 --gtest_filter=-*Ignore*:*Suppress*:*EnableRaceDetectionTest*:*Rep*Test*:*NotPhb*:*Barrier*:*Death*:*PositiveTests_RaceInSignal*:StressTests.FlushStateTest:*Mmap84GTest:*.LibcStringFunctions:LockTests.UnlockingALockHeldByAnotherThread:LockTests.UnlockTwice:PrintfTests.RaceOnPutsArgument)
+
+#Ignore: ignores do not work yet
+#Suppress: suppressions do not work yet
+#EnableRaceDetectionTest: the annotation is not supported
+#Rep*Test: uses inline assembly
+#NotPhb: not-phb is not supported
+#Barrier: pthread_barrier_t is not fully supported yet
+#Death: there is some flakyness
+#PositiveTests_RaceInSignal: signal() is not intercepted yet
+#StressTests.FlushStateTest: uses suppressions
+#Mmap84GTest: too slow, causes paging
+#LockTests.UnlockingALockHeldByAnotherThread: causes tsan report and non-zero exit code
+#LockTests.UnlockTwice: causes tsan report and non-zero exit code
+#PrintfTests.RaceOnPutsArgument: seems to be an issue with tsan shadow eviction, lit tests contain a similar test and it passes
+